feat: a design-time coverage review, measured and left off
A reviewer role reads the requirements and recommendations against the Checks when the Target architecture is first submitted. Three runs of case 03: it did not stop the validator sending the design back, and the designs it reviewed ended with 87 and 93 Checks against 52, so they cost more. It stays behind DESIGN_COVERAGE_REVIEW, off (iteration 6 step 13). Also: the schema review no longer reads `true` or `false` in backticks as a setting name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -57,7 +57,8 @@ flowchart TD
|
||||
| 9 | Validation gives a verdict per Check | M | complete |
|
||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
|
||||
| 12 | Why the Target architecture is revised | S | investigated 2026-10-10: two causes, about half each; two instructions changed; a design-time coverage review is proposed, not built (`steps/step-12.md`) |
|
||||
| 12 | Why the Target architecture is revised | S | investigated 2026-10-10: two causes, about half each; two instructions changed; the design-time coverage review it proposed is step 13 (`steps/step-12.md`) |
|
||||
| 13 | A model reviews coverage at design time | M | built and measured on 2026-10-10: it did not prevent the validator's gaps and made designs larger; left off (`steps/step-13.md`) |
|
||||
|
||||
Size: S under a day, M a few days.
|
||||
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Step 13 — A model reviews coverage at design time: built, measured, left off
|
||||
|
||||
Built and run on 2026-10-10. `DESIGN_COVERAGE_REVIEW` is off.
|
||||
|
||||
## What was built
|
||||
|
||||
Step 12 proposed making the validator's reading of coverage when the Target architecture is first submitted, before any code.
|
||||
|
||||
- `agents/coverage-reviewer.md`: a role on the validator's model. It is given the non-functional requirements, the recommendations and the graph, and reports every condition no Check verifies, or that a Check verifies more weakly than the words.
|
||||
- `DesignCoverageReview` in `app/tools/design_review.py`, on the submission-review seam: what it finds goes back to the target architect once and never blocks. A gap naming an id that does not exist is dropped. It runs once in an Execution.
|
||||
|
||||
Tests: `tests/execution/test_design_review.py` (7).
|
||||
|
||||
## Measured
|
||||
|
||||
Case 03, three runs side by side.
|
||||
|
||||
| | Review off | Review on | Review on |
|
||||
|---|---|---|---|
|
||||
| Results | `20261010T105532Z` | `20261010T105535Z` | `20261010T105540Z` |
|
||||
| Score | 100% | 100% | 100% |
|
||||
| Minutes | 23 | 28 | 31 |
|
||||
| Cost, about | $2.90 | $3.33 | $3.75 |
|
||||
| Revisions of the Target architecture | 1, a Check that cannot be implemented | 0 | 1, a gap the validator found |
|
||||
| Gaps the reviewer raised | | 4 | 22 |
|
||||
| Checks in the delivered design | 52 | 87 | 93 |
|
||||
|
||||
The reviewer itself cost $0.19 a run.
|
||||
|
||||
## Why it stays off
|
||||
|
||||
- **It did not stop the validator sending the design back.** In one of the two runs the validator still found a gap, after the reviewer had raised 22.
|
||||
- **It makes the design bigger.** The designs it reviewed ended with 87 and 93 Checks against 52. Every Check is code to write and a verdict to give, so both runs cost more than the run without it, including the one with no Revision.
|
||||
- **Two readers do not agree on what a gap is.** The reviewer and the validator are the same model given the same question, and they raised different things. Asking earlier does not settle what will be asked later.
|
||||
|
||||
Three runs are a small sample, but nothing in them points the other way. The code stays, off, for a later look; removing it is the owner's call.
|
||||
|
||||
## What the runs did show
|
||||
|
||||
Two of the faults behind Revisions are now recurring ones, which is what the lessons loop is for:
|
||||
|
||||
- **A Check asking for a separate encryption key for database backups**: unimplementable, on 2026-10-09 and again today.
|
||||
- **A condition that a region is "low-carbon"**: the validator's gap today, and among what the Jev coverage review kept raising on 2026-10-09. Code can show which region is set; it cannot show the region's carbon rating.
|
||||
|
||||
And one fault in the schema review, fixed: a value in backticks, `true` or `false`, was read as a setting name and raised. `named_settings` now leaves those literals out.
|
||||
Reference in New Issue
Block a user