fix: an unimplementable Check is corrected everywhere it applies, in one Revision
The reason sent back to the target architect says to read every Check against it, and the IaC author records every Check that cannot be met in one submission. Records the investigation of why the Target architecture is revised: two causes, about half each (iteration 6 step 12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# Step 12 — Why the Target architecture is revised
|
||||
|
||||
An investigation on 2026-10-10, from the seven runs of case 03's request on 2026-10-09 (cases 03, 06 and 07). No new runs were made for it.
|
||||
|
||||
## What was found
|
||||
|
||||
Six of the seven runs revised the Target architecture, nine times in all. The run with none cost about $2.44 and took 22 minutes; the run with three cost $5.50 and took 35. Every Revision has one of two causes.
|
||||
|
||||
**A. A Check the code cannot meet as written: 4 Revisions in 3 runs.** The IaC author records it as unimplementable and the design is revised before validation.
|
||||
|
||||
| Check asked for | Why no code can meet it |
|
||||
|---|---|
|
||||
| A separate encryption key for database backups | The key is set on the instance; backups have no key of their own |
|
||||
| A database flag that restricts query access (twice, one Revision apart) | There is no such flag; access is IAM and database grants |
|
||||
| A health check on a backend service in front of a serverless endpoint | A serverless backend does not take health checks |
|
||||
|
||||
Each names a setting that exists in the provider's schema, so the schema review passes it. The fault is in what the product does, not in a name.
|
||||
|
||||
**B. A condition no Check verifies, found by the validator: 5 Revisions in 4 runs.**
|
||||
|
||||
| Requirement | What was missing |
|
||||
|---|---|
|
||||
| NFR-9 | Deletion protection on one Element |
|
||||
| NFR-4 | An SSL policy enforcing TLS 1.2, attached to the proxy |
|
||||
| DR-15 | The region set on three Elements |
|
||||
| NFR-10 | The alert's two thresholds; the Check said only that conditions were set |
|
||||
| NFR-6 | Audit logs for secret access switched on |
|
||||
|
||||
In four of the five a Check existed for the requirement and was weaker than the requirement's words.
|
||||
|
||||
A Revision for A costs about 4 minutes: the design, then the code. One for B costs about 6: the design, the code, and the whole validation again.
|
||||
|
||||
## Would the coverage review have caught B
|
||||
|
||||
`CoverageReview` (iteration 5 step 4), which asks Jev which Check verifies each condition, has been off since its first run. Tried offline against live Jev on these runs' delivered designs:
|
||||
|
||||
- On five finished designs the validator had accepted, it raised 4 to 8 of 42 to 66 conditions each, about one in ten. Most are conditions a Check does cover in other words, or that no code can show, such as encryption at rest that is on by default.
|
||||
- With the Checks added for each gap taken out again, it raised the gap in 1 of 5 (DR-15), raised a different condition of the same requirement in 1, and missed 3.
|
||||
|
||||
It stays off. It is noisy and it misses what the validator finds.
|
||||
|
||||
A structural rule, each Check naming the condition it verifies, would have caught perhaps one of the five: in the other four a Check did claim the condition and was too weak.
|
||||
|
||||
## Changed now
|
||||
|
||||
Two instructions, from the run where the same fault came back a Revision later:
|
||||
|
||||
- The reason sent to the target architect for an unimplementable Check now says to read every Check against it and correct each one it applies to (`app/nodes/router.py`).
|
||||
- The IaC author is told to record every Check that cannot be met in one submission, including a Check with the same fault as one already recorded (`agents/iac-author.md`).
|
||||
|
||||
Neither has run with a real model.
|
||||
|
||||
## What would address B
|
||||
|
||||
The validator finds these gaps by reading the requirements against the Checks. That reading needs no code: it could be done when the Target architecture is submitted, by the same kind of model, and returned to the target architect before anyone approves the design or writes anything. It would add one model run to every Execution and remove a Revision of the design, the code and the validation from the four in seven that had one. Not built: it needs a decision, and a run each way to measure.
|
||||
Reference in New Issue
Block a user