fix: an unimplementable Check is corrected everywhere it applies, in one Revision
The reason sent back to the target architect says to read every Check against it, and the IaC author records every Check that cannot be met in one submission. Records the investigation of why the Target architecture is revised: two causes, about half each (iteration 6 step 12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -57,6 +57,7 @@ flowchart TD
|
|||||||
| 9 | Validation gives a verdict per Check | M | complete |
|
| 9 | Validation gives a verdict per Check | M | complete |
|
||||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||||
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
|
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
|
||||||
|
| 12 | Why the Target architecture is revised | S | investigated 2026-10-10: two causes, about half each; two instructions changed; a design-time coverage review is proposed, not built (`steps/step-12.md`) |
|
||||||
|
|
||||||
Size: S under a day, M a few days.
|
Size: S under a day, M a few days.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# Step 12 — Why the Target architecture is revised
|
||||||
|
|
||||||
|
An investigation on 2026-10-10, from the seven runs of case 03's request on 2026-10-09 (cases 03, 06 and 07). No new runs were made for it.
|
||||||
|
|
||||||
|
## What was found
|
||||||
|
|
||||||
|
Six of the seven runs revised the Target architecture, nine times in all. The run with none cost about $2.44 and took 22 minutes; the run with three cost $5.50 and took 35. Every Revision has one of two causes.
|
||||||
|
|
||||||
|
**A. A Check the code cannot meet as written: 4 Revisions in 3 runs.** The IaC author records it as unimplementable and the design is revised before validation.
|
||||||
|
|
||||||
|
| Check asked for | Why no code can meet it |
|
||||||
|
|---|---|
|
||||||
|
| A separate encryption key for database backups | The key is set on the instance; backups have no key of their own |
|
||||||
|
| A database flag that restricts query access (twice, one Revision apart) | There is no such flag; access is IAM and database grants |
|
||||||
|
| A health check on a backend service in front of a serverless endpoint | A serverless backend does not take health checks |
|
||||||
|
|
||||||
|
Each names a setting that exists in the provider's schema, so the schema review passes it. The fault is in what the product does, not in a name.
|
||||||
|
|
||||||
|
**B. A condition no Check verifies, found by the validator: 5 Revisions in 4 runs.**
|
||||||
|
|
||||||
|
| Requirement | What was missing |
|
||||||
|
|---|---|
|
||||||
|
| NFR-9 | Deletion protection on one Element |
|
||||||
|
| NFR-4 | An SSL policy enforcing TLS 1.2, attached to the proxy |
|
||||||
|
| DR-15 | The region set on three Elements |
|
||||||
|
| NFR-10 | The alert's two thresholds; the Check said only that conditions were set |
|
||||||
|
| NFR-6 | Audit logs for secret access switched on |
|
||||||
|
|
||||||
|
In four of the five a Check existed for the requirement and was weaker than the requirement's words.
|
||||||
|
|
||||||
|
A Revision for A costs about 4 minutes: the design, then the code. One for B costs about 6: the design, the code, and the whole validation again.
|
||||||
|
|
||||||
|
## Would the coverage review have caught B
|
||||||
|
|
||||||
|
`CoverageReview` (iteration 5 step 4), which asks Jev which Check verifies each condition, has been off since its first run. Tried offline against live Jev on these runs' delivered designs:
|
||||||
|
|
||||||
|
- On five finished designs the validator had accepted, it raised 4 to 8 of 42 to 66 conditions each, about one in ten. Most are conditions a Check does cover in other words, or that no code can show, such as encryption at rest that is on by default.
|
||||||
|
- With the Checks added for each gap taken out again, it raised the gap in 1 of 5 (DR-15), raised a different condition of the same requirement in 1, and missed 3.
|
||||||
|
|
||||||
|
It stays off. It is noisy and it misses what the validator finds.
|
||||||
|
|
||||||
|
A structural rule, each Check naming the condition it verifies, would have caught perhaps one of the five: in the other four a Check did claim the condition and was too weak.
|
||||||
|
|
||||||
|
## Changed now
|
||||||
|
|
||||||
|
Two instructions, from the run where the same fault came back a Revision later:
|
||||||
|
|
||||||
|
- The reason sent to the target architect for an unimplementable Check now says to read every Check against it and correct each one it applies to (`app/nodes/router.py`).
|
||||||
|
- The IaC author is told to record every Check that cannot be met in one submission, including a Check with the same fault as one already recorded (`agents/iac-author.md`).
|
||||||
|
|
||||||
|
Neither has run with a real model.
|
||||||
|
|
||||||
|
## What would address B
|
||||||
|
|
||||||
|
The validator finds these gaps by reading the requirements against the Checks. That reading needs no code: it could be done when the Target architecture is submitted, by the same kind of model, and returned to the target architect before anyone approves the design or writes anything. It would add one model run to every Execution and remove a Revision of the design, the code and the validation from the four in seven that had one. Not built: it needs a decision, and a run each way to measure.
|
||||||
@@ -13,7 +13,8 @@ You produce the **deployment guidance** Deliverable: the Skill's task for genera
|
|||||||
- Keep the files in your working copy. Write each file once with `write_file`, and make every later change with `edit_file`, changing only the lines that need it. Never write a file again to fix part of it.
|
- Keep the files in your working copy. Write each file once with `write_file`, and make every later change with `edit_file`, changing only the lines that need it. Never write a file again to fix part of it.
|
||||||
- Format and validate with `terraform_fmt` and `terraform_validate` when they are available. They run on your working copy, so pass them no files. Fix everything they report.
|
- Format and validate with `terraform_fmt` and `terraform_validate` when they are available. They run on your working copy, so pass them no files. Fix everything they report.
|
||||||
- When validation says an argument or block is not expected, look the resource type up with `terraform_schema` instead of guessing where it belongs.
|
- When validation says an argument or block is not expected, look the resource type up with `terraform_schema` instead of guessing where it belongs.
|
||||||
- When a Check names a setting its resource type does not have, do not invent code for it. Record the Check in `implementation.json` as unimplementable, with what the schema shows, and implement the rest. The Target architecture is then revised.
|
- When a Check names a setting its resource type does not have, or asks for something that resource cannot do, do not invent code for it. Record the Check in `implementation.json` as unimplementable, with what the schema shows, and implement the rest. The Target architecture is then revised.
|
||||||
|
- Go through every Check before you submit and record every one that cannot be met, in that one submission. A Check with the same fault as one you recorded is recorded too. Each one found later costs another Revision of the design and of your code.
|
||||||
- Write `implementation.json` to your working copy, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
|
- Write `implementation.json` to your working copy, in the format the requirements-chain Skill gives, naming where the code satisfies each Check.
|
||||||
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
|
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
|
||||||
- Submit as soon as validation passes, with your content and no attachments: your working copy is attached for you. If a submission is refused, fix exactly what the refusal lists and submit again.
|
- Submit as soon as validation passes, with your content and no attachments: your working copy is attached for you. If a submission is refused, fix exactly what the refusal lists and submit again.
|
||||||
|
|||||||
@@ -634,6 +634,12 @@ class Router:
|
|||||||
"a reason."
|
"a reason."
|
||||||
for item in r.unimplementable
|
for item in r.unimplementable
|
||||||
)
|
)
|
||||||
|
# In the run of 2026-10-09 a second Check with the same fault came back
|
||||||
|
# one Revision later, for the same reason word for word.
|
||||||
|
reason += (
|
||||||
|
" The same fault may be in other Checks: read every Check against "
|
||||||
|
"each reason given here and correct every one it applies to."
|
||||||
|
)
|
||||||
return await call(
|
return await call(
|
||||||
"reopen", kind=DeliverableKind.TARGET_ARCHITECTURE.value, reason=reason
|
"reopen", kind=DeliverableKind.TARGET_ARCHITECTURE.value, reason=reason
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -303,7 +303,9 @@ async def test_a_check_the_code_cannot_meet_goes_back_to_the_graph_unvalidated(
|
|||||||
"target_architecture",
|
"target_architecture",
|
||||||
(
|
(
|
||||||
"Check K-2 cannot be implemented as written (no such argument). "
|
"Check K-2 cannot be implemented as written (no such argument). "
|
||||||
"Rewrite it so it can be met, or remove it with a reason."
|
"Rewrite it so it can be met, or remove it with a reason. The same "
|
||||||
|
"fault may be in other Checks: read every Check against each reason "
|
||||||
|
"given here and correct every one it applies to."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
]
|
]
|
||||||
|
|||||||
Reference in New Issue
Block a user