feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Step 24 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08 as a second `SubmissionReview` (the seam from iteration 5 step 4), with no model and no Jev.
|
||||
|
||||
1. `named_settings` reads the settings a Check's statement names: anything in backticks that reads as a name or dotted path, and bare snake_case names and dotted paths. Quoted text, upper-case values and paths such as `roles/x.y` are skipped.
|
||||
2. `SchemaReview` loads the pinned provider's schema once (iteration 4 step 18). For each Check on an Element it looks each named setting up anywhere in that Element's resource type; a path is judged by the setting at its end, and a resource type named in a statement is not a setting. It also reports an Element whose resource type the provider does not have, with close matches.
|
||||
3. `Reviews` runs several reviews as one. The assembly installs the schema review whenever the bundle pins a Terraform provider, and the coverage review after it when that is switched on.
|
||||
4. As with any review, what it raises is returned to the target architect once per run and the next submission is accepted.
|
||||
|
||||
Measured on 2026-10-08 against the real provider:
|
||||
|
||||
| Design | Checks | Raised |
|
||||
|---|---|---|
|
||||
| Case 01, passed validation | 42 | 0 |
|
||||
| Case 02, passed validation | 54 | 2, both the same metric name used as a value (`oldest_unacked_message_age`) |
|
||||
| Known faults added: `startup_cpu_boost` on a Cloud Run job, `ssl_policy` on a backend service, a misspelt resource type | 3 | all 3 |
|
||||
|
||||
So 2 false alarms in 96 Checks, which is why it is a review and not a rule. It does not catch a Check that names no setting at all, such as "deny read roles" on a service account; that still reaches the IaC author and comes back by the route of step 19.
|
||||
|
||||
**Unrun in an Execution.**
|
||||
|
||||
Tests: `tests/execution/test_validation_in_pieces.py`, "the design against the provider's schema" (6).
|
||||
Reference in New Issue
Block a user