Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Jonathan Boniface 58e778ca7a feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07:

- The validator records verdicts a few Checks at a time (verdict_put) and
  they are attached on submission. It timed out five times writing them in
  one reply. Verdicts on the same code survive a run cut short; verdicts on
  changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
  looked up on its Element's resource type, and an unknown resource type is
  reported. Two Revisions in that run were for Checks no code could meet.
  On the two designs that passed validation it raises 2 of 96 Checks, both
  one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.

Plans record both steps and the cancelled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:39:13 +01:00

1.8 KiB

Step 24 — Implementation Detail

Built on 2026-10-08 as a second SubmissionReview (the seam from iteration 5 step 4), with no model and no Jev.

  1. named_settings reads the settings a Check's statement names: anything in backticks that reads as a name or dotted path, and bare snake_case names and dotted paths. Quoted text, upper-case values and paths such as roles/x.y are skipped.
  2. SchemaReview loads the pinned provider's schema once (iteration 4 step 18). For each Check on an Element it looks each named setting up anywhere in that Element's resource type; a path is judged by the setting at its end, and a resource type named in a statement is not a setting. It also reports an Element whose resource type the provider does not have, with close matches.
  3. Reviews runs several reviews as one. The assembly installs the schema review whenever the bundle pins a Terraform provider, and the coverage review after it when that is switched on.
  4. As with any review, what it raises is returned to the target architect once per run and the next submission is accepted.

Measured on 2026-10-08 against the real provider:

Design Checks Raised
Case 01, passed validation 42 0
Case 02, passed validation 54 2, both the same metric name used as a value (oldest_unacked_message_age)
Known faults added: startup_cpu_boost on a Cloud Run job, ssl_policy on a backend service, a misspelt resource type 3 all 3

So 2 false alarms in 96 Checks, which is why it is a review and not a rule. It does not catch a Check that names no setting at all, such as "deny read roles" on a service account; that still reaches the IaC author and comes back by the route of step 19.

Unrun in an Execution.

Tests: tests/execution/test_validation_in_pieces.py, "the design against the provider's schema" (6).