feat: record verdicts in pieces and check the design against the schema

From the case 02 run cancelled at 66 minutes on 2026-10-07:

- The validator records verdicts a few Checks at a time (verdict_put) and
  they are attached on submission. It timed out five times writing them in
  one reply. Verdicts on the same code survive a run cut short; verdicts on
  changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
  looked up on its Element's resource type, and an unknown resource type is
  reported. Two Revisions in that run were for Checks no code could meet.
  On the two designs that passed validation it raises 2 of 96 Checks, both
  one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.

Plans record both steps and the cancelled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-08 09:39:13 +01:00
parent 5adc911a82
commit 58e778ca7a
16 changed files with 691 additions and 14 deletions

View File

@@ -186,3 +186,19 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
## Step 23: Validation recorded a few verdicts at a time
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
- **Capability:** The validator's work survives a slow or lost model reply.
- **Rationale:** In the run cancelled on 2026-10-07 the validator's model request timed out five times at 180 seconds, about 15 minutes, each time while writing every verdict in one reply. Nothing it had judged was kept. The target architect had the same fault until iteration 4 step 17.
- **Where:** `verdict_tools` in `app/tools/workspace.py`, `specialist_tools` in `app/tools/execution.py`, `providers/gcp.yaml`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
- **Acceptance criteria:** Verdicts can be recorded over several calls and are attached as `verdicts.json` on submission. Verdicts recorded on the same code are kept across attempts; verdicts on code that has since changed are dropped. The validator has its own model timeout. On the cases, the validator's model timeouts fall to none.
## Step 24: The design checked against the provider's schema, in code
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
- **Capability:** A Check that names a setting its resource type lacks is caught before any code is written.
- **Rationale:** In the cancelled run two Revisions of the Target architecture were for Checks no code could meet: `ssl_policy` on a backend service, and a deny rule on a service account. The target architect had the schema lookup and did not use it for them. The earlier K-30 (`startup_cpu_boost` on a Cloud Run job) was the same fault.
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.