feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -186,3 +186,19 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
||||
|
||||
## Step 23: Validation recorded a few verdicts at a time
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||
- **Capability:** The validator's work survives a slow or lost model reply.
|
||||
- **Rationale:** In the run cancelled on 2026-10-07 the validator's model request timed out five times at 180 seconds, about 15 minutes, each time while writing every verdict in one reply. Nothing it had judged was kept. The target architect had the same fault until iteration 4 step 17.
|
||||
- **Where:** `verdict_tools` in `app/tools/workspace.py`, `specialist_tools` in `app/tools/execution.py`, `providers/gcp.yaml`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
|
||||
- **Acceptance criteria:** Verdicts can be recorded over several calls and are attached as `verdicts.json` on submission. Verdicts recorded on the same code are kept across attempts; verdicts on code that has since changed are dropped. The validator has its own model timeout. On the cases, the validator's model timeouts fall to none.
|
||||
|
||||
## Step 24: The design checked against the provider's schema, in code
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||
- **Capability:** A Check that names a setting its resource type lacks is caught before any code is written.
|
||||
- **Rationale:** In the cancelled run two Revisions of the Target architecture were for Checks no code could meet: `ssl_policy` on a backend service, and a deny rule on a service account. The target architect had the schema lookup and did not use it for them. The earlier K-30 (`startup_cpu_boost` on a Cloud Run job) was the same fault.
|
||||
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
|
||||
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user