feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 23 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `verdict_put` records verdicts a few Checks per call, replacing any already recorded for the same Check, and takes the dry-run's result and proposed additions. Each reply says how many Checks have a verdict and names those left. `verdict_read` returns what is recorded.
|
||||
2. `submit_deliverable` attaches the recorded verdicts as `verdicts.json` when none is passed. Passing the file still works.
|
||||
3. The verdicts are kept in the Deliverable's working memory with a mark of the code they were given on (a hash of the submitted deployment guidance's files). A later attempt on the same code carries on from them; on changed code they are dropped before any tool runs.
|
||||
4. The validator's model timeout is 300 seconds (`models.timeout_seconds` in `providers/gcp.yaml`); it had the default 180.
|
||||
5. `agents/validator.md` tells it to record as it goes, one Element's Checks at a time, and to submit with no attachments.
|
||||
|
||||
**Unrun:** no evaluation since.
|
||||
|
||||
Tests: `tests/execution/test_validation_in_pieces.py`, "verdicts a few at a time" (7).
|
||||
Reference in New Issue
Block a user