feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07: - The validator records verdicts a few Checks at a time (verdict_put) and they are attached on submission. It timed out five times writing them in one reply. Verdicts on the same code survive a run cut short; verdicts on changed code are dropped. Its model timeout is 300 s. - A schema review at submission, in code: every setting a Check names is looked up on its Element's resource type, and an unknown resource type is reported. Two Revisions in that run were for Checks no code could meet. On the two designs that passed validation it raises 2 of 96 Checks, both one metric name used as a value, so it is returned once and never blocks. - Reviews run together; the Jev coverage review stays off. Plans record both steps and the cancelled run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -9,7 +9,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 1 | `wf-decompose-65ea49fc2d04/` | complete, superseded | The decomposer's scaffold: a linear four-phase pipeline of template functions. Its code was deleted in iteration 3 and survives only in git history. Generated by the platform decomposer; leave its files as written. |
|
||||
| 2 | `iter-02-solution-architect-agent/` | complete; step 9 was done in iteration 3 | The rebuild as a hierarchical ReAct agent: Orchestrator, one Specialist per Deliverable, Skills, provider bundle, A2A. |
|
||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–22 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
@@ -186,3 +186,19 @@ Every step ends with the full suite passing. Steps that change agent behaviour (
|
||||
- **Where:** `agents/validator.md`, `agents/target-architect.md`, `skills/requirements-chain/SKILL.md`.
|
||||
- **Acceptance criteria:** The validator is told to review coverage completely in its first pass and, after a Revision made for its additions, to propose only what is still uncovered. The target architect is told to write a Check for each condition and each Element a requirement names. On the cases, Target architecture Revisions fall against case 02's two.
|
||||
|
||||
## Step 23: Validation recorded a few verdicts at a time
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||
- **Capability:** The validator's work survives a slow or lost model reply.
|
||||
- **Rationale:** In the run cancelled on 2026-10-07 the validator's model request timed out five times at 180 seconds, about 15 minutes, each time while writing every verdict in one reply. Nothing it had judged was kept. The target architect had the same fault until iteration 4 step 17.
|
||||
- **Where:** `verdict_tools` in `app/tools/workspace.py`, `specialist_tools` in `app/tools/execution.py`, `providers/gcp.yaml`, `agents/validator.md`, `skills/iac-validation/SKILL.md`.
|
||||
- **Acceptance criteria:** Verdicts can be recorded over several calls and are attached as `verdicts.json` on submission. Verdicts recorded on the same code are kept across attempts; verdicts on code that has since changed are dropped. The validator has its own model timeout. On the cases, the validator's model timeouts fall to none.
|
||||
|
||||
## Step 24: The design checked against the provider's schema, in code
|
||||
|
||||
- **Report ref:** none; requested by the owner on 2026-10-08 after the cancelled run
|
||||
- **Capability:** A Check that names a setting its resource type lacks is caught before any code is written.
|
||||
- **Rationale:** In the cancelled run two Revisions of the Target architecture were for Checks no code could meet: `ssl_policy` on a backend service, and a deny rule on a service account. The target architect had the schema lookup and did not use it for them. The earlier K-30 (`startup_cpu_boost` on a Cloud Run job) was the same fault.
|
||||
- **Where:** `app/tools/schema_review.py` (new), the assembly in `app/agent.py`, `agents/target-architect.md`.
|
||||
- **Acceptance criteria:** At submission, every setting a Check's statement names is looked up on its Element's resource type, and a resource type the provider lacks is reported. What is raised is returned once and never blocks. On designs that passed validation it raises little or nothing.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
||||
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 drafted and awaiting the owner; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
||||
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
||||
**This directory covers:** steps 1 to 22
|
||||
**This directory covers:** steps 1 to 24
|
||||
|
||||
## Sources of truth
|
||||
|
||||
@@ -39,6 +39,8 @@
|
||||
| 20 | A model timeout per role | added | S | built |
|
||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
||||
| 23 | Validation recorded a few verdicts at a time | added | S | built; unrun |
|
||||
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; unrun in an Execution |
|
||||
|
||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||
|
||||
|
||||
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
13
.agents/plans/iter-04-learning-and-skills/steps/step-23.md
Normal file
@@ -0,0 +1,13 @@
|
||||
# Step 23 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08.
|
||||
|
||||
1. `verdict_put` records verdicts a few Checks per call, replacing any already recorded for the same Check, and takes the dry-run's result and proposed additions. Each reply says how many Checks have a verdict and names those left. `verdict_read` returns what is recorded.
|
||||
2. `submit_deliverable` attaches the recorded verdicts as `verdicts.json` when none is passed. Passing the file still works.
|
||||
3. The verdicts are kept in the Deliverable's working memory with a mark of the code they were given on (a hash of the submitted deployment guidance's files). A later attempt on the same code carries on from them; on changed code they are dropped before any tool runs.
|
||||
4. The validator's model timeout is 300 seconds (`models.timeout_seconds` in `providers/gcp.yaml`); it had the default 180.
|
||||
5. `agents/validator.md` tells it to record as it goes, one Element's Checks at a time, and to submit with no attachments.
|
||||
|
||||
**Unrun:** no evaluation since.
|
||||
|
||||
Tests: `tests/execution/test_validation_in_pieces.py`, "verdicts a few at a time" (7).
|
||||
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
22
.agents/plans/iter-04-learning-and-skills/steps/step-24.md
Normal file
@@ -0,0 +1,22 @@
|
||||
# Step 24 — Implementation Detail
|
||||
|
||||
Built on 2026-10-08 as a second `SubmissionReview` (the seam from iteration 5 step 4), with no model and no Jev.
|
||||
|
||||
1. `named_settings` reads the settings a Check's statement names: anything in backticks that reads as a name or dotted path, and bare snake_case names and dotted paths. Quoted text, upper-case values and paths such as `roles/x.y` are skipped.
|
||||
2. `SchemaReview` loads the pinned provider's schema once (iteration 4 step 18). For each Check on an Element it looks each named setting up anywhere in that Element's resource type; a path is judged by the setting at its end, and a resource type named in a statement is not a setting. It also reports an Element whose resource type the provider does not have, with close matches.
|
||||
3. `Reviews` runs several reviews as one. The assembly installs the schema review whenever the bundle pins a Terraform provider, and the coverage review after it when that is switched on.
|
||||
4. As with any review, what it raises is returned to the target architect once per run and the next submission is accepted.
|
||||
|
||||
Measured on 2026-10-08 against the real provider:
|
||||
|
||||
| Design | Checks | Raised |
|
||||
|---|---|---|
|
||||
| Case 01, passed validation | 42 | 0 |
|
||||
| Case 02, passed validation | 54 | 2, both the same metric name used as a value (`oldest_unacked_message_age`) |
|
||||
| Known faults added: `startup_cpu_boost` on a Cloud Run job, `ssl_policy` on a backend service, a misspelt resource type | 3 | all 3 |
|
||||
|
||||
So 2 false alarms in 96 Checks, which is why it is a review and not a rule. It does not catch a Check that names no setting at all, such as "deny read roles" on a service account; that still reaches the IaC author and comes back by the route of step 19.
|
||||
|
||||
**Unrun in an Execution.**
|
||||
|
||||
Tests: `tests/execution/test_validation_in_pieces.py`, "the design against the provider's schema" (6).
|
||||
@@ -93,3 +93,14 @@ Changed after the run, not yet run: iteration 4 steps 21 and 22.
|
||||
|
||||
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
|
||||
|
||||
## Run of 2026-10-07, 19:58 UTC, cancelled (`.cache/eval-logs/`; no report)
|
||||
|
||||
Case 02 again on commit `5adc911`, with `JEV_COVERAGE_REVIEW=true`. Cancelled by the owner at 66 minutes, during a fifth validation. No score and no token count were written.
|
||||
|
||||
- The Target architecture was reopened 3 times (the limit): twice for a Check the code's author reported as unimplementable, once for additions validation proposed. The code was reopened twice for failed Checks. Target architect 5 runs, IaC author 6, validator 5.
|
||||
- 6 model timeouts, 5 of them the validator writing all its verdicts in one reply: about 15 minutes.
|
||||
- The coverage review raised 5, 10, 11 and 9 of 61 conditions over four submissions. Validation still proposed a TLS policy and an access grant afterwards. The agents had written conditions no infrastructure code can prove, such as an availability percentage, and Jev answered "none" to those. It stays off.
|
||||
- What held: the description was kept on two Revisions (iteration 4 step 21); the unimplementable-Check route sent the design back before any validation was spent (step 19); code fixes took 1 to 5 minutes from the working copy; no `terraform validate` failure.
|
||||
|
||||
Changed after it, not yet run: iteration 4 steps 23 and 24.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user