chore: raise the validator's budget and record the runs of cases 02, 03 and 04
- Validator budget 15 to 30: recording verdicts in pieces for 100 Checks used all 15 calls in one attempt. - Plans record three runs on 2026-10-08: case 02 completed at 100% with no Revision in 24 minutes; case 03 completed at 100% with two Revisions of the design in 40 minutes; case 04 (interactive) ended with warnings at 75%, on two router faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit. Neither is fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,7 +11,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
|||||||
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
| 3 | `iter-03-kab-layout/` | complete (commit `7ee49ab`) | The container starts the new engine, the engine moved into the KAB workflow-agent folder layout, and the original pipeline was deleted. |
|
||||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–24 built; case 01 completed on them on 2026-10-07 (18 minutes against 60, 100%, no Revision); cases 02 to 04 have not run; 9 awaits the owner; 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||||
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01 and 02 pass with real models; cases 03 and 04 outstanding | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
| 6 | `iter-06-target-architecture-graph/` | steps 1–10 built; cases 01, 02 and 03 pass with real models; case 04 (interactive) ends with warnings on two router faults | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||||
|
|
||||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||||
|
|||||||
@@ -38,9 +38,9 @@
|
|||||||
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
||||||
| 20 | A model timeout per role | added | S | built |
|
| 20 | A model timeout per role | added | S | built |
|
||||||
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | built; unrun |
|
||||||
| 22 | Validation reports every gap in one pass | added | S | instructions changed; unrun |
|
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
|
||||||
| 23 | Validation recorded a few verdicts at a time | added | S | built; unrun |
|
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
|
||||||
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; unrun in an Execution |
|
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
|
||||||
|
|
||||||
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
||||||
|
|
||||||
|
|||||||
@@ -8,6 +8,6 @@ Built on 2026-10-08.
|
|||||||
4. The validator's model timeout is 300 seconds (`models.timeout_seconds` in `providers/gcp.yaml`); it had the default 180.
|
4. The validator's model timeout is 300 seconds (`models.timeout_seconds` in `providers/gcp.yaml`); it had the default 180.
|
||||||
5. `agents/validator.md` tells it to record as it goes, one Element's Checks at a time, and to submit with no attachments.
|
5. `agents/validator.md` tells it to record as it goes, one Element's Checks at a time, and to submit with no attachments.
|
||||||
|
|
||||||
**Unrun:** no evaluation since.
|
Run on 2026-10-08 (case 02): no model timeout. The validator used all 15 of its tool calls on its first attempt, having recorded verdicts; the second attempt read them and submitted in 68 seconds. Its budget is now 30 (`agents/validator.md`).
|
||||||
|
|
||||||
Tests: `tests/execution/test_validation_in_pieces.py`, "verdicts a few at a time" (7).
|
Tests: `tests/execution/test_validation_in_pieces.py`, "verdicts a few at a time" (7).
|
||||||
|
|||||||
@@ -86,7 +86,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
|||||||
| 9 | In-run context compaction | J10 | planned |
|
| 9 | In-run context compaction | J10 | planned |
|
||||||
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
| 10 | Evaluation rubric and lesson triage | J8 | planned |
|
||||||
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
| 11 | ADR: Jev is the Orchestrator's decision point | none | complete: ADR-0007 accepted 2026-10-07 |
|
||||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation |
|
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built for messages inside an Execution and validation outcomes; unverified against live Jev and by evaluation; case 04 on 2026-10-08 found two faults: a reply that echoes the question is read as a change request, and the router loops at the Revision limit (`../iter-06-target-architecture-graph/steps/step-11.md`) |
|
||||||
| 13 | Cloud-read call guard | J7 | planned |
|
| 13 | Cloud-read call guard | J7 | planned |
|
||||||
| 14 | Input guardrail | J11 | planned |
|
| 14 | Input guardrail | J11 | planned |
|
||||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
**Iteration:** 6, the Target architecture graph
|
**Iteration:** 6, the Target architecture graph
|
||||||
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
|
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
|
||||||
**Status:** steps 1 to 10 built and covered by the deterministic suite; cases 01 and 02 completed at 100% on 2026-10-07 (`steps/step-11.md`), case 02 after two Revisions of the Target architecture; cases 03 and 04 have not run on this build
|
**Status:** steps 1 to 10 built; cases 01, 02 and 03 completed at 100% with real models on 2026-10-07 and 08 (`steps/step-11.md`); case 04, the interactive one, does not complete
|
||||||
**This directory covers:** steps 1 to 11
|
**This directory covers:** steps 1 to 11
|
||||||
|
|
||||||
## Order of work
|
## Order of work
|
||||||
@@ -56,7 +56,7 @@ flowchart TD
|
|||||||
| 8 | Deployment guidance implements the graph | M | complete |
|
| 8 | Deployment guidance implements the graph | M | complete |
|
||||||
| 9 | Validation gives a verdict per Check | M | complete |
|
| 9 | Validation gives a verdict per Check | M | complete |
|
||||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||||
| 11 | Evaluation: prove the loop converges | M | cases 01 and 02 pass at 100%; cases 03 and 04 and the thresholds outstanding |
|
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) ends with warnings at 75% on two router faults (`steps/step-11.md`); thresholds outstanding |
|
||||||
|
|
||||||
Size: S under a day, M a few days.
|
Size: S under a day, M a few days.
|
||||||
|
|
||||||
|
|||||||
@@ -104,3 +104,36 @@ Case 02 again on commit `5adc911`, with `JEV_COVERAGE_REVIEW=true`. Cancelled by
|
|||||||
|
|
||||||
Changed after it, not yet run: iteration 4 steps 23 and 24.
|
Changed after it, not yet run: iteration 4 steps 23 and 24.
|
||||||
|
|
||||||
|
## Run of 2026-10-08, 08:39 UTC (`eval/results/20261008T083917Z/`)
|
||||||
|
|
||||||
|
Case 02 on commit `58e778c`, coverage review off. Completed, 100% (11 of 11 checks), no warnings, no Revision: about 24 minutes, an estimated $3.52.
|
||||||
|
|
||||||
|
- The graph had 21 Elements, 22 Relationships and 102 Checks, against 54 in the run of 19:07 the day before. The requirements and recommendations now list their conditions, and the target architect wrote a Check for each. Validation passed all 102 and proposed nothing.
|
||||||
|
- No Check was unimplementable, and the schema review raised nothing at submission.
|
||||||
|
- No model timeout. The validator exhausted its 15 tool calls on its first attempt with verdicts recorded; its second attempt carried on from them and submitted in 68 seconds. Its budget was raised to 30 after the run.
|
||||||
|
- The target architect took 388 seconds and 29 requests (21 `graph_put`, 15 schema lookups): now the slowest and most expensive step.
|
||||||
|
|
||||||
|
Case 02 across its three runs: 30 minutes with 2 Revisions; cancelled at 66 minutes; 24 minutes with none. One clean run does not show the variance is gone.
|
||||||
|
|
||||||
|
Acceptance criteria: cases 01 and 02 complete at 1.0, case 02 now with no Revision. Cases 03 and 04 were started after this run.
|
||||||
|
|
||||||
|
## Run of 2026-10-08, 10:33 UTC (`eval/results/20261008T103338Z/`)
|
||||||
|
|
||||||
|
Cases 03 and 04, on `58e778c` with the validator's budget raised to 30 (uncommitted).
|
||||||
|
|
||||||
|
**Case 03 (internal web app, pre-approved): completed, 100%, no warnings.** 40 minutes (25 to the first validation), 5.21 million tokens, an estimated $5.10. The code passed every Check in all three validations. The Target architecture was revised twice for proposed additions: region choices for a carbon-free-energy requirement, then a role grant for database access. The description was kept on the first Revision and rewritten on the second, which added an Element. No timeout, no `terraform validate` failure, no unimplementable Check. The validator was the most expensive role (31 requests, about $1.78).
|
||||||
|
|
||||||
|
**Case 04 (the same workload, interactive): completed with warnings, 75%.** 53 minutes, 8.72 million tokens, an estimated $8.11. It stopped at the limit of 100 Orchestrator turns. Three checks failed: completed, the guide (never packaged), and `specialists_ask_before_submitting` for validation results.
|
||||||
|
|
||||||
|
What went wrong in case 04, in order:
|
||||||
|
|
||||||
|
1. **A reply to a Specialist's question was read as a change request.** The scripted user replies with `Q: <the question> A: <the answer>`. The router gives that whole text to Jev as the user's message. Twice, a describer's question asking the user to confirm its draft, echoed back, was judged a request to change the Target architecture (asks_change 0.76 to 0.78), and the design was reopened with the question as the reason. With one real unimplementable Check, that used all three Revisions.
|
||||||
|
2. **At the Revision limit the router loops.** Validation then proposed an addition; reopening was refused at the limit; the router asked the user; the reply was again read as a change request for the same Deliverable; the reopen was refused again. 40 escalations in two seconds, each question nesting the last, until the turn limit. No model ran in the loop, but Jev was called each time (73 judged decisions in the run).
|
||||||
|
3. The validator submitted without confirming a draft with the user, which the interactive protocol asks of every Specialist.
|
||||||
|
|
||||||
|
What held in case 04: every Specialist's questions reached the user and their answers came back; four Approvals were granted by structured decisions; the unimplementable-Check route fired once, correctly; the code validated.
|
||||||
|
|
||||||
|
Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not complete. "At most one Revision of deployment guidance in two of three cases" holds (none in 01, 02 and 03). The thresholds remain unset until case 04 passes.
|
||||||
|
|
||||||
|
Not yet fixed: the two router faults above.
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
role: validator
|
role: validator
|
||||||
deliverable: validation_results
|
deliverable: validation_results
|
||||||
tools: [terraform_validate, terraform_plan, terraform_schema]
|
tools: [terraform_validate, terraform_plan, terraform_schema]
|
||||||
|
budget: 30
|
||||||
includes: [specialist-protocol]
|
includes: [specialist-protocol]
|
||||||
---
|
---
|
||||||
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
|
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
|
||||||
|
|||||||
@@ -36,7 +36,9 @@ async def test_assembly_wires_roles_budgets_and_tools_from_settings(tmp_path):
|
|||||||
assert interaction.prompt["questions"][0]["question"] == "Which region?"
|
assert interaction.prompt["questions"][0]["question"] == "Which region?"
|
||||||
assert assembly.bundle.name == "gcp"
|
assert assembly.bundle.name == "gcp"
|
||||||
assert assembly.roles["orchestrator"].budget == 9
|
assert assembly.roles["orchestrator"].budget == 9
|
||||||
assert assembly.roles["validator"].budget == 4
|
# A role with no budget of its own takes the setting; the validator has one.
|
||||||
|
assert assembly.roles["decomposer"].budget == 4
|
||||||
|
assert assembly.roles["validator"].budget == 30
|
||||||
assert assembly.mcp_gaps == {}
|
assert assembly.mcp_gaps == {}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user