Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
19 KiB
Step 11 — Implementation Detail
Partly done. The checks are built; the evaluation run that this step exists for has not been made.
Built in eval/scoring.py:
requirements_chain_complete: the delivered data files form a chain with no gap.every_check_passes: validation ended with no Check failed or not verifiable, and the dry-run not failed.diagram_matches_graph: the delivered diagram is exactly the rendering of the delivered graph.
These replaced components_trace_to_requirements and no_blocking_findings from iteration 4 step 8.
Outstanding, and why:
- No evaluation run. The owner stopped the last run on 2026-10-06 and asked that no further cases start until they advise. Nothing in this iteration has run against a real model.
- The acceptance criteria are therefore unmet: all cases scoring at least 0.8, the failing-Check set never growing between passes, and at most one Revision of the deployment guidance in two of three cases.
- Not built: the per-case record of validation passes and the Check ids failing at each. The evaluation reads only the final verdicts today.
- Thresholds from iteration 4 step 8 are still unset.
Do not mark this step complete until a run meets the criteria. (legacy-baseline was removed on 2026-10-07 on the owner's instruction, ahead of this.)
Evaluation run of 2026-10-07, case 01 (commit e924682)
Report: eval/results/20261007T125917Z/report.json. Result: completed_with_warnings, score 73% (8 of 11 checks), 3,627 seconds, 4.25 million tokens, 144 model requests. No solution guide was written.
| Stopped run, 2026-10-06 | This run | |
|---|---|---|
| First pass to a validation | about 17 minutes | 25 minutes |
| Validation passes | 3 | 2 |
| Failing items per pass | 4, then 7, then 5 findings | 0 failed Checks and 7 proposed additions, then 1 failed Check, 1 not verifiable and 1 proposed addition |
| Revisions | deployment guidance × 3 | Target architecture × 1, deployment guidance × 3 |
| Outcome | stopped at 35 minutes | ended with warnings at 60 minutes |
What held:
- Every chain link was accepted on a first attempt except the Target architecture, which was refused once per attempt (7 requirements with no Check; then 4 Elements with no Check) and fixed inside its own run.
requirements_chain_completeanddiagram_matches_graphpassed.- The Terraform met every Check on its first attempt (0 failed of the first graph's Checks).
- The Orchestrator followed the new routing: additions reopened the Target architecture; a failed Check reopened the code.
- After one revision of the graph the validator proposed 1 addition, down from 7.
What failed, in order of cause:
- A bug in this iteration (fixed after the run, with a test). Once the deployment guidance was reopened, the Orchestrator's context still carried the old validation's verdicts. It reopened the code for K-30 three times in a row without a validation in between, reaching the Revision limit.
_orchestrator_contextnow shows a validation only while the validation results stand as submitted. - A Check that cannot be implemented had nowhere to go. K-30 required
startup_cpu_booston a Cloud Run job. Terraform rejects that argument there, so the IaC author could not satisfy it, and the validator marked it failed, which sends it back to the code. The fault was in the graph. The IaC author used 57 requests and 2.35 million tokens, 55% of the run. - The target architect put a recommendation on an Element it does not apply to. K-30 came from DR-19, proposed by the validator in pass 1 for two services and applied to the job as well.
- Model timeouts. The target architect's model request timed out at 180 seconds in both attempts; each cost about 3 minutes.
Not yet exercised: the second pass also had a Check marked not verifiable (K-55) and a proposed addition (NFR-15). The failed-Check route took precedence, as instructed, so the gap route was not taken a second time.
Acceptance criteria: not met. The case did not complete or reach 0.8.
Changes this points to, not yet made:
- Let the IaC author report a Check it cannot implement, with the tool's error as evidence, so that it is routed to the Target architecture and not retried as code.
- Have the validator distinguish "the code does not do this" from "this cannot be done to this resource".
- Check a proposed addition against the Element it names before it becomes a Check.
- Raise or tune the model timeout for the target architect.
Run of 2026-10-07, 16:59 UTC (eval/results/20261007T165920Z/)
Case 01 again, after iteration 4 steps 11, 13 and 17 to 20 and the router of iteration 5. Not judged.
| Run of 12:59 | This run | |
|---|---|---|
| Status | completed with warnings | completed |
| Score | 73% | 100% (11 of 11 checks) |
| Seconds | 3,627 | 1,091 |
| Tokens | 4.25 million | 2.38 million |
| Revisions | Revision limit reached | none |
| Model timeouts | target architect, both attempts | none |
terraform validate failures |
13 | 0 |
| IaC author | 57 requests, 2.35 million tokens | 11 requests, 271 thousand tokens |
| Target architect | timed out twice | 55 requests, 1.15 million tokens |
The graph had 13 Elements, 17 Relationships and 42 Checks. The code met all 42 on the first validation; no Check was unimplementable or not verifiable, and nothing was proposed. The Orchestrator was the router: 11 turns, no model chose an action, and no decision needed Jev.
What went wrong: the target architect exhausted its 30 tool calls on its first attempt. It made 39 terraform_schema calls for a graph with 8 distinct resource types, one per setting, as its instructions told it to. Its second attempt started from the working graph and submitted in 92 seconds. Changed after the run: it is told to look up each distinct resource type once, find takes several names in one call, and its budget is 45. That change has not been run.
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.
Run of 2026-10-07, 19:07 UTC (eval/results/20261007T190702Z/)
Case 02, on commit f1f6f1b. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.
- The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
- The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
- The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
- The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
- By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.
Changed after the run, not yet run: iteration 4 steps 21 and 22.
Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.
Run of 2026-10-07, 19:58 UTC, cancelled (.cache/eval-logs/; no report)
Case 02 again on commit 5adc911, with JEV_COVERAGE_REVIEW=true. Cancelled by the owner at 66 minutes, during a fifth validation. No score and no token count were written.
- The Target architecture was reopened 3 times (the limit): twice for a Check the code's author reported as unimplementable, once for additions validation proposed. The code was reopened twice for failed Checks. Target architect 5 runs, IaC author 6, validator 5.
- 6 model timeouts, 5 of them the validator writing all its verdicts in one reply: about 15 minutes.
- The coverage review raised 5, 10, 11 and 9 of 61 conditions over four submissions. Validation still proposed a TLS policy and an access grant afterwards. The agents had written conditions no infrastructure code can prove, such as an availability percentage, and Jev answered "none" to those. It stays off.
- What held: the description was kept on two Revisions (iteration 4 step 21); the unimplementable-Check route sent the design back before any validation was spent (step 19); code fixes took 1 to 5 minutes from the working copy; no
terraform validatefailure.
Changed after it, not yet run: iteration 4 steps 23 and 24.
Run of 2026-10-08, 08:39 UTC (eval/results/20261008T083917Z/)
Case 02 on commit 58e778c, coverage review off. Completed, 100% (11 of 11 checks), no warnings, no Revision: about 24 minutes, an estimated $3.52.
- The graph had 21 Elements, 22 Relationships and 102 Checks, against 54 in the run of 19:07 the day before. The requirements and recommendations now list their conditions, and the target architect wrote a Check for each. Validation passed all 102 and proposed nothing.
- No Check was unimplementable, and the schema review raised nothing at submission.
- No model timeout. The validator exhausted its 15 tool calls on its first attempt with verdicts recorded; its second attempt carried on from them and submitted in 68 seconds. Its budget was raised to 30 after the run.
- The target architect took 388 seconds and 29 requests (21
graph_put, 15 schema lookups): now the slowest and most expensive step.
Case 02 across its three runs: 30 minutes with 2 Revisions; cancelled at 66 minutes; 24 minutes with none. One clean run does not show the variance is gone.
Acceptance criteria: cases 01 and 02 complete at 1.0, case 02 now with no Revision. Cases 03 and 04 were started after this run.
Run of 2026-10-08, 10:33 UTC (eval/results/20261008T103338Z/)
Cases 03 and 04, on 58e778c with the validator's budget raised to 30 (uncommitted).
Case 03 (internal web app, pre-approved): completed, 100%, no warnings. 40 minutes (25 to the first validation), 5.21 million tokens, an estimated $5.10. The code passed every Check in all three validations. The Target architecture was revised twice for proposed additions: region choices for a carbon-free-energy requirement, then a role grant for database access. The description was kept on the first Revision and rewritten on the second, which added an Element. No timeout, no terraform validate failure, no unimplementable Check. The validator was the most expensive role (31 requests, about $1.78).
Case 04 (the same workload, interactive): completed with warnings, 75%. 53 minutes, 8.72 million tokens, an estimated $8.11. It stopped at the limit of 100 Orchestrator turns. Three checks failed: completed, the guide (never packaged), and specialists_ask_before_submitting for validation results.
What went wrong in case 04, in order:
- A reply to a Specialist's question was read as a change request. The scripted user replies with
Q: <the question> A: <the answer>. The router gives that whole text to Jev as the user's message. Twice, a describer's question asking the user to confirm its draft, echoed back, was judged a request to change the Target architecture (asks_change 0.76 to 0.78), and the design was reopened with the question as the reason. With one real unimplementable Check, that used all three Revisions. - At the Revision limit the router loops. Validation then proposed an addition; reopening was refused at the limit; the router asked the user; the reply was again read as a change request for the same Deliverable; the reopen was refused again. 40 escalations in two seconds, each question nesting the last, until the turn limit. No model ran in the loop, but Jev was called each time (73 judged decisions in the run).
- The validator submitted without confirming a draft with the user, which the interactive protocol asks of every Specialist.
What held in case 04: every Specialist's questions reached the user and their answers came back; four Approvals were granted by structured decisions; the unimplementable-Check route fired once, correctly; the code validated.
Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not complete. "At most one Revision of deployment guidance in two of three cases" holds (none in 01, 02 and 03). The thresholds remain unset until case 04 passes.
Not yet fixed: the two router faults above.
Run of 2026-10-08, 12:22 UTC, unfinished (.cache/eval-logs/case-04-20261008T122200Z.log; no report)
Case 04 again, with the three router fixes of iteration 5 step 12 (uncommitted at the time). The process stopped at about 14:48 local time, 86 minutes in, when the session that started it ended. It was not cancelled and wrote no score, report or token count.
What the fixes did:
- No reopen came from a question quoted back in a reply.
- At 14:41
requirementsreached its Revision limit. The user was told once and the Execution carried on; there was no loop. - The schema review raised items at submission twice (1, then 3), its first use in a real Execution.
- The unimplementable-Check route fired once and the design was revised in 92 seconds; the description was kept.
What went wrong:
- A scripted answer was read as a new requirement, three times. The scripted user answers by matching words in the question against patterns written for the requirements analyst. Asked by the validator, and later the design advisor and the IaC author, to confirm a draft, it replied with requirement statements ("GDPR applies to employee names and emails...", "99.5% during business hours...", "PostgreSQL. Under 20 GB..."). Jev read each as a request to change
requirements, and the router reopened it: twice, then the limit. Each reopen returned every later Deliverable to draft, so the Execution was redone from the start twice. - Validation failed one Check twice (an alert condition; a log bucket not locked). Each was sent back to the code, which is the intended route.
The cause of the first is mostly the harness: no person answers "please confirm my draft" with a requirement. It also shows a weakness in the router: a reply that restates what a submitted Deliverable already says is treated as a request to change it.
Not yet changed:
- Scripted answers should apply only to the Specialist they were written for, and a draft-confirmation question should get an approval.
- A reply that restates what a submitted Deliverable already says should be read as an answer.
Acceptance criteria: unchanged. Cases 01, 02 and 03 complete at 1.0; case 04 has one completed run at 0.75 and this unfinished one.
Changes to the evaluation after the unfinished run (2026-10-08)
- A scripted answer may name who it is for:
[pattern, answer, asker]. Case 04's fourteen facts are tagged forrequirements; any other Specialist, and the Orchestrator, gets "Approved. Please proceed." (automatic_reply,eval/datasets/solution_cases.json). - The evaluation writes
<case>-progress.jsonin the run folder after every agent run: seconds, runs, outcomes, tokens and the timeline so far. A run that is cut off leaves it behind with"finished": false.
Runs of 2026-10-08, 14:25 UTC: all four cases, twice (eval/results/20261008T142509Z/ and ...142514Z/)
Two batches run side by side on the build with the case 04 changes, intake and the outside_code split (uncommitted at the time). Batch B added the judged checks. All eight runs completed; none ended with warnings.
| Case | Batch A | Batch B (judged) |
|---|---|---|
| 01 | 1.00, 31 min, 3.42 M tokens, 1 Revision | 0.92, 39 min, 4.06 M, 1 Revision |
| 02 | 1.00, 47 min, 7.02 M, 3 Revisions | 0.92, 24 min, 3.12 M, none |
| 03 | 1.00, 41 min, 6.33 M, 2 Revisions | 0.92, 29 min, 4.06 M, 1 Revision |
| 04 | 0.92, 45 min, 5.88 M, 1 Revision | 0.93, 33 min, 3.63 M, 1 Revision |
Estimated cost: $22.16 for batch A and $16.40 for batch B. Running two batches at once shares the machine and the model quota, so these times are longer than a case run alone.
- Case 04 completes. Both runs went through all five Phases with every draft confirmed by the user and no reply misread as a change. One Revision each: the code for a failed Check in batch B, the design for a proposed addition in batch A.
- Every structural check passed in all eight runs, with no model timeout, no loop and no
terraform validatefailure. pillar_checklists_coveredfailed in all four judged runs, at 0% to 55% a pillar, whilerecommendations_qualitypassed at 0.83 to 0.90. The cause is structural: 78 checklist items against 11 to 14 recommendations. Iteration 4 step 26 was built in answer; no judged run has happened since.specialists_ask_before_submittingfailed once, in batch A's case 04: the target architect exhausted its 45 tool calls, and on its second attempt submitted without asking. The first submission with a user present is now refused until the draft has been put to them (CONFIRM_DRAFTS, on by default).- The same case varies widely. Case 02 took 24 minutes with no Revision in one batch and 47 minutes with three in the other, at the same time on the same build. Revisions of the Target architecture for additions validation proposes remain the main cost.
Thresholds
Set in eval/datasets/solution_cases.json from every completed run of each case on this build, at about a quarter above the worst seen:
| Case | Revisions | Seconds | Tokens |
|---|---|---|---|
| 01 | 2 | 3,000 | 5.0 M |
| 02 | 3 | 3,600 | 8.5 M |
| 03 | 3 | 3,200 | 8.0 M |
| 04 | 2 | 3,400 | 7.5 M |
They are alarms for a run that is clearly worse than any seen, not targets. With two to four runs a case and this much spread they are loose; tighten them as runs accumulate.
Acceptance criteria: met. All four cases complete and score at least 0.8; deployment guidance needed at most one Revision in every run; the thresholds are set. One criterion is met only loosely: the set of gaps did not always shrink from one validation to the next (case 02, 2026-10-07).
Run of 2026-10-08, 17:20 UTC (eval/results/20261008T172049Z/): case 04, judged
To verify two changes made after the batches began: the pillar checklist review (iteration 4 step 26) and the confirm-first rule (step 27). Completed, 86% (12 of 14 checks), no Revision, 36 minutes, 4.10 million tokens.
pillar_checklists_coveredpasses: 100% on all six pillars, against 9% to 45% in batch B's case 04. The review is in the Deliverable.recommendations_qualitypassed at 0.80 (specific 0.84, actionable 0.81, trade-offs 0.90, traceable 0.66).specialists_ask_before_submittingfailed again, for the Target architecture. The rule opened once the user had answered any question about the Deliverable; the target architect ran out of tool calls, the Orchestrator asked the user what to do, and that answer opened it. Fixed after the run: the rule now opens only when the user has been shown a draft of that Deliverable (drafts_shown). Not rerun.requirements_name_no_productsfailed for the first time in any run: the requirements named Secret Manager. This is what the requirements review of iteration 5 step 4 looks for; it is off by default and was off here.