Files
gcp_solution_architecture_a…/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md
Jonathan Boniface 4e7a16cedd chore: raise the validator's budget and record the runs of cases 02, 03 and 04
- Validator budget 15 to 30: recording verdicts in pieces for 100 Checks
  used all 15 calls in one attempt.
- Plans record three runs on 2026-10-08:
  case 02 completed at 100% with no Revision in 24 minutes;
  case 03 completed at 100% with two Revisions of the design in 40 minutes;
  case 04 (interactive) ended with warnings at 75%, on two router faults:
  a reply that echoes the question is read as a change request, and the
  router loops at the Revision limit. Neither is fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:16:36 +01:00

12 KiB
Raw Blame History

Step 11 — Implementation Detail

Partly done. The checks are built; the evaluation run that this step exists for has not been made.

Built in eval/scoring.py:

  • requirements_chain_complete: the delivered data files form a chain with no gap.
  • every_check_passes: validation ended with no Check failed or not verifiable, and the dry-run not failed.
  • diagram_matches_graph: the delivered diagram is exactly the rendering of the delivered graph.

These replaced components_trace_to_requirements and no_blocking_findings from iteration 4 step 8.

Outstanding, and why:

  • No evaluation run. The owner stopped the last run on 2026-10-06 and asked that no further cases start until they advise. Nothing in this iteration has run against a real model.
  • The acceptance criteria are therefore unmet: all cases scoring at least 0.8, the failing-Check set never growing between passes, and at most one Revision of the deployment guidance in two of three cases.
  • Not built: the per-case record of validation passes and the Check ids failing at each. The evaluation reads only the final verdicts today.
  • Thresholds from iteration 4 step 8 are still unset.

Do not mark this step complete until a run meets the criteria. (legacy-baseline was removed on 2026-10-07 on the owner's instruction, ahead of this.)

Evaluation run of 2026-10-07, case 01 (commit e924682)

Report: eval/results/20261007T125917Z/report.json. Result: completed_with_warnings, score 73% (8 of 11 checks), 3,627 seconds, 4.25 million tokens, 144 model requests. No solution guide was written.

Stopped run, 2026-10-06 This run
First pass to a validation about 17 minutes 25 minutes
Validation passes 3 2
Failing items per pass 4, then 7, then 5 findings 0 failed Checks and 7 proposed additions, then 1 failed Check, 1 not verifiable and 1 proposed addition
Revisions deployment guidance × 3 Target architecture × 1, deployment guidance × 3
Outcome stopped at 35 minutes ended with warnings at 60 minutes

What held:

  • Every chain link was accepted on a first attempt except the Target architecture, which was refused once per attempt (7 requirements with no Check; then 4 Elements with no Check) and fixed inside its own run.
  • requirements_chain_complete and diagram_matches_graph passed.
  • The Terraform met every Check on its first attempt (0 failed of the first graph's Checks).
  • The Orchestrator followed the new routing: additions reopened the Target architecture; a failed Check reopened the code.
  • After one revision of the graph the validator proposed 1 addition, down from 7.

What failed, in order of cause:

  1. A bug in this iteration (fixed after the run, with a test). Once the deployment guidance was reopened, the Orchestrator's context still carried the old validation's verdicts. It reopened the code for K-30 three times in a row without a validation in between, reaching the Revision limit. _orchestrator_context now shows a validation only while the validation results stand as submitted.
  2. A Check that cannot be implemented had nowhere to go. K-30 required startup_cpu_boost on a Cloud Run job. Terraform rejects that argument there, so the IaC author could not satisfy it, and the validator marked it failed, which sends it back to the code. The fault was in the graph. The IaC author used 57 requests and 2.35 million tokens, 55% of the run.
  3. The target architect put a recommendation on an Element it does not apply to. K-30 came from DR-19, proposed by the validator in pass 1 for two services and applied to the job as well.
  4. Model timeouts. The target architect's model request timed out at 180 seconds in both attempts; each cost about 3 minutes.

Not yet exercised: the second pass also had a Check marked not verifiable (K-55) and a proposed addition (NFR-15). The failed-Check route took precedence, as instructed, so the gap route was not taken a second time.

Acceptance criteria: not met. The case did not complete or reach 0.8.

Changes this points to, not yet made:

  • Let the IaC author report a Check it cannot implement, with the tool's error as evidence, so that it is routed to the Target architecture and not retried as code.
  • Have the validator distinguish "the code does not do this" from "this cannot be done to this resource".
  • Check a proposed addition against the Element it names before it becomes a Check.
  • Raise or tune the model timeout for the target architect.

Run of 2026-10-07, 16:59 UTC (eval/results/20261007T165920Z/)

Case 01 again, after iteration 4 steps 11, 13 and 17 to 20 and the router of iteration 5. Not judged.

Run of 12:59 This run
Status completed with warnings completed
Score 73% 100% (11 of 11 checks)
Seconds 3,627 1,091
Tokens 4.25 million 2.38 million
Revisions Revision limit reached none
Model timeouts target architect, both attempts none
terraform validate failures 13 0
IaC author 57 requests, 2.35 million tokens 11 requests, 271 thousand tokens
Target architect timed out twice 55 requests, 1.15 million tokens

The graph had 13 Elements, 17 Relationships and 42 Checks. The code met all 42 on the first validation; no Check was unimplementable or not verifiable, and nothing was proposed. The Orchestrator was the router: 11 turns, no model chose an action, and no decision needed Jev.

What went wrong: the target architect exhausted its 30 tool calls on its first attempt. It made 39 terraform_schema calls for a graph with 8 distinct resource types, one per setting, as its instructions told it to. Its second attempt started from the working graph and submitted in 92 seconds. Changed after the run: it is told to look up each distinct resource type once, find takes several names in one call, and its budget is 45. That change has not been run.

Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.

Run of 2026-10-07, 19:07 UTC (eval/results/20261007T190702Z/)

Case 02, on commit f1f6f1b. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.

  • The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
  • The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
  • The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
  • The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
  • By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.

Changed after the run, not yet run: iteration 4 steps 21 and 22.

Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.

Run of 2026-10-07, 19:58 UTC, cancelled (.cache/eval-logs/; no report)

Case 02 again on commit 5adc911, with JEV_COVERAGE_REVIEW=true. Cancelled by the owner at 66 minutes, during a fifth validation. No score and no token count were written.

  • The Target architecture was reopened 3 times (the limit): twice for a Check the code's author reported as unimplementable, once for additions validation proposed. The code was reopened twice for failed Checks. Target architect 5 runs, IaC author 6, validator 5.
  • 6 model timeouts, 5 of them the validator writing all its verdicts in one reply: about 15 minutes.
  • The coverage review raised 5, 10, 11 and 9 of 61 conditions over four submissions. Validation still proposed a TLS policy and an access grant afterwards. The agents had written conditions no infrastructure code can prove, such as an availability percentage, and Jev answered "none" to those. It stays off.
  • What held: the description was kept on two Revisions (iteration 4 step 21); the unimplementable-Check route sent the design back before any validation was spent (step 19); code fixes took 1 to 5 minutes from the working copy; no terraform validate failure.

Changed after it, not yet run: iteration 4 steps 23 and 24.

Run of 2026-10-08, 08:39 UTC (eval/results/20261008T083917Z/)

Case 02 on commit 58e778c, coverage review off. Completed, 100% (11 of 11 checks), no warnings, no Revision: about 24 minutes, an estimated $3.52.

  • The graph had 21 Elements, 22 Relationships and 102 Checks, against 54 in the run of 19:07 the day before. The requirements and recommendations now list their conditions, and the target architect wrote a Check for each. Validation passed all 102 and proposed nothing.
  • No Check was unimplementable, and the schema review raised nothing at submission.
  • No model timeout. The validator exhausted its 15 tool calls on its first attempt with verdicts recorded; its second attempt carried on from them and submitted in 68 seconds. Its budget was raised to 30 after the run.
  • The target architect took 388 seconds and 29 requests (21 graph_put, 15 schema lookups): now the slowest and most expensive step.

Case 02 across its three runs: 30 minutes with 2 Revisions; cancelled at 66 minutes; 24 minutes with none. One clean run does not show the variance is gone.

Acceptance criteria: cases 01 and 02 complete at 1.0, case 02 now with no Revision. Cases 03 and 04 were started after this run.

Run of 2026-10-08, 10:33 UTC (eval/results/20261008T103338Z/)

Cases 03 and 04, on 58e778c with the validator's budget raised to 30 (uncommitted).

Case 03 (internal web app, pre-approved): completed, 100%, no warnings. 40 minutes (25 to the first validation), 5.21 million tokens, an estimated $5.10. The code passed every Check in all three validations. The Target architecture was revised twice for proposed additions: region choices for a carbon-free-energy requirement, then a role grant for database access. The description was kept on the first Revision and rewritten on the second, which added an Element. No timeout, no terraform validate failure, no unimplementable Check. The validator was the most expensive role (31 requests, about $1.78).

Case 04 (the same workload, interactive): completed with warnings, 75%. 53 minutes, 8.72 million tokens, an estimated $8.11. It stopped at the limit of 100 Orchestrator turns. Three checks failed: completed, the guide (never packaged), and specialists_ask_before_submitting for validation results.

What went wrong in case 04, in order:

  1. A reply to a Specialist's question was read as a change request. The scripted user replies with Q: <the question> A: <the answer>. The router gives that whole text to Jev as the user's message. Twice, a describer's question asking the user to confirm its draft, echoed back, was judged a request to change the Target architecture (asks_change 0.76 to 0.78), and the design was reopened with the question as the reason. With one real unimplementable Check, that used all three Revisions.
  2. At the Revision limit the router loops. Validation then proposed an addition; reopening was refused at the limit; the router asked the user; the reply was again read as a change request for the same Deliverable; the reopen was refused again. 40 escalations in two seconds, each question nesting the last, until the turn limit. No model ran in the loop, but Jev was called each time (73 judged decisions in the run).
  3. The validator submitted without confirming a draft with the user, which the interactive protocol asks of every Specialist.

What held in case 04: every Specialist's questions reached the user and their answers came back; four Approvals were granted by structured decisions; the unimplementable-Check route fired once, correctly; the code validated.

Acceptance criteria: cases 01, 02 and 03 complete at 1.0. Case 04 does not complete. "At most one Revision of deployment guidance in two of three cases" holds (none in 01, 02 and 03). The thresholds remain unset until case 04 passes.

Not yet fixed: the two router faults above.