Files
gcp_solution_architecture_a…/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md
Jonathan Boniface 5adc911a82 feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:58:38 +01:00

7.2 KiB
Raw Blame History

Step 11 — Implementation Detail

Partly done. The checks are built; the evaluation run that this step exists for has not been made.

Built in eval/scoring.py:

  • requirements_chain_complete: the delivered data files form a chain with no gap.
  • every_check_passes: validation ended with no Check failed or not verifiable, and the dry-run not failed.
  • diagram_matches_graph: the delivered diagram is exactly the rendering of the delivered graph.

These replaced components_trace_to_requirements and no_blocking_findings from iteration 4 step 8.

Outstanding, and why:

  • No evaluation run. The owner stopped the last run on 2026-10-06 and asked that no further cases start until they advise. Nothing in this iteration has run against a real model.
  • The acceptance criteria are therefore unmet: all cases scoring at least 0.8, the failing-Check set never growing between passes, and at most one Revision of the deployment guidance in two of three cases.
  • Not built: the per-case record of validation passes and the Check ids failing at each. The evaluation reads only the final verdicts today.
  • Thresholds from iteration 4 step 8 are still unset.

Do not mark this step complete until a run meets the criteria. (legacy-baseline was removed on 2026-10-07 on the owner's instruction, ahead of this.)

Evaluation run of 2026-10-07, case 01 (commit e924682)

Report: eval/results/20261007T125917Z/report.json. Result: completed_with_warnings, score 73% (8 of 11 checks), 3,627 seconds, 4.25 million tokens, 144 model requests. No solution guide was written.

Stopped run, 2026-10-06 This run
First pass to a validation about 17 minutes 25 minutes
Validation passes 3 2
Failing items per pass 4, then 7, then 5 findings 0 failed Checks and 7 proposed additions, then 1 failed Check, 1 not verifiable and 1 proposed addition
Revisions deployment guidance × 3 Target architecture × 1, deployment guidance × 3
Outcome stopped at 35 minutes ended with warnings at 60 minutes

What held:

  • Every chain link was accepted on a first attempt except the Target architecture, which was refused once per attempt (7 requirements with no Check; then 4 Elements with no Check) and fixed inside its own run.
  • requirements_chain_complete and diagram_matches_graph passed.
  • The Terraform met every Check on its first attempt (0 failed of the first graph's Checks).
  • The Orchestrator followed the new routing: additions reopened the Target architecture; a failed Check reopened the code.
  • After one revision of the graph the validator proposed 1 addition, down from 7.

What failed, in order of cause:

  1. A bug in this iteration (fixed after the run, with a test). Once the deployment guidance was reopened, the Orchestrator's context still carried the old validation's verdicts. It reopened the code for K-30 three times in a row without a validation in between, reaching the Revision limit. _orchestrator_context now shows a validation only while the validation results stand as submitted.
  2. A Check that cannot be implemented had nowhere to go. K-30 required startup_cpu_boost on a Cloud Run job. Terraform rejects that argument there, so the IaC author could not satisfy it, and the validator marked it failed, which sends it back to the code. The fault was in the graph. The IaC author used 57 requests and 2.35 million tokens, 55% of the run.
  3. The target architect put a recommendation on an Element it does not apply to. K-30 came from DR-19, proposed by the validator in pass 1 for two services and applied to the job as well.
  4. Model timeouts. The target architect's model request timed out at 180 seconds in both attempts; each cost about 3 minutes.

Not yet exercised: the second pass also had a Check marked not verifiable (K-55) and a proposed addition (NFR-15). The failed-Check route took precedence, as instructed, so the gap route was not taken a second time.

Acceptance criteria: not met. The case did not complete or reach 0.8.

Changes this points to, not yet made:

  • Let the IaC author report a Check it cannot implement, with the tool's error as evidence, so that it is routed to the Target architecture and not retried as code.
  • Have the validator distinguish "the code does not do this" from "this cannot be done to this resource".
  • Check a proposed addition against the Element it names before it becomes a Check.
  • Raise or tune the model timeout for the target architect.

Run of 2026-10-07, 16:59 UTC (eval/results/20261007T165920Z/)

Case 01 again, after iteration 4 steps 11, 13 and 17 to 20 and the router of iteration 5. Not judged.

Run of 12:59 This run
Status completed with warnings completed
Score 73% 100% (11 of 11 checks)
Seconds 3,627 1,091
Tokens 4.25 million 2.38 million
Revisions Revision limit reached none
Model timeouts target architect, both attempts none
terraform validate failures 13 0
IaC author 57 requests, 2.35 million tokens 11 requests, 271 thousand tokens
Target architect timed out twice 55 requests, 1.15 million tokens

The graph had 13 Elements, 17 Relationships and 42 Checks. The code met all 42 on the first validation; no Check was unimplementable or not verifiable, and nothing was proposed. The Orchestrator was the router: 11 turns, no model chose an action, and no decision needed Jev.

What went wrong: the target architect exhausted its 30 tool calls on its first attempt. It made 39 terraform_schema calls for a graph with 8 distinct resource types, one per setting, as its instructions told it to. Its second attempt started from the working graph and submitted in 92 seconds. Changed after the run: it is told to look up each distinct resource type once, find takes several names in one call, and its budget is 45. That change has not been run.

Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.

Run of 2026-10-07, 19:07 UTC (eval/results/20261007T190702Z/)

Case 02, on commit f1f6f1b. Completed, 100% (11 of 11 checks), no warnings: 1,781 seconds and 4.78 million tokens; 904 seconds to the first validation.

  • The graph ended with 20 Elements, 23 Relationships and 54 Checks. The code passed every Check in all three validations; no Check failed, was not verifiable or unimplementable.
  • The Target architecture was revised twice, both for proposed additions: 1 after the first validation, 3 after the second, 0 after the third. The set of gaps did not shrink between the first two passes, so the loop converged without the criterion "never grows" holding.
  • The router asked Jev whether each gap was material: 0.92 and 0.88, both above the 0.75 threshold. This is the first use of Jev inside an Execution.
  • The target architect made 17 schema lookups in its first attempt (39 in case 01) and did not exhaust its budget.
  • By role: IaC author 56 requests and 1.32 million tokens; target architect 57 and 1.20 million; describer 24 and 942 thousand, rewriting the description on each Revision.

Changed after the run, not yet run: iteration 4 steps 21 and 22.

Acceptance criteria: cases 01 and 02 complete at 1.0. Cases 03 and 04 have not run on this build.