Files
gcp_solution_architecture_a…/.agents/plans/iter-04-learning-and-skills/steps/step-8.md
Jonathan Boniface e9246828b3 feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:59:09 +01:00

3.3 KiB

Step 8 — Implementation Detail

Built, with two acceptance criteria still open (listed at the end).

  1. Structural checks added to every case (eval/scoring.py):
    • components_trace_to_requirements: every row of the decomposition's component table names a requirement.
    • no_blocking_findings: the final findings.json holds no critical or high finding.
  2. Judged checks, run with --judge:
    • pillar_checklists_covered: each item of each pillar Skill's "Validation checklist" is put to the judge as a yes-or-no question against the design recommendations. The weakest pillar must reach 70%.
    • recommendations_quality: four graded questions (specific, actionable, trade-offs, traceable); the mean must reach 0.67.
    • checklist_items reads the checklists from the vendored Skills, nested bullets included, so a vendor sync updates the questions.
  3. Judge seam (eval/judge.py): a Judge takes one document and a map of typed questions and returns a score from 0 to 1 each. GeminiJudge uses the bundle's default model, or EVAL_JUDGE_MODEL. The shape matches a decision model's, for iteration 5 step 10. One live call confirmed it works.
  4. Interactive case. A case marked "interactive": true runs with a scripted user: run_with_scripted_user grants Approvals, denies cloud reads and answers questions by pattern. It adds the check specialists_ask_before_submitting. case-04-interactive-web-app is case 03 run this way.
  5. Thresholds. A case may set thresholds for revisions, seconds and tokens. Exceeding one is printed as REGRESSION, listed in the report, and makes the exit code non-zero. It does not change the score.

automatic_reply in app/nodes/execution_nodes.py lost its leading underscore so the evaluation can reuse it.

Tests: tests/eval/test_scoring.py, with a fake judge.

Open:

  • Thresholds are not set. They need a baseline per case from a completed run of the current code. None exists: see below.
  • No judged or interactive run yet.
  • Judge stability across repeated runs on the same output is unmeasured (iteration 5 step 10 names it).

The evaluation run of 2026-10-06 (stopped, no score)

A three-case run was started on the code as of steps 1 to 6 and stopped by the owner after 35 minutes, during case 01. Cases 02 and 03 never started. Nothing was scored and no files were written, so it gives no baseline. What its log showed:

  • Seven Deliverables were submitted on their first attempt in about 15 minutes, including the decomposition and description under their new contracts (step 4).
  • The Orchestrator reopened deployment guidance after each failed validation without first asking for Approval of the validation Phase, which is what step 6 intended. Observed three times in one case.
  • No Revision of the design recommendations occurred, so step 3 was not exercised.
  • Validation did not converge: three passes gave 4, then 7, then 5 different findings, and deployment guidance used all three Revisions. The four findings of the first pass were all still present after the first Revision.
  • The Orchestrator stopped without calling a tool four times and was nudged; one such turn took 234 seconds.

The non-convergence led to ADR-0005 (proposed): establish the Target architecture as an explicit graph with numbered Checks before code is written.