Design is now approved before any infrastructure code is written (ADR-0005). A new target-architect Specialist produces one graph of Elements, Relationships and Checks; the diagram is rendered from it, deployment guidance must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations carry ids in data files, and a submission with a gap in the Requirements chain is refused. Also in this commit: - Customer cases group Executions, with an intent history per case - lesson candidates are captured when an Execution ends - evaluation gains chain, verdict, judged and interactive checks, and per-case thresholds - Terraform providers are cached between checks - the requirements analyst proposes defaults instead of asking - the legacy-baseline case and the checks that only read it are removed - ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed - plans for iterations 4 to 7 under .agents/plans/ Unproven with a real model: no evaluation run has completed on this design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.3 KiB
3.3 KiB
Step 8 — Implementation Detail
Built, with two acceptance criteria still open (listed at the end).
- Structural checks added to every case (
eval/scoring.py):components_trace_to_requirements: every row of the decomposition's component table names a requirement.no_blocking_findings: the finalfindings.jsonholds nocriticalorhighfinding.
- Judged checks, run with
--judge:pillar_checklists_covered: each item of each pillar Skill's "Validation checklist" is put to the judge as a yes-or-no question against the design recommendations. The weakest pillar must reach 70%.recommendations_quality: four graded questions (specific, actionable, trade-offs, traceable); the mean must reach 0.67.checklist_itemsreads the checklists from the vendored Skills, nested bullets included, so a vendor sync updates the questions.
- Judge seam (
eval/judge.py): aJudgetakes one document and a map of typed questions and returns a score from 0 to 1 each.GeminiJudgeuses the bundle's default model, orEVAL_JUDGE_MODEL. The shape matches a decision model's, for iteration 5 step 10. One live call confirmed it works. - Interactive case. A case marked
"interactive": trueruns with a scripted user:run_with_scripted_usergrants Approvals, denies cloud reads and answers questions by pattern. It adds the checkspecialists_ask_before_submitting.case-04-interactive-web-appis case 03 run this way. - Thresholds. A case may set
thresholdsforrevisions,secondsandtokens. Exceeding one is printed asREGRESSION, listed in the report, and makes the exit code non-zero. It does not change the score.
automatic_reply in app/nodes/execution_nodes.py lost its leading underscore so the evaluation can reuse it.
Tests: tests/eval/test_scoring.py, with a fake judge.
Open:
- Thresholds are not set. They need a baseline per case from a completed run of the current code. None exists: see below.
- No judged or interactive run yet.
- Judge stability across repeated runs on the same output is unmeasured (iteration 5 step 10 names it).
The evaluation run of 2026-10-06 (stopped, no score)
A three-case run was started on the code as of steps 1 to 6 and stopped by the owner after 35 minutes, during case 01. Cases 02 and 03 never started. Nothing was scored and no files were written, so it gives no baseline. What its log showed:
- Seven Deliverables were submitted on their first attempt in about 15 minutes, including the decomposition and description under their new contracts (step 4).
- The Orchestrator reopened deployment guidance after each failed validation without first asking for Approval of the validation Phase, which is what step 6 intended. Observed three times in one case.
- No Revision of the design recommendations occurred, so step 3 was not exercised.
- Validation did not converge: three passes gave 4, then 7, then 5 different findings, and deployment guidance used all three Revisions. The four findings of the first pass were all still present after the first Revision.
- The Orchestrator stopped without calling a tool four times and was nudged; one such turn took 234 seconds.
The non-convergence led to ADR-0005 (proposed): establish the Target architecture as an explicit graph with numbered Checks before code is written.