# Step 8 — Implementation Detail Built, with two acceptance criteria still open (listed at the end). 1. **Structural checks added to every case** (`eval/scoring.py`): - `components_trace_to_requirements`: every row of the decomposition's component table names a requirement. - `no_blocking_findings`: the final `findings.json` holds no `critical` or `high` finding. 2. **Judged checks**, run with `--judge`: - `pillar_checklists_covered`: each item of each pillar Skill's "Validation checklist" is put to the judge as a yes-or-no question against the design recommendations. The weakest pillar must reach 70%. - `recommendations_quality`: four graded questions (specific, actionable, trade-offs, traceable); the mean must reach 0.67. - `checklist_items` reads the checklists from the vendored Skills, nested bullets included, so a vendor sync updates the questions. 3. **Judge seam** (`eval/judge.py`): a `Judge` takes one document and a map of typed questions and returns a score from 0 to 1 each. `GeminiJudge` uses the bundle's default model, or `EVAL_JUDGE_MODEL`. The shape matches a decision model's, for iteration 5 step 10. One live call confirmed it works. 4. **Interactive case.** A case marked `"interactive": true` runs with a scripted user: `run_with_scripted_user` grants Approvals, denies cloud reads and answers questions by pattern. It adds the check `specialists_ask_before_submitting`. `case-04-interactive-web-app` is case 03 run this way. 5. **Thresholds.** A case may set `thresholds` for `revisions`, `seconds` and `tokens`. Exceeding one is printed as `REGRESSION`, listed in the report, and makes the exit code non-zero. It does not change the score. `automatic_reply` in `app/nodes/execution_nodes.py` lost its leading underscore so the evaluation can reuse it. Tests: `tests/eval/test_scoring.py`, with a fake judge. Open: - **Thresholds are not set.** They need a baseline per case from a completed run of the current code. None exists: see below. - **No judged or interactive run yet.** - **Judge stability** across repeated runs on the same output is unmeasured (iteration 5 step 10 names it). ## The evaluation run of 2026-10-06 (stopped, no score) A three-case run was started on the code as of steps 1 to 6 and stopped by the owner after 35 minutes, during case 01. Cases 02 and 03 never started. Nothing was scored and no files were written, so it gives no baseline. What its log showed: - Seven Deliverables were submitted on their first attempt in about 15 minutes, including the decomposition and description under their new contracts (step 4). - The Orchestrator reopened deployment guidance after each failed validation without first asking for Approval of the validation Phase, which is what step 6 intended. Observed three times in one case. - No Revision of the design recommendations occurred, so step 3 was not exercised. - Validation did not converge: three passes gave 4, then 7, then 5 different findings, and deployment guidance used all three Revisions. The four findings of the first pass were all still present after the first Revision. - The Orchestrator stopped without calling a tool four times and was nudged; one such turn took 234 seconds. The non-convergence led to ADR-0005 (proposed): establish the Target architecture as an explicit graph with numbered Checks before code is written.