Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1.5 KiB
1.5 KiB
Step 16 — Implementation Detail
The judge (eval/judge.py: DeciderJudge for Jev, GeminiJudge for a language model) was built earlier. Built on 2026-10-08: the calibration.
eval/datasets/judge/holds two saved design recommendations (cases 02 and 03, from runs that passed) andlabels.json: 32 questions about them, 12 checklist items and the 4 quality-rubric questions for each document, withlabelempty.eval/calibrate_judge.pyasks both judges every question several times and reports, per judge, stability (the same answer to the same question across runs) and agreement with the labels, and agreement between the two judges.--buildregenerates the question set. A new Jev client is used for each run, because the client caches identical requests and a cached answer would make stability look perfect.
Measured on 2026-10-08, three runs, no labels yet (eval/datasets/judge/calibration.json):
| Jev | Gemini (the bundle's default model) | |
|---|---|---|
| Stability | 1.00 | 0.92 |
| Mean answer | 0.37 | 0.32 |
| Agreement between the two | 0.91 |
So Jev gave the same answer every time, the language-model judge changed its answer on about one question in twelve, and the two agree on nine in ten.
Open, and the owner's to do: fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run .venv/bin/python -m eval.calibrate_judge again. Only that gives agreement with a person, which is what decides the default judge.
Tests: tests/eval/test_scoring.py, "calibrating the judges" (2).