# Step 16 — Implementation Detail The judge (`eval/judge.py`: `DeciderJudge` for Jev, `GeminiJudge` for a language model) was built earlier. Built on 2026-10-08: the calibration. 1. `eval/datasets/judge/` holds two saved design recommendations (cases 02 and 03, from runs that passed) and `labels.json`: 32 questions about them, 12 checklist items and the 4 quality-rubric questions for each document, with `label` empty. 2. `eval/calibrate_judge.py` asks both judges every question several times and reports, per judge, stability (the same answer to the same question across runs) and agreement with the labels, and agreement between the two judges. `--build` regenerates the question set. A new Jev client is used for each run, because the client caches identical requests and a cached answer would make stability look perfect. Measured on 2026-10-08, three runs, no labels yet (`eval/datasets/judge/calibration.json`): | | Jev | Gemini (the bundle's default model) | |---|---|---| | Stability | 1.00 | 0.92 | | Mean answer | 0.37 | 0.32 | | Agreement between the two | 0.91 | | So Jev gave the same answer every time, the language-model judge changed its answer on about one question in twelve, and the two agree on nine in ten. **Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge. Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).