feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s
Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -18,3 +18,9 @@ So Jev gave the same answer every time, the language-model judge changed its ans
|
||||
**Open, and the owner's to do:** fill in the 32 labels (1 or 0 for a yes_no question; 0, 0.33, 0.67 or 1 for a graded one) and run `.venv/bin/python -m eval.calibrate_judge` again. Only that gives agreement with a person, which is what decides the default judge.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "calibrating the judges" (2).
|
||||
|
||||
## The owner's labels, 2026-10-10
|
||||
|
||||
The owner labelled all 32: yes to every checklist item, "mostly" on every quality question. Against them Jev agreed on 28% and Gemini on 16%; the judges answer no to most checklist items (means 0.37 and 0.32), and agree with each other on 88%.
|
||||
|
||||
The disagreement was about the question, not the judges. The owner counts an item the request gives no information about as something to ask the customer, not as unaddressed. That is now an outcome of the checklist review and part of the judged question (iteration 4 step 29). The labels were not kept, and the set is to be rebuilt from documents produced after that change.
|
||||
|
||||
Reference in New Issue
Block a user