feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s
Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,7 @@ from typing import Any
|
||||
from app.agent import open_decider
|
||||
from app.config import get_settings
|
||||
from app.providers import load_provider
|
||||
from app.tools.assessment import checklist_question
|
||||
from eval.judge import DeciderJudge, GeminiJudge, Judge, JudgeUnavailable
|
||||
from eval.scoring import QUALITY_RUBRIC, checklist_items
|
||||
|
||||
@@ -55,9 +56,7 @@ def build(seed: int = 7) -> list[dict[str, Any]]:
|
||||
"id": f"{document.stem}.{pillar}.{len(items)}",
|
||||
"document": document.name,
|
||||
"kind": "yes_no",
|
||||
"text": f"Do the recommendations address this {pillar} "
|
||||
"checklist item, either by recommending it or by "
|
||||
f"explaining why it does not apply? {item}",
|
||||
"text": checklist_question(pillar, item),
|
||||
"label": None,
|
||||
}
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user