feat: what a request does not say is a question for the customer
Some checks failed
validation / verify (push) Failing after 11s

A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.

Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 19:30:12 +01:00
parent f9cc34817b
commit a48468d477
15 changed files with 375 additions and 17 deletions

View File

@@ -31,6 +31,7 @@ from typing import Any
from app.agent import open_decider
from app.config import get_settings
from app.providers import load_provider
from app.tools.assessment import checklist_question
from eval.judge import DeciderJudge, GeminiJudge, Judge, JudgeUnavailable
from eval.scoring import QUALITY_RUBRIC, checklist_items
@@ -55,9 +56,7 @@ def build(seed: int = 7) -> list[dict[str, Any]]:
"id": f"{document.stem}.{pillar}.{len(items)}",
"document": document.name,
"kind": "yes_no",
"text": f"Do the recommendations address this {pillar} "
"checklist item, either by recommending it or by "
f"explaining why it does not apply? {item}",
"text": checklist_question(pillar, item),
"label": None,
}
)

View File

@@ -20,7 +20,7 @@ from typing import Any
from app.adk.runtime import ToolResult
from app.agent import Interaction
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC, checklist_question
from app.tools.local import mermaid_check
from app.workflows.chain import (
COMPONENTS_FILE,
@@ -417,8 +417,7 @@ async def _checklist_coverage(
questions = {
f"{pillar}.{index}": {
"kind": "yes_no",
"text": f"Do the recommendations address this {pillar} checklist item, "
f"either by recommending it or by explaining why it does not apply? {item}",
"text": checklist_question(pillar, item),
}
for pillar, items in checklists.items()
for index, item in enumerate(items)