feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -139,6 +139,9 @@ class Readings:
|
||||
judged: dict[str, Any] = field(default_factory=dict) # what the decider answered
|
||||
|
||||
|
||||
# How much of the agreed requirements is shown to the decider with a user's message.
|
||||
_AGREED_LIMIT = 24_000
|
||||
|
||||
# The actions that reopen a Deliverable, and which one.
|
||||
_REOPENS = (
|
||||
"revise_requested",
|
||||
@@ -356,6 +359,15 @@ class Router:
|
||||
# our own question about a change they already asked for.
|
||||
structured = reply.get("kind") == "changes" or bool(pending)
|
||||
|
||||
agreed = (
|
||||
""
|
||||
if structured
|
||||
else str(
|
||||
view.deliverables.get(DeliverableKind.REQUIREMENTS.value, {})
|
||||
.get("attachments", {})
|
||||
.get(REQUIREMENTS_FILE, "")
|
||||
)[:_AGREED_LIMIT]
|
||||
)
|
||||
questions = {
|
||||
"change_target": Question(
|
||||
"choice",
|
||||
@@ -375,9 +387,18 @@ class Router:
|
||||
"Is the user asking for a change to work that has already been "
|
||||
"submitted, as opposed to answering a question or adding information?",
|
||||
)
|
||||
if agreed:
|
||||
# Someone who restates what is already agreed is answering, however
|
||||
# much it sounds like a requirement.
|
||||
questions["already_agreed"] = Question(
|
||||
"noul",
|
||||
"Does the user's message only confirm or repeat what the agreed "
|
||||
"requirements above already say, asking for nothing different?",
|
||||
)
|
||||
state = (
|
||||
f"Submitted Deliverables: {', '.join(kind.value for kind in candidates)}.\n"
|
||||
f"Questions waiting for the user: {len(view.pending_questions)}.\n"
|
||||
+ (f"The agreed requirements:\n{agreed}\n" if agreed else "")
|
||||
+ f"Questions waiting for the user: {len(view.pending_questions)}.\n"
|
||||
# What we asked is background. Only the user's own words are their
|
||||
# message: our question echoed back is not something they said.
|
||||
+ (
|
||||
@@ -412,6 +433,10 @@ class Router:
|
||||
}
|
||||
probability = 1.0 if structured else float(answers["asks_change"].value)
|
||||
disposition = self._thresholds.disposition(probability)
|
||||
restated = answers.get("already_agreed")
|
||||
if restated is not None and restated.is_yes(self._thresholds.act):
|
||||
# It restates what is agreed: an answer, not a change.
|
||||
disposition = CLOSE
|
||||
target_answer = answers["change_target"]
|
||||
target = DeliverableKind(str(target_answer.value))
|
||||
readings.change_target = target
|
||||
|
||||
Reference in New Issue
Block a user