feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -21,6 +21,7 @@ from app.adk.runtime import (
|
||||
files_from_argument,
|
||||
)
|
||||
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
|
||||
from app.tools.pillars import REVIEW_HEADING, render_checklist_review
|
||||
from app.tools.workspace import (
|
||||
FILES,
|
||||
basis_of,
|
||||
@@ -269,6 +270,7 @@ def specialist_tools(
|
||||
working: MutableMapping[str, Any] | None = None,
|
||||
library: Mapping[str, Mapping[str, Any]] | None = None,
|
||||
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
|
||||
confirm_first: bool = False,
|
||||
) -> list[ToolSpec]:
|
||||
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
||||
the cloud-read gate.
|
||||
@@ -279,6 +281,9 @@ def specialist_tools(
|
||||
given to the Specialist as data files alone, to read whole when it needs one.
|
||||
`review` looks at the meaning of a structurally complete submission; what it
|
||||
raises is returned to the Specialist once per run, and never a second time.
|
||||
`confirm_first` is set while a user is present and has not yet been asked about
|
||||
this Deliverable: a submission is then refused until the Specialist has put its
|
||||
draft to them. A rule, because an instruction alone was not always followed.
|
||||
"""
|
||||
working = {} if working is None else working
|
||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||
@@ -339,6 +344,21 @@ def specialist_tools(
|
||||
"it is already covered. This review runs once, so your next "
|
||||
"submission is accepted."
|
||||
)
|
||||
if confirm_first:
|
||||
return ToolResult(
|
||||
"Not submitted. The user has not seen this Deliverable yet. Call "
|
||||
"`needs_input` with a short summary asking them to approve it or "
|
||||
"request changes, passing this content as `draft` with its "
|
||||
"attachments. What you have recorded is kept. Submit when they "
|
||||
"have replied."
|
||||
)
|
||||
if kind is DeliverableKind.DESIGN_RECOMMENDATIONS:
|
||||
# The pillar advisors' checklist reviews, written from what they
|
||||
# recorded. A Revision replaces the section, never stacks a second.
|
||||
checklist_review = render_checklist_review(working)
|
||||
content = content.split(f"\n## {REVIEW_HEADING}")[0].rstrip() + "\n"
|
||||
if checklist_review:
|
||||
content += "\n" + checklist_review
|
||||
if is_graph:
|
||||
# The tables and the diagram are views of the graph, so the engine
|
||||
# writes them: they cannot disagree with it.
|
||||
|
||||
Reference in New Issue
Block a user