feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -21,6 +21,7 @@ from app.adk.runtime import (
files_from_argument,
)
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
from app.tools.pillars import REVIEW_HEADING, render_checklist_review
from app.tools.workspace import (
FILES,
basis_of,
@@ -269,6 +270,7 @@ def specialist_tools(
working: MutableMapping[str, Any] | None = None,
library: Mapping[str, Mapping[str, Any]] | None = None,
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
confirm_first: bool = False,
) -> list[ToolSpec]:
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
the cloud-read gate.
@@ -279,6 +281,9 @@ def specialist_tools(
given to the Specialist as data files alone, to read whole when it needs one.
`review` looks at the meaning of a structurally complete submission; what it
raises is returned to the Specialist once per run, and never a second time.
`confirm_first` is set while a user is present and has not yet been asked about
this Deliverable: a submission is then refused until the Specialist has put its
draft to them. A rule, because an instruction alone was not always followed.
"""
working = {} if working is None else working
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
@@ -339,6 +344,21 @@ def specialist_tools(
"it is already covered. This review runs once, so your next "
"submission is accepted."
)
if confirm_first:
return ToolResult(
"Not submitted. The user has not seen this Deliverable yet. Call "
"`needs_input` with a short summary asking them to approve it or "
"request changes, passing this content as `draft` with its "
"attachments. What you have recorded is kept. Submit when they "
"have replied."
)
if kind is DeliverableKind.DESIGN_RECOMMENDATIONS:
# The pillar advisors' checklist reviews, written from what they
# recorded. A Revision replaces the section, never stacks a second.
checklist_review = render_checklist_review(working)
content = content.split(f"\n## {REVIEW_HEADING}")[0].rstrip() + "\n"
if checklist_review:
content += "\n" + checklist_review
if is_graph:
# The tables and the diagram are views of the graph, so the engine
# writes them: they cannot disagree with it.