feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -529,6 +529,7 @@ async def test_a_free_text_reply_that_asks_for_a_change_is_acted_on(tmp_path):
|
||||
"asks_change": Answer(0.93, 0.86),
|
||||
"change_target": Answer("requirements", 0.9),
|
||||
"change_clarity": Answer(1.9, 0.8),
|
||||
"already_agreed": Answer(0.03, 0.94),
|
||||
}
|
||||
)
|
||||
engine, _ = engine_for(tmp_path, turns, decider)
|
||||
@@ -755,3 +756,46 @@ async def test_asking_the_user_again_and_again_with_nothing_done_stops_the_run(
|
||||
assert asked == MAX_IDLE_ASKS + 1
|
||||
assert "asked" in result.state["warnings"][-1]
|
||||
assert "no work done in between" in result.state["warnings"][-1]
|
||||
|
||||
|
||||
async def test_a_reply_that_restates_what_is_agreed_is_an_answer_not_a_change(
|
||||
tmp_path,
|
||||
):
|
||||
"""Evaluation case 04: a requirement said again in reply to a later question."""
|
||||
from app.nodes.router import Readings, TurnView
|
||||
from app.states.ledger import Ledger
|
||||
|
||||
ledger = Ledger()
|
||||
ledger.start([K.REQUIREMENTS], actor="orchestrator")
|
||||
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
|
||||
view = TurnView(
|
||||
execution_id="e1",
|
||||
pre_approved=False,
|
||||
ledger=ledger,
|
||||
deliverables={
|
||||
"requirements": {
|
||||
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
|
||||
}
|
||||
},
|
||||
pending_questions=[],
|
||||
outcomes=[],
|
||||
last_reply={"kind": "reply", "text": "99.9% availability.", "asked": []},
|
||||
)
|
||||
asks = {
|
||||
"asks_change": Answer(0.9, 0.8),
|
||||
"change_target": Answer("requirements", 0.95),
|
||||
"change_clarity": Answer(1.9, 0.8),
|
||||
}
|
||||
|
||||
restated, changed = Readings(), Readings()
|
||||
decider = FakeDecider({**asks, "already_agreed": Answer(0.85, 0.7)})
|
||||
await Router(decider, None, Thresholds())._read_message(view, restated)
|
||||
await Router(
|
||||
FakeDecider({**asks, "already_agreed": Answer(0.02, 0.96)}), None, Thresholds()
|
||||
)._read_message(view, changed)
|
||||
|
||||
assert restated.change == "close"
|
||||
assert changed.change == "act"
|
||||
state, questions, _ = decider.asked[0]
|
||||
assert "The agreed requirements:" in state and "NFR-1" in state
|
||||
assert "already_agreed" in questions
|
||||
|
||||
Reference in New Issue
Block a user