feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -399,3 +399,44 @@ async def test_feedback_on_the_description_itself_has_it_written_again(tmp_path)
|
||||
assert asked["working"]["architecture_description"][STANDING]["feedback"] == [
|
||||
"Make the flow clearer."
|
||||
]
|
||||
|
||||
|
||||
# --- requirements name no products ------------------------------------------------------
|
||||
|
||||
|
||||
async def test_a_requirement_that_names_a_product_is_raised():
|
||||
from app.tools.requirements_review import RequirementsReview
|
||||
|
||||
requirements = [
|
||||
{"id": "FR-1", "kind": "functional", "text": "Run the API on Cloud Run."},
|
||||
{"id": "FR-2", "kind": "functional", "text": "Accept signed webhooks."},
|
||||
]
|
||||
decider = FakeDecider({"FR-1": Answer(0.93, 0.86), "FR-2": Answer(0.05, 0.9)})
|
||||
|
||||
raised = await RequirementsReview(decider)(
|
||||
K.REQUIREMENTS, {"requirements.json": json.dumps(requirements)}, None, "e1"
|
||||
)
|
||||
|
||||
assert len(raised) == 1 and raised[0].startswith("FR-1 names a product")
|
||||
_, questions, scope = decider.asked[0]
|
||||
assert set(questions) == {"FR-1", "FR-2"} and scope == "e1"
|
||||
assert "Accept signed webhooks." in questions["FR-2"].instructions
|
||||
|
||||
|
||||
async def test_the_requirements_review_looks_only_at_requirements_and_never_blocks():
|
||||
from app.tools.requirements_review import RequirementsReview
|
||||
|
||||
decider = FakeDecider({})
|
||||
review = RequirementsReview(decider)
|
||||
|
||||
assert await review(K.TARGET_ARCHITECTURE, GRAPH, UPSTREAM) == ()
|
||||
assert await review(K.REQUIREMENTS, {"requirements.json": "not json"}, None) == ()
|
||||
assert decider.asked == []
|
||||
assert (
|
||||
await RequirementsReview(NoDecider())(
|
||||
K.REQUIREMENTS,
|
||||
{"requirements.json": SAMPLE_DATA["requirements.json"]},
|
||||
None,
|
||||
)
|
||||
== ()
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user