feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -689,13 +689,11 @@ def test_a_non_functional_requirement_must_list_its_conditions():
{"requirements.json": json.dumps(requirements)},
)
needs = (
"needs `conditions`: a list with one entry for each separate thing it asks for"
)
assert missing == (
needs = "needs `conditions`:"
assert [line.split(" a list")[0] for line in missing] == [
f"requirements.json: NFR-1 {needs}",
f"requirements.json: NFR-2 {needs}",
)
]
def test_a_recommendation_must_list_its_conditions():
@@ -709,7 +707,61 @@ def test_a_recommendation_must_list_its_conditions():
{"recommendations.json": json.dumps(recommendations)},
)
needs = (
"needs `conditions`: a list with one entry for each separate thing it asks for"
needs = "needs `conditions`:"
assert [line.split(" a list")[0] for line in missing] == [
f"recommendations.json: DR-1 {needs}"
]
def test_what_code_cannot_show_is_listed_apart_and_needs_no_check():
"""An outcome measured in operation is recorded, and no Check is asked for it."""
requirements = json.loads(SAMPLE_DATA["requirements.json"])
requirements.append(
{
"id": "NFR-2",
"kind": "non_functional",
"text": "Pages load within one second at p95.",
"conditions": [],
"outside_code": ["Measured p95 page load is under one second."],
}
)
assert missing == (f"recommendations.json: DR-1 {needs}",)
args = sample_submission(K.REQUIREMENTS).args
accepted = check_submission(
K.REQUIREMENTS,
args["content"],
{"requirements.json": json.dumps(requirements)},
)
gaps = chain_gaps(
parse_graph(SAMPLE_DATA["target-architecture.json"]),
requirements=requirements,
components=json.loads(SAMPLE_DATA["components.json"]),
products=json.loads(SAMPLE_DATA["products.json"]),
recommendations=json.loads(SAMPLE_DATA["recommendations.json"]),
)
assert accepted == ()
assert gaps == ()
def test_a_constraint_with_conditions_still_needs_a_check():
requirements = json.loads(SAMPLE_DATA["requirements.json"])
requirements.append(
{
"id": "NFR-2",
"kind": "non_functional",
"text": "Two zones, 99.9%.",
"conditions": ["The service runs in two zones."],
"outside_code": ["Measured availability is 99.9%."],
}
)
gaps = chain_gaps(
parse_graph(SAMPLE_DATA["target-architecture.json"]),
requirements=requirements,
components=json.loads(SAMPLE_DATA["components.json"]),
products=json.loads(SAMPLE_DATA["products.json"]),
recommendations=json.loads(SAMPLE_DATA["recommendations.json"]),
)
assert gaps == ("Requirement NFR-2 has no Check",)