feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -195,7 +195,8 @@ def chain_gaps(
|
||||
A functional requirement is carried by a component that has an Element, or by a
|
||||
Check of its own. A non-functional requirement and a design recommendation are
|
||||
each a constraint, so each must be the source of a Check (a recommendation may
|
||||
instead be marked not applicable, with a reason).
|
||||
instead be marked not applicable, with a reason). A constraint that lists
|
||||
nothing infrastructure code can show, only what lies outside it, needs none.
|
||||
"""
|
||||
gaps: list[str] = []
|
||||
requirement_ids = {item["id"] for item in requirements}
|
||||
@@ -229,6 +230,8 @@ def chain_gaps(
|
||||
)
|
||||
if requirement.get("kind") == "functional" and served:
|
||||
continue
|
||||
if _only_outside_code(requirement):
|
||||
continue
|
||||
gaps.append(f"Requirement {requirement['id']} has no Check")
|
||||
for selection in products:
|
||||
used = any(
|
||||
@@ -249,11 +252,19 @@ def chain_gaps(
|
||||
"which is not a design recommendation"
|
||||
)
|
||||
for recommendation in recommendations:
|
||||
if recommendation["id"] not in sources | excused:
|
||||
if recommendation["id"] not in sources | excused and not _only_outside_code(
|
||||
recommendation
|
||||
):
|
||||
gaps.append(f"Recommendation {recommendation['id']} has no Check")
|
||||
return tuple(gaps)
|
||||
|
||||
|
||||
def _only_outside_code(item: Mapping[str, Any]) -> bool:
|
||||
"""Whether a constraint says, explicitly, that nothing it asks for can be shown
|
||||
by infrastructure code."""
|
||||
return item.get("conditions") == [] and bool(item.get("outside_code"))
|
||||
|
||||
|
||||
# --- the diagram is a view of the graph ----------------------------------------------
|
||||
|
||||
_UNSAFE = re.compile(r'["\[\]{}()<>|`#;]')
|
||||
|
||||
Reference in New Issue
Block a user