feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -153,19 +153,36 @@ def conditions_of(item: Mapping[str, Any]) -> tuple[str, ...]:
return (text.strip(),) if isinstance(text, str) and text.strip() else ()
OUTSIDE_CODE = "outside_code"
def outside_code_of(item: Mapping[str, Any]) -> tuple[str, ...]:
"""What a requirement asks for that infrastructure code cannot show: an outcome
measured in operation, application behaviour, or a process people follow."""
listed = item.get(OUTSIDE_CODE)
if not isinstance(listed, list):
return ()
return tuple(c.strip() for c in listed if isinstance(c, str) and c.strip())
def _lacking_conditions(items: Sequence[Mapping[str, Any]], name: str) -> list[str]:
"""A constraint must say, one by one, what would have to be true for it to be met."""
"""A constraint must say, one by one, what would have to be true for it to be met:
under `conditions` what infrastructure code can show, under `outside_code` what
it cannot. At least one of the two has an entry."""
problems = []
for item in items:
listed = item.get("conditions")
if (
not isinstance(listed, list)
or not listed
or not all(isinstance(c, str) and c.strip() for c in listed)
):
outside = item.get(OUTSIDE_CODE, [])
well_formed = all(
isinstance(group, list)
and all(isinstance(c, str) and c.strip() for c in group)
for group in (listed, outside)
)
if not well_formed or not (listed or outside):
problems.append(
f"{name}: {item['id']} needs `conditions`: a list with one entry "
"for each separate thing it asks for"
"for each separate thing it asks for that infrastructure code can "
f"show (and `{OUTSIDE_CODE}` for what it cannot)"
)
return problems

View File

@@ -43,6 +43,7 @@ def build_execution_graph(
skill_versions: Mapping[str, str] | None = None,
orchestrator: OrchestratorPolicy | None = None,
submission_review: SubmissionReview | None = None,
confirm_drafts: bool = False,
) -> StateGraph:
nodes = make_execution_nodes(
runtime=runtime,
@@ -55,6 +56,7 @@ def build_execution_graph(
skill_versions=skill_versions,
orchestrator=orchestrator,
submission_review=submission_review,
confirm_drafts=confirm_drafts,
)
graph = StateGraph(ExecutionState)

View File

@@ -195,7 +195,8 @@ def chain_gaps(
A functional requirement is carried by a component that has an Element, or by a
Check of its own. A non-functional requirement and a design recommendation are
each a constraint, so each must be the source of a Check (a recommendation may
instead be marked not applicable, with a reason).
instead be marked not applicable, with a reason). A constraint that lists
nothing infrastructure code can show, only what lies outside it, needs none.
"""
gaps: list[str] = []
requirement_ids = {item["id"] for item in requirements}
@@ -229,6 +230,8 @@ def chain_gaps(
)
if requirement.get("kind") == "functional" and served:
continue
if _only_outside_code(requirement):
continue
gaps.append(f"Requirement {requirement['id']} has no Check")
for selection in products:
used = any(
@@ -249,11 +252,19 @@ def chain_gaps(
"which is not a design recommendation"
)
for recommendation in recommendations:
if recommendation["id"] not in sources | excused:
if recommendation["id"] not in sources | excused and not _only_outside_code(
recommendation
):
gaps.append(f"Recommendation {recommendation['id']} has no Check")
return tuple(gaps)
def _only_outside_code(item: Mapping[str, Any]) -> bool:
"""Whether a constraint says, explicitly, that nothing it asks for can be shown
by infrastructure code."""
return item.get("conditions") == [] and bool(item.get("outside_code"))
# --- the diagram is a view of the graph ----------------------------------------------
_UNSAFE = re.compile(r'["\[\]{}()<>|`#;]')