feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -26,11 +26,14 @@ A list. Number functional requirements `FR-1`, `FR-2` and non-functional ones `N
[
{"id": "FR-1", "kind": "functional", "text": "Accept signed webhooks over HTTPS."},
{"id": "NFR-1", "kind": "non_functional", "text": "Raw events are encrypted at rest and retained for 30 days.",
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]}
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]},
{"id": "NFR-2", "kind": "non_functional", "text": "The service is available 99.9% of each month, across two zones.",
"conditions": ["The service runs in at least two zones."],
"outside_code": ["Measured availability is at least 99.9% a month."]}
]
```
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met and that infrastructure configuration can show. What it asks for that configuration cannot show goes under `outside_code`: an outcome measured in operation, behaviour of the application's own code, or a process people follow. A requirement has at least one entry between the two; one that is entirely outside code has `"conditions": []`. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
`kind` is `functional` or `non_functional`. Write one requirement per entry, so each can be checked on its own.
@@ -59,7 +62,7 @@ A list. Each recommendation names its pillar and the components it applies to; u
"conditions": ["The topic is encrypted with a customer-managed key.", "The messaging service's own identity may use that key."]}]
```
Every recommendation lists its `conditions`, as a non-functional requirement does. When it applies to several components, give a condition for each.
Every recommendation lists its `conditions`, and `outside_code` where it applies, as a non-functional requirement does. When it applies to several components, give a condition for each.
Write one configurable decision per recommendation. "Harden the service" cannot become a Check; "disable public ingress on the worker service" can.
@@ -87,7 +90,7 @@ Write one configurable decision per recommendation. "Harden the service" cannot
- **Elements.** One per resource to be created. `component` is the component it realises; leave it out for a supporting Element such as an identity, a key or a network. `product` for a component's Element is exactly the product selected for that component.
- **Relationships.** `from` and `to` are Element ids. `kind` says what the connection is: calls, publishes to, reads from, runs as, is granted access to.
- **Checks.** `applies_to` is an Element or Relationship id. `source` is the requirement or recommendation id the Check comes from. Every Element has at least one Check.
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. An entry under `outside_code` gets no Check, and a requirement or recommendation with nothing but `outside_code` needs none. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
- **Completeness.** Every component has an Element. Every non-functional requirement and every recommendation is the `source` of at least one Check, unless the recommendation is listed under `not_applicable` with a reason. Every selected product appears on its component's Element.
#### Writing a Check
@@ -135,5 +138,5 @@ Implement every Element, Relationship and Check, and nothing the Target architec
- Give every Check exactly one verdict: `pass`, `fail` or `not_verifiable`, with the evidence you read in the code.
- `dry_run.verdict` is `pass`, `fail`, or `not_run` when the tool could not run.
- Find every gap in one pass: go through each requirement and recommendation in turn and compare what its text asks with the Checks that name it as `source`.
- Find every gap in one pass: go through each requirement and recommendation in turn and compare its `conditions` with the Checks that name it as `source`. Propose nothing for its `outside_code`.
- Judge the code against the Checks and nothing else. Something the Target architecture lacks is a `proposed_additions` entry, with the requirement or recommendation id it serves in `reason`. It is never a failed Check.