feat: every Operational requirement verified by a Check; eval spread and report

The operating model is refused while an Operational requirement is verified
by no Check, from it or from a Condition it comes from; the architect is
told so, and the operating-chain Skill lists it. Six of nine earlier runs
left a requirement unverified.

The evaluation checks it too, runs each case N times (--repeat), reports
the spread of people, cost, Roles, Checks and Capabilities (spread.json),
and writes report.md to publish. Result folders are relative to the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:32:05 +01:00
parent efe543e6e2
commit 6fdac22876
12 changed files with 356 additions and 23 deletions

View File

@@ -29,3 +29,4 @@
| T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) |
| T13 | Staffing and cost in code | built (`steps/step-1.md`) |
| T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) |
| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); runs in progress |

View File

@@ -65,3 +65,14 @@ It also filed a gap in its own model as a finding: service continuity staffed on
- **The architect is told** what a finding is and is not.
177 tests pass. Not yet run with real models.
## Case 02 after findings were kept to the solution, 2026-10-14 16:28
Recorded 2026-10-21 from `eval/results/20261014T162852Z/`. Case 02 passed every check in 15.1 minutes: 9 Items, 16 Operational requirements, 21 Capabilities, 11 Roles, 13 Checks, 33 of 33 scenarios, no findings, 3 business outcomes and 6 Measures, 12.8 people at 92,903 GBP a month, one Revision of the operating model.
Read against the runs before it, two things the checks did not show:
- **Cost moved 55% between two runs of the same case** (59,997 GBP at 13:26, 92,903 GBP at 16:28), with 6 Roles against 11. Staffing and cost are computed, so the spread is the model's design, not the arithmetic. Nothing measured it.
- **Three Operational requirements were verified by no Check** (OR-14 the sourcing model, OR-15 the delivery countries, OR-16 the tooling), each served by a Capability. Six of the nine runs of 2026-10-13 and 14 left at least one so. The design required each requirement to be served by a Capability, not verified by a Check, and Checks fell from 31 to 36 a run to 13 or 14 once business outcomes came in.
Both are taken up in step 4.

View File

@@ -0,0 +1,21 @@
# Step 4: every requirement verified, and the spread of repeated runs
Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, after reading the run of 2026-10-14 16:28 (`step-3.md`).
## What was built
- **Every Operational requirement is verified by a Check** (`graph.py`): one whose `source` is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the `operating-chain` Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks.
- **The evaluation:**
- a check that every Operational requirement is verified, naming those that are not, and `requirements_unverified` among the figures;
- `--repeat N`: each case run N times under its own Execution, `<case>-r<n>`;
- `spread.json`: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;
- `report.md`: the run as a report that can be published, with each run's result, the checks that failed and the spread;
- each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.
## Verified
184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.
## The runs
Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one.