4.4 KiB
Step 4: every requirement verified, and the spread of repeated runs
Built 2026-10-21 on feat/eval-coverage-and-spread, at the owner's request, after reading the run of 2026-10-14 16:28 (step-3.md).
What was built
- Every Operational requirement is verified by a Check (
graph.py): one whosesourceis the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; theoperating-chainSkill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks. - The evaluation:
- a check that every Operational requirement is verified, naming those that are not, and
requirements_unverifiedamong the figures; --repeat N: each case run N times under its own Execution,<case>-r<n>;spread.json: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;report.md: the run as a report that can be published, with each run's result, the checks that failed and the spread;- each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.
- a check that every Operational requirement is verified, naming those that are not, and
Verified
184 tests pass; ruff and mypy clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.
The runs, 2026-10-21
Both cases three times (eval/results/20261021T203624Z/)
| Case | Run | Result | Minutes | Revisions | Requirements verified | Roles | People | Cost a month |
|---|---|---|---|---|---|---|---|---|
| 01, written spec | 1 | pass | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP |
| 01 | 2 | pass | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP |
| 01 | 3 | pass | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP |
| 02, digested architecture | 1 | fail | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP |
| 02 | 2 | pass | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP |
| 02 | 3 | pass | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP |
- Every requirement was verified in every run. Checks rose to 15 to 20 a run, from 13 or 14.
- Case 02, run 1 ended with warnings at the operating model's revision limit. The validator failed five Checks whose words the Capability they are on does not bear out: "continuously maintained" and "continuously reviewed" on Capabilities staffed in business hours, a 10-week validation on the Capability that manages the roadmap rather than the one that delivers, sign-off the Capability does not mention, and alignment with DORA, PCI DSS and ISO 27001 on a Capability covering PCI DSS alone. Four of the five come from Conditions, not requirements: the weakness of step 3 ("a Check whose hours or Capability did not fit"), not the new rule.
- The spread: case 01's cost moved 6.2% across its runs while its people moved 47.3%: more people, cheaper. Case 02's cost moved 46.6% (73,863 to 116,409 GBP) and its people 46.4%; Roles 27 to 33%, Checks 17 to 25%; Capabilities not at all (21 each time).
Through the Orchestration engine from the solution architecture agent (eval/results/20261021T203623Z-handoff/)
One pre-approved request for architecture_and_operating_model, Customer case case-08-landing-zone, from the benchmark landing-zone spec, with case 02's operating fields under operating_model. Completed in 74.9 minutes: the architecture in about 50, its 15 kept files given to this agent as sources, which digested them (metadata.source: architecture:6c316f05…) and completed: 24 Operational requirements, all verified, 21 Capabilities, 8 Roles, 26 Checks, 44 of 44 scenarios, 12.2 people at 87,959 GBP a month. Both Executions' files are kept in the Orchestration engine's store (MinIO, 28 objects) and were fetched through their links. The signatures were taken off the links in the results before publishing.
What remains
- The spread is the larger finding. The same architecture costs between 74 and 116 thousand GBP a month across three runs. The figures are computed; the design is not stable. What moves is the Roles and how work is shared between them.
- Checks that the Capability they are on cannot bear out end a run at its revision limit about one time in three for case 02. A data check that refuses a "continuous" or 24x7 Check on a Capability whose hours are narrower would catch the commonest of them before validation.