From 6fdac22876349021634c8a3ffda0d56b3bdd7797 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Wed, 21 Oct 2026 21:32:05 +0100 Subject: [PATCH] feat: every Operational requirement verified by a Check; eval spread and report The operating model is refused while an Operational requirement is verified by no Check, from it or from a Condition it comes from; the architect is told so, and the operating-chain Skill lists it. Six of nine earlier runs left a requirement unverified. The evaluation checks it too, runs each case N times (--repeat), reports the spread of people, cost, Roles, Checks and Capabilities (spread.json), and writes report.md to publish. Result folders are relative to the run. Co-Authored-By: Claude Opus 5.5 --- .agents/plans/iter-01-first-slice/MAP.md | 1 + .../plans/iter-01-first-slice/steps/step-3.md | 11 ++ .../plans/iter-01-first-slice/steps/step-4.md | 21 ++ agents/operating-model-architect.md | 1 + app/operating_model/graph.py | 18 +- eval/operating_model_eval.py | 182 ++++++++++++++++-- eval/scoring.py | 38 +++- skills/operating-chain/SKILL.md | 2 +- tests/support.py | 12 ++ tests/test_eval.py | 77 +++++++- tests/test_graph.py | 12 ++ tests/test_scenarios.py | 4 +- 12 files changed, 356 insertions(+), 23 deletions(-) create mode 100644 .agents/plans/iter-01-first-slice/steps/step-4.md diff --git a/.agents/plans/iter-01-first-slice/MAP.md b/.agents/plans/iter-01-first-slice/MAP.md index 07a5a7b..2ae3622 100644 --- a/.agents/plans/iter-01-first-slice/MAP.md +++ b/.agents/plans/iter-01-first-slice/MAP.md @@ -29,3 +29,4 @@ | T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) | | T13 | Staffing and cost in code | built (`steps/step-1.md`) | | T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) | +| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); runs in progress | diff --git a/.agents/plans/iter-01-first-slice/steps/step-3.md b/.agents/plans/iter-01-first-slice/steps/step-3.md index 4762128..f1142fd 100644 --- a/.agents/plans/iter-01-first-slice/steps/step-3.md +++ b/.agents/plans/iter-01-first-slice/steps/step-3.md @@ -65,3 +65,14 @@ It also filed a gap in its own model as a finding: service continuity staffed on - **The architect is told** what a finding is and is not. 177 tests pass. Not yet run with real models. + +## Case 02 after findings were kept to the solution, 2026-10-14 16:28 + +Recorded 2026-10-21 from `eval/results/20261014T162852Z/`. Case 02 passed every check in 15.1 minutes: 9 Items, 16 Operational requirements, 21 Capabilities, 11 Roles, 13 Checks, 33 of 33 scenarios, no findings, 3 business outcomes and 6 Measures, 12.8 people at 92,903 GBP a month, one Revision of the operating model. + +Read against the runs before it, two things the checks did not show: + +- **Cost moved 55% between two runs of the same case** (59,997 GBP at 13:26, 92,903 GBP at 16:28), with 6 Roles against 11. Staffing and cost are computed, so the spread is the model's design, not the arithmetic. Nothing measured it. +- **Three Operational requirements were verified by no Check** (OR-14 the sourcing model, OR-15 the delivery countries, OR-16 the tooling), each served by a Capability. Six of the nine runs of 2026-10-13 and 14 left at least one so. The design required each requirement to be served by a Capability, not verified by a Check, and Checks fell from 31 to 36 a run to 13 or 14 once business outcomes came in. + +Both are taken up in step 4. diff --git a/.agents/plans/iter-01-first-slice/steps/step-4.md b/.agents/plans/iter-01-first-slice/steps/step-4.md new file mode 100644 index 0000000..cc4c4bc --- /dev/null +++ b/.agents/plans/iter-01-first-slice/steps/step-4.md @@ -0,0 +1,21 @@ +# Step 4: every requirement verified, and the spread of repeated runs + +Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, after reading the run of 2026-10-14 16:28 (`step-3.md`). + +## What was built + +- **Every Operational requirement is verified by a Check** (`graph.py`): one whose `source` is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the `operating-chain` Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks. +- **The evaluation:** + - a check that every Operational requirement is verified, naming those that are not, and `requirements_unverified` among the figures; + - `--repeat N`: each case run N times under its own Execution, `-r`; + - `spread.json`: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed; + - `report.md`: the run as a report that can be published, with each run's result, the checks that failed and the spread; + - each result's folder relative to the run's: results are published, and hold no path of the machine they ran on. + +## Verified + +184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution. + +## The runs + +Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one. diff --git a/agents/operating-model-architect.md b/agents/operating-model-architect.md index 5c0934a..89b2771 100644 --- a/agents/operating-model-architect.md +++ b/agents/operating-model-architect.md @@ -62,6 +62,7 @@ Checks: - A Check is a condition that can be shown to hold or not, on a Capability, a Role or a Responsibility. Its `source` is the Operational requirement or the Condition it comes from. - Every Condition in `scope.stated.solution.conditions` must be carried by at least one Check whose `source` is that Condition's id. +- Every Operational requirement must be verified by at least one Check: one whose `source` is the requirement's id, or the id of a Condition the requirement comes from. Organisation and tooling requirements need one too: a requirement that staff work only from certain countries is verified by a Check on the Roles' countries, one that tooling is integrated by a Check on the Capability that runs it. - Start from the catalogue's `typical_checks` and rewrite each for this solution, with the customer's own numbers. - Put a Check on the Capability that does the work it describes. A Check that service is restored within four hours is on the Capability that restores it, not on the one that reports service levels. diff --git a/app/operating_model/graph.py b/app/operating_model/graph.py index 464163b..73e7f72 100644 --- a/app/operating_model/graph.py +++ b/app/operating_model/graph.py @@ -18,7 +18,8 @@ The Capabilities are the capability model's, by id. A submission is refused while the graph does not hold together, so what the customer approves is complete: every Capability has one Role answerable for it and someone to do it at the hours it is -needed, and everything the customer said must be kept true is carried by a Check. +needed, everything the customer said must be kept true is carried by a Check, and +every Operational requirement is verified by one. The responsibility matrix, the organisation chart and the staffing and cost are written from the graph by code, as views of it. @@ -301,6 +302,21 @@ def check(card: RateCard, frameworks: Collection[str] = ()) -> chain.DataCheck: for identifier in conditions if identifier not in carried ] + # Every Operational requirement is verified: by a Check from it, or from a + # Condition it comes from. Serving it with a Capability says who does the + # work; a Check says how anyone would know it is done. + for requirement in chain.requirements(upstream): + identifier = str(requirement["id"]) + named = requirement.get("source") or [] + named = [named] if isinstance(named, str) else named + if identifier not in carried and not any( + str(source) in carried and str(source) in conditions for source in named + ): + problems.append( + f"{name}: Operational requirement {identifier} is verified by no " + "Check: give it one, from the requirement or from a Condition " + "it comes from" + ) problems += check_measures( graph, scope, capabilities, by_capability, roles, frameworks, name ) diff --git a/eval/operating_model_eval.py b/eval/operating_model_eval.py index b9cb36c..8e04953 100644 --- a/eval/operating_model_eval.py +++ b/eval/operating_model_eval.py @@ -6,12 +6,14 @@ It costs model usage: run it deliberately. Usage: GOOGLE_GENAI_USE_ENTERPRISE=true GOOGLE_CLOUD_PROJECT= GOOGLE_CLOUD_LOCATION=global \\ - .venv/bin/python -m eval.operating_model_eval [case-id ...] + .venv/bin/python -m eval.operating_model_eval [--repeat N] [case-id ...] The model settings are read from the environment or from `.env`. With no case named, -every case in eval/datasets/cases.json runs, one after another. Results are written -under eval/results/