diff --git a/.agents/plans/iter-01-first-slice/MAP.md b/.agents/plans/iter-01-first-slice/MAP.md index 07a5a7b..861826a 100644 --- a/.agents/plans/iter-01-first-slice/MAP.md +++ b/.agents/plans/iter-01-first-slice/MAP.md @@ -29,3 +29,4 @@ | T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) | | T13 | Staffing and cost in code | built (`steps/step-1.md`) | | T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) | +| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); run 2026-10-21: five of six passed, every requirement verified; through the Orchestration engine from the solution architecture agent, completed | diff --git a/.agents/plans/iter-01-first-slice/steps/step-3.md b/.agents/plans/iter-01-first-slice/steps/step-3.md index 4762128..f1142fd 100644 --- a/.agents/plans/iter-01-first-slice/steps/step-3.md +++ b/.agents/plans/iter-01-first-slice/steps/step-3.md @@ -65,3 +65,14 @@ It also filed a gap in its own model as a finding: service continuity staffed on - **The architect is told** what a finding is and is not. 177 tests pass. Not yet run with real models. + +## Case 02 after findings were kept to the solution, 2026-10-14 16:28 + +Recorded 2026-10-21 from `eval/results/20261014T162852Z/`. Case 02 passed every check in 15.1 minutes: 9 Items, 16 Operational requirements, 21 Capabilities, 11 Roles, 13 Checks, 33 of 33 scenarios, no findings, 3 business outcomes and 6 Measures, 12.8 people at 92,903 GBP a month, one Revision of the operating model. + +Read against the runs before it, two things the checks did not show: + +- **Cost moved 55% between two runs of the same case** (59,997 GBP at 13:26, 92,903 GBP at 16:28), with 6 Roles against 11. Staffing and cost are computed, so the spread is the model's design, not the arithmetic. Nothing measured it. +- **Three Operational requirements were verified by no Check** (OR-14 the sourcing model, OR-15 the delivery countries, OR-16 the tooling), each served by a Capability. Six of the nine runs of 2026-10-13 and 14 left at least one so. The design required each requirement to be served by a Capability, not verified by a Check, and Checks fell from 31 to 36 a run to 13 or 14 once business outcomes came in. + +Both are taken up in step 4. diff --git a/.agents/plans/iter-01-first-slice/steps/step-4.md b/.agents/plans/iter-01-first-slice/steps/step-4.md new file mode 100644 index 0000000..5241433 --- /dev/null +++ b/.agents/plans/iter-01-first-slice/steps/step-4.md @@ -0,0 +1,43 @@ +# Step 4: every requirement verified, and the spread of repeated runs + +Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, after reading the run of 2026-10-14 16:28 (`step-3.md`). + +## What was built + +- **Every Operational requirement is verified by a Check** (`graph.py`): one whose `source` is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the `operating-chain` Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks. +- **The evaluation:** + - a check that every Operational requirement is verified, naming those that are not, and `requirements_unverified` among the figures; + - `--repeat N`: each case run N times under its own Execution, `-r`; + - `spread.json`: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed; + - `report.md`: the run as a report that can be published, with each run's result, the checks that failed and the spread; + - each result's folder relative to the run's: results are published, and hold no path of the machine they ran on. + +## Verified + +184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution. + +## The runs, 2026-10-21 + +### Both cases three times (`eval/results/20261021T203624Z/`) + +| Case | Run | Result | Minutes | Revisions | Requirements verified | Roles | People | Cost a month | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | +| 01, written spec | 1 | pass | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP | +| 01 | 2 | pass | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP | +| 01 | 3 | pass | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP | +| 02, digested architecture | 1 | **fail** | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP | +| 02 | 2 | pass | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP | +| 02 | 3 | pass | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP | + +- **Every requirement was verified in every run.** Checks rose to 15 to 20 a run, from 13 or 14. +- **Case 02, run 1 ended with warnings** at the operating model's revision limit. The validator failed five Checks whose words the Capability they are on does not bear out: "continuously maintained" and "continuously reviewed" on Capabilities staffed in business hours, a 10-week validation on the Capability that manages the roadmap rather than the one that delivers, sign-off the Capability does not mention, and alignment with DORA, PCI DSS and ISO 27001 on a Capability covering PCI DSS alone. Four of the five come from Conditions, not requirements: the weakness of step 3 ("a Check whose hours or Capability did not fit"), not the new rule. +- **The spread:** case 01's cost moved 6.2% across its runs while its people moved 47.3%: more people, cheaper. Case 02's cost moved 46.6% (73,863 to 116,409 GBP) and its people 46.4%; Roles 27 to 33%, Checks 17 to 25%; Capabilities not at all (21 each time). + +### Through the Orchestration engine from the solution architecture agent (`eval/results/20261021T203623Z-handoff/`) + +One pre-approved request for `architecture_and_operating_model`, Customer case `case-08-landing-zone`, from the benchmark landing-zone spec, with case 02's operating fields under `operating_model`. Completed in 74.9 minutes: the architecture in about 50, its 15 kept files given to this agent as `sources`, which digested them (`metadata.source: architecture:6c316f05…`) and completed: 24 Operational requirements, all verified, 21 Capabilities, 8 Roles, 26 Checks, 44 of 44 scenarios, 12.2 people at 87,959 GBP a month. Both Executions' files are kept in the Orchestration engine's store (MinIO, 28 objects) and were fetched through their links. The signatures were taken off the links in the results before publishing. + +## What remains + +- **The spread is the larger finding.** The same architecture costs between 74 and 116 thousand GBP a month across three runs. The figures are computed; the design is not stable. What moves is the Roles and how work is shared between them. +- **Checks that the Capability they are on cannot bear out** end a run at its revision limit about one time in three for case 02. A data check that refuses a "continuous" or 24x7 Check on a Capability whose hours are narrower would catch the commonest of them before validation. diff --git a/.gitignore b/.gitignore index 66713d0..ea78f13 100644 --- a/.gitignore +++ b/.gitignore @@ -19,8 +19,13 @@ venv/ # name people, teams and suppliers deliverables/ -# Evaluation run outputs -eval/results/ +# Evaluation run outputs, but for the runs published (they hold fictional benchmark +# data only, and links without their signatures) +eval/results/* +!eval/results/20261021T203624Z/ +!eval/results/20261021T203623Z-handoff/ +!eval/results/20261021T203624Z/deliverables/ +!eval/results/20261021T203624Z/deliverables/**/deliverables/ # Lesson candidates captured from Executions (may quote user feedback) lessons/candidates/ diff --git a/agents/operating-model-architect.md b/agents/operating-model-architect.md index 5c0934a..89b2771 100644 --- a/agents/operating-model-architect.md +++ b/agents/operating-model-architect.md @@ -62,6 +62,7 @@ Checks: - A Check is a condition that can be shown to hold or not, on a Capability, a Role or a Responsibility. Its `source` is the Operational requirement or the Condition it comes from. - Every Condition in `scope.stated.solution.conditions` must be carried by at least one Check whose `source` is that Condition's id. +- Every Operational requirement must be verified by at least one Check: one whose `source` is the requirement's id, or the id of a Condition the requirement comes from. Organisation and tooling requirements need one too: a requirement that staff work only from certain countries is verified by a Check on the Roles' countries, one that tooling is integrated by a Check on the Capability that runs it. - Start from the catalogue's `typical_checks` and rewrite each for this solution, with the customer's own numbers. - Put a Check on the Capability that does the work it describes. A Check that service is restored within four hours is on the Capability that restores it, not on the one that reports service levels. diff --git a/app/operating_model/graph.py b/app/operating_model/graph.py index 464163b..73e7f72 100644 --- a/app/operating_model/graph.py +++ b/app/operating_model/graph.py @@ -18,7 +18,8 @@ The Capabilities are the capability model's, by id. A submission is refused while the graph does not hold together, so what the customer approves is complete: every Capability has one Role answerable for it and someone to do it at the hours it is -needed, and everything the customer said must be kept true is carried by a Check. +needed, everything the customer said must be kept true is carried by a Check, and +every Operational requirement is verified by one. The responsibility matrix, the organisation chart and the staffing and cost are written from the graph by code, as views of it. @@ -301,6 +302,21 @@ def check(card: RateCard, frameworks: Collection[str] = ()) -> chain.DataCheck: for identifier in conditions if identifier not in carried ] + # Every Operational requirement is verified: by a Check from it, or from a + # Condition it comes from. Serving it with a Capability says who does the + # work; a Check says how anyone would know it is done. + for requirement in chain.requirements(upstream): + identifier = str(requirement["id"]) + named = requirement.get("source") or [] + named = [named] if isinstance(named, str) else named + if identifier not in carried and not any( + str(source) in carried and str(source) in conditions for source in named + ): + problems.append( + f"{name}: Operational requirement {identifier} is verified by no " + "Check: give it one, from the requirement or from a Condition " + "it comes from" + ) problems += check_measures( graph, scope, capabilities, by_capability, roles, frameworks, name ) diff --git a/eval/operating_model_eval.py b/eval/operating_model_eval.py index b9cb36c..8e04953 100644 --- a/eval/operating_model_eval.py +++ b/eval/operating_model_eval.py @@ -6,12 +6,14 @@ It costs model usage: run it deliberately. Usage: GOOGLE_GENAI_USE_ENTERPRISE=true GOOGLE_CLOUD_PROJECT= GOOGLE_CLOUD_LOCATION=global \\ - .venv/bin/python -m eval.operating_model_eval [case-id ...] + .venv/bin/python -m eval.operating_model_eval [--repeat N] [case-id ...] The model settings are read from the environment or from `.env`. With no case named, -every case in eval/datasets/cases.json runs, one after another. Results are written -under eval/results/