feat: every Operational requirement verified by a Check; eval spread and report

The operating model is refused while an Operational requirement is verified
by no Check, from it or from a Condition it comes from; the architect is
told so, and the operating-chain Skill lists it. Six of nine earlier runs
left a requirement unverified.

The evaluation checks it too, runs each case N times (--repeat), reports
the spread of people, cost, Roles, Checks and Capabilities (spread.json),
and writes report.md to publish. Result folders are relative to the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 21:32:05 +01:00
parent efe543e6e2
commit 6fdac22876
12 changed files with 356 additions and 23 deletions

View File

@@ -29,3 +29,4 @@
| T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) |
| T13 | Staffing and cost in code | built (`steps/step-1.md`) |
| T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) |
| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); runs in progress |

View File

@@ -65,3 +65,14 @@ It also filed a gap in its own model as a finding: service continuity staffed on
- **The architect is told** what a finding is and is not.
177 tests pass. Not yet run with real models.
## Case 02 after findings were kept to the solution, 2026-10-14 16:28
Recorded 2026-10-21 from `eval/results/20261014T162852Z/`. Case 02 passed every check in 15.1 minutes: 9 Items, 16 Operational requirements, 21 Capabilities, 11 Roles, 13 Checks, 33 of 33 scenarios, no findings, 3 business outcomes and 6 Measures, 12.8 people at 92,903 GBP a month, one Revision of the operating model.
Read against the runs before it, two things the checks did not show:
- **Cost moved 55% between two runs of the same case** (59,997 GBP at 13:26, 92,903 GBP at 16:28), with 6 Roles against 11. Staffing and cost are computed, so the spread is the model's design, not the arithmetic. Nothing measured it.
- **Three Operational requirements were verified by no Check** (OR-14 the sourcing model, OR-15 the delivery countries, OR-16 the tooling), each served by a Capability. Six of the nine runs of 2026-10-13 and 14 left at least one so. The design required each requirement to be served by a Capability, not verified by a Check, and Checks fell from 31 to 36 a run to 13 or 14 once business outcomes came in.
Both are taken up in step 4.

View File

@@ -0,0 +1,21 @@
# Step 4: every requirement verified, and the spread of repeated runs
Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, after reading the run of 2026-10-14 16:28 (`step-3.md`).
## What was built
- **Every Operational requirement is verified by a Check** (`graph.py`): one whose `source` is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the `operating-chain` Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks.
- **The evaluation:**
- a check that every Operational requirement is verified, naming those that are not, and `requirements_unverified` among the figures;
- `--repeat N`: each case run N times under its own Execution, `<case>-r<n>`;
- `spread.json`: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;
- `report.md`: the run as a report that can be published, with each run's result, the checks that failed and the spread;
- each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.
## Verified
184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.
## The runs
Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one.

View File

@@ -62,6 +62,7 @@ Checks:
- A Check is a condition that can be shown to hold or not, on a Capability, a Role or a Responsibility. Its `source` is the Operational requirement or the Condition it comes from.
- Every Condition in `scope.stated.solution.conditions` must be carried by at least one Check whose `source` is that Condition's id.
- Every Operational requirement must be verified by at least one Check: one whose `source` is the requirement's id, or the id of a Condition the requirement comes from. Organisation and tooling requirements need one too: a requirement that staff work only from certain countries is verified by a Check on the Roles' countries, one that tooling is integrated by a Check on the Capability that runs it.
- Start from the catalogue's `typical_checks` and rewrite each for this solution, with the customer's own numbers.
- Put a Check on the Capability that does the work it describes. A Check that service is restored within four hours is on the Capability that restores it, not on the one that reports service levels.

View File

@@ -18,7 +18,8 @@
The Capabilities are the capability model's, by id. A submission is refused while
the graph does not hold together, so what the customer approves is complete: every
Capability has one Role answerable for it and someone to do it at the hours it is
needed, and everything the customer said must be kept true is carried by a Check.
needed, everything the customer said must be kept true is carried by a Check, and
every Operational requirement is verified by one.
The responsibility matrix, the organisation chart and the staffing and cost are
written from the graph by code, as views of it.
@@ -301,6 +302,21 @@ def check(card: RateCard, frameworks: Collection[str] = ()) -> chain.DataCheck:
for identifier in conditions
if identifier not in carried
]
# Every Operational requirement is verified: by a Check from it, or from a
# Condition it comes from. Serving it with a Capability says who does the
# work; a Check says how anyone would know it is done.
for requirement in chain.requirements(upstream):
identifier = str(requirement["id"])
named = requirement.get("source") or []
named = [named] if isinstance(named, str) else named
if identifier not in carried and not any(
str(source) in carried and str(source) in conditions for source in named
):
problems.append(
f"{name}: Operational requirement {identifier} is verified by no "
"Check: give it one, from the requirement or from a Condition "
"it comes from"
)
problems += check_measures(
graph, scope, capabilities, by_capability, roles, frameworks, name
)

View File

@@ -6,12 +6,14 @@ It costs model usage: run it deliberately.
Usage:
GOOGLE_GENAI_USE_ENTERPRISE=true GOOGLE_CLOUD_PROJECT=<project> GOOGLE_CLOUD_LOCATION=global \\
.venv/bin/python -m eval.operating_model_eval [case-id ...]
.venv/bin/python -m eval.operating_model_eval [--repeat N] [case-id ...]
The model settings are read from the environment or from `.env`. With no case named,
every case in eval/datasets/cases.json runs, one after another. Results are written
under eval/results/<time>/: each Execution's files, and results.json. The exit code
is non-zero when a case fails a check or runs over its time.
every case in eval/datasets/cases.json runs, one after another; `--repeat N` runs each
N times, so the spread of what is staffed and priced can be seen. Results are written
under eval/results/<time>/: each Execution's files, results.json, spread.json, and
report.md, which holds no path of the machine it ran on and can be published. The
exit code is non-zero when a case fails a check or runs over its time.
"""
from __future__ import annotations
@@ -19,6 +21,7 @@ from __future__ import annotations
import asyncio
import json
import logging
import statistics
import sys
import time
from collections.abc import Mapping, Sequence
@@ -78,15 +81,24 @@ def _tokens(runtime: Any) -> dict[str, dict[str, int]]:
return by_role
def execution_ids(case: Mapping[str, Any], repeat: int) -> list[str]:
"""The Execution of each run of a case: the case's id, or one per repeat."""
if repeat <= 1:
return [str(case["id"])]
return [f"{case['id']}-r{n}" for n in range(1, repeat + 1)]
async def run_case(
case: Mapping[str, Any],
output_root: Path,
*,
settings: Settings | None = None,
runtime: AgentRuntime | None = None,
execution_id: str | None = None,
) -> CaseResult:
"""Run one case to the end and score it. `runtime` replaces the real one, for a
scripted run of the harness itself."""
execution_id = execution_id or str(case["id"])
settings = settings or get_settings()
workflow = build(ROOT)
started = time.monotonic()
@@ -105,7 +117,7 @@ async def run_case(
try:
interaction = await asyncio.wait_for(
assembly.engine.start(
"", pre_approved=True, execution_id=case["id"], **inputs(case)
"", pre_approved=True, execution_id=execution_id, **inputs(case)
),
timeout=limit * 60,
)
@@ -150,17 +162,140 @@ async def run_case(
tokens=_tokens(used),
figures=figures,
warnings=list(state.get("warnings") or []),
folder=str(
output_root
/ "deliverables"
# Relative to the run's folder: results are published, and the machine
# they ran on is no part of them.
folder=(
Path("deliverables")
/ "cases"
/ str(state.get("customer_case"))
/ "executions"
/ case["id"]
),
/ execution_id
).as_posix(),
)
# The figures whose spread across repeated runs is reported.
SPREAD = ("people", "cost_per_month", "roles", "checks", "capabilities")
def spread(results: Sequence[CaseResult]) -> dict[str, dict[str, Any]]:
"""For each case run more than once, how far its figures moved: least, most,
mean, and the range as a share of the mean."""
by_case: dict[str, list[CaseResult]] = {}
for result in results:
by_case.setdefault(result.case, []).append(result)
found: dict[str, dict[str, Any]] = {}
for case, runs in by_case.items():
if len(runs) < 2:
continue
figures: dict[str, Any] = {"runs": len(runs)}
for name in SPREAD:
values = [
r.figures[name]
for r in runs
if isinstance(r.figures.get(name), (int, float))
]
if len(values) < 2:
continue
mean = statistics.fmean(values)
figures[name] = {
"min": min(values),
"max": max(values),
"mean": round(mean, 1),
"range_pct": round((max(values) - min(values)) / mean * 100, 1)
if mean
else 0.0,
}
found[case] = figures
return found
def markdown(
results: Sequence[CaseResult], spreads: Mapping[str, Mapping[str, Any]], stamp: str
) -> str:
"""The run, as a report that can be published."""
lines = [
"# Target operating model agent: evaluation",
"",
(
f"Run `{stamp}`. Each run is one pre-approved Execution with real models "
"on a benchmark case. Every case is fictional; no customer's data is in it."
),
"",
"## Results",
"",
(
"| Case | Run | Result | Status | Minutes | Revisions "
"| Requirements verified | Roles | People | Cost a month |"
),
"|---|---|---|---|---|---|---|---|---|---|",
]
seen: dict[str, int] = {}
for result in results:
seen[result.case] = seen.get(result.case, 0) + 1
figures = result.figures
coverage = next(
(
c.detail.split(";")[0]
for c in result.checks
if c.name.startswith("every Operational")
),
"",
)
cost = figures.get("cost_per_month")
lines.append(
f"| {result.case} | {seen[result.case]} | "
f"{'PASS' if result.passed else 'FAIL'} | {result.status} | "
f"{result.minutes} | {sum(result.revisions.values())} | {coverage} | "
f"{figures.get('roles', '')} | {figures.get('people', '')} | "
+ (
f"{cost:,} {figures.get('currency', '')} |"
if isinstance(cost, (int, float))
else " |"
)
)
failed = [(r, c) for r in results for c in r.checks if not c.passed]
lines += ["", "## Checks that failed", ""]
if failed:
lines += [
f"- **{r.case}**: {c.name}" + (f" ({c.detail})" if c.detail else "")
for r, c in failed
]
else:
lines.append("None: every run passed every check.")
if spreads:
lines += [
"",
"## Spread across repeated runs",
"",
"The same case and inputs, run again: how far the operating model moved.",
"",
"| Case | Figure | Least | Most | Mean | Range of the mean |",
"|---|---|---|---|---|---|",
]
for case, measured in spreads.items():
for name in SPREAD:
if name in measured:
f = measured[name]
lines.append(
f"| {case} | {name} | {f['min']} | {f['max']} | {f['mean']} | "
f"{f['range_pct']}% |"
)
return "\n".join(lines) + "\n"
def parse_arguments(arguments: Sequence[str]) -> tuple[list[str], int]:
"""The cases named, and how many times each is to run."""
names, repeat, rest = [], 1, list(arguments)
while rest:
argument = rest.pop(0)
if argument == "--repeat":
repeat = int(rest.pop(0))
else:
names.append(argument)
return names, repeat
def report(results: Sequence[CaseResult]) -> str:
lines = []
for result in results:
@@ -183,7 +318,8 @@ def report(results: Sequence[CaseResult]) -> str:
async def main(arguments: Sequence[str]) -> int:
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(message)s")
cases = load_cases(arguments)
wanted, repeat = parse_arguments(arguments)
cases = load_cases(wanted)
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
output_root = RESULTS / stamp
output_root.mkdir(parents=True, exist_ok=True)
@@ -193,12 +329,24 @@ async def main(arguments: Sequence[str]) -> int:
)
results = []
for case in cases:
logger.info("Running %s", case["id"])
results.append(await run_case(case, output_root, settings=settings))
(output_root / "results.json").write_text(
json.dumps([result.as_dict() for result in results], indent=2),
encoding="utf-8",
)
for execution_id in execution_ids(case, repeat):
logger.info("Running %s as %s", case["id"], execution_id)
results.append(
await run_case(
case, output_root, settings=settings, execution_id=execution_id
)
)
spreads = spread(results)
(output_root / "results.json").write_text(
json.dumps([result.as_dict() for result in results], indent=2),
encoding="utf-8",
)
(output_root / "spread.json").write_text(
json.dumps(spreads, indent=2), encoding="utf-8"
)
(output_root / "report.md").write_text(
markdown(results, spreads, stamp), encoding="utf-8"
)
print(report(results))
print(f"\nWritten to {output_root}")
return 0 if all(result.passed for result in results) else 1

View File

@@ -9,7 +9,7 @@ against one, and is not built.
from __future__ import annotations
import json
from collections.abc import Mapping
from collections.abc import Mapping, Sequence
from dataclasses import asdict, dataclass, field
from typing import Any
@@ -61,6 +61,26 @@ def _attachment(deliverables: Mapping[str, Any], kind: str, name: str) -> Any:
return None
def unverified(
requirements: Sequence[Mapping[str, Any]], model: Mapping[str, Any]
) -> list[str]:
"""The Operational requirements no Check verifies: none has the requirement, or
a Condition it comes from, as its source."""
carried = {str(check.get("source")) for check in model.get("checks") or []}
missing = []
for requirement in requirements:
named = requirement.get("source") or []
named = [named] if isinstance(named, str) else named
# A Check's source is a requirement or a Condition (the graph refuses
# anything else), so a source of the requirement that a Check carries is a
# Condition it comes from.
if str(requirement.get("id")) not in carried and not any(
str(source) in carried for source in named
):
missing.append(str(requirement.get("id")))
return missing
def score(
workflow: Workflow,
state: Mapping[str, Any],
@@ -155,6 +175,21 @@ def score(
_attachment(deliverables, TARGET_OPERATING_MODEL, "target-operating-model.json")
or {}
)
requirements = (
_attachment(
deliverables, "operational_requirements", "operational-requirements.json"
)
or []
)
missing = unverified(requirements, model)
checks.append(
Check(
"every Operational requirement is verified by a Check",
bool(requirements) and not missing,
f"{len(requirements) - len(missing)} of {len(requirements)}"
+ (f"; not verified: {', '.join(missing)}" if missing else ""),
)
)
figures = {
"items": len(spec.items(scope)),
"conditions": len(spec.conditions(scope)),
@@ -178,6 +213,7 @@ def score(
"parties": len(model.get("parties") or []),
"roles": len(model.get("roles") or []),
"checks": len(own),
"requirements_unverified": len(missing),
"checks_not_verifiable": sum(
1 for v in own if v.get("verdict") == "not_verifiable"
),

View File

@@ -33,7 +33,7 @@ verdicts.json a verdict on every OC id
| --- | --- |
| `operational-requirements.json` | A requirement has no `source`, or names one that is not a stated requirement, a Condition or a settled field. A stated requirement or a binding field is carried by no requirement. |
| `capabilities.json` | A Capability names no catalogue entry and is not `custom` with a reason. It serves nothing, or names an id that does not exist. An Item is covered by no Capability. A requirement is served by none. A Capability that responds to faults is staffed for narrower hours than faults are responded to. Its effort is missing, or has no basis. |
| `target-operating-model.json` | A Role is in no Party, or in a country or at a level the rate card lacks, or outside the countries the customer allows. A Capability has no accountable Role, or more than one, or no responsible Role, or none that works its hours. A Condition is carried by no Check. A Check is on, or sourced from, something that does not exist. |
| `target-operating-model.json` | A Role is in no Party, or in a country or at a level the rate card lacks, or outside the countries the customer allows. A Capability has no accountable Role, or more than one, or no responsible Role, or none that works its hours. A Condition is carried by no Check. An Operational requirement is verified by no Check, from it or from a Condition it comes from. A Check is on, or sourced from, something that does not exist. |
| measures, in `target-operating-model.json` | A business outcome the customer stated is shown by no Measure. A Measure shows an outcome that was not stated, or is taken by a Capability nobody is responsible for. Where the model measures how work flows, the outcome, guardrail and leading layers are not all measured. |
| `findings.json` | A finding lacks `lacks`, `proposed` or `needed_by`, names no Item of the solution under `about`, or names an id that does not exist. |
| `verdicts.json` | A Check has no verdict. A Check that depends on a finding passes: it is `not_verifiable` while the finding stands, or `fail` when the validator judges the operating model could meet it itself, such as by staffing or hours. |

View File

@@ -232,6 +232,18 @@ GRAPH: dict[str, list[dict[str, Any]]] = {
"source": "OR-2",
"condition": "No change to the network is applied without the platform owner's approval.",
},
{
"id": "OC-5",
"on": "C-1",
"source": "OR-4",
"condition": "A recovery of the hub network is rehearsed each quarter within 4 hours.",
},
{
"id": "OC-6",
"on": "C-3",
"source": "OR-5",
"condition": "Every Role is placed in a country the customer allows.",
},
],
}

View File

@@ -1,12 +1,15 @@
"""The evaluation harness itself, run on the scripted runtime: no model is called."""
import asyncio
import json
from execution_engine.adk.fake_runtime import FakeAgentRuntime
from execution_engine.config import Settings
from eval import operating_model_eval as harness
from tests.support import SPEC
from eval import scoring
from eval.scoring import CaseResult, Check
from tests.support import GRAPH, REQUIREMENTS, SPEC
from tests.test_execution import turns
@@ -67,3 +70,75 @@ def test_the_benchmark_spec_is_complete_enough_to_run_with_nobody_to_ask():
assert scope.gaps(ESSENTIAL) == []
assert scope.customer_case == "bench-landing-zone-shared"
def test_a_sound_execution_verifies_every_requirement_and_is_kept_relatively(
tmp_path, monkeypatch
):
result = asyncio.run(run(tmp_path, turns(), monkeypatch))
coverage = next(c for c in result.checks if c.name.startswith("every Operational"))
assert coverage.passed, coverage.detail
assert result.figures["requirements_unverified"] == 0
# Results are published: no path of the machine they ran on.
assert (
result.folder == "deliverables/cases/northwind-landing-zone/executions/scripted"
)
assert str(tmp_path) not in json.dumps(result.as_dict())
def test_a_requirement_no_check_verifies_is_named():
model = {"checks": [c for c in GRAPH["checks"] if c["id"] != "OC-5"]}
assert scoring.unverified(REQUIREMENTS, model) == ["OR-4"]
assert scoring.unverified(REQUIREMENTS, GRAPH) == []
def result(case: str, people: float, cost: int, passed: bool = True) -> CaseResult:
return CaseResult(
case=case,
status="completed",
checks=[Check("completed", passed)],
figures={
"people": people,
"cost_per_month": cost,
"roles": 5,
"checks": 9,
"capabilities": 17,
"currency": "GBP",
},
)
def test_the_spread_of_repeated_runs_is_reported():
spread = harness.spread(
[result("a", 10.0, 50000), result("a", 12.0, 70000), result("b", 5.0, 1)]
)
assert set(spread) == {"a"} # one run has no spread
assert spread["a"]["runs"] == 2
assert spread["a"]["cost_per_month"] == {
"min": 50000,
"max": 70000,
"mean": 60000.0,
"range_pct": 33.3,
}
assert spread["a"]["people"]["range_pct"] == 18.2
def test_the_report_can_be_published():
results = [result("a", 10.0, 50000), result("a", 12.0, 70000, passed=False)]
text = harness.markdown(results, harness.spread(results), "20261021T120000Z")
assert text.startswith("# Target operating model agent: evaluation")
assert "| a | 1 | PASS |" in text and "| a | 2 | FAIL |" in text
assert "Spread across repeated runs" in text
assert "/home/" not in text
def test_runs_are_repeated_under_their_own_ids():
assert harness.parse_arguments(["--repeat", "3", "case-x"]) == (["case-x"], 3)
assert harness.parse_arguments([]) == ([], 1)
assert harness.execution_ids({"id": "case-x"}, 1) == ["case-x"]
assert harness.execution_ids({"id": "case-x"}, 2) == ["case-x-r1", "case-x-r2"]

View File

@@ -92,6 +92,10 @@ def test_a_graph_that_holds_together_is_accepted():
"Capability C-1 is needed 24x7, and no Role responsible for it works those hours",
),
(lambda g: g["checks"].pop(1), "Condition K-1 is carried by no Check"),
(
lambda g: g["checks"].pop(0),
"Operational requirement OR-1 is verified by no Check",
),
(
lambda g: g["checks"][0].update(on="C-9"),
"Check OC-1 is on C-9, which is not",
@@ -119,6 +123,14 @@ def test_a_graph_that_does_not_hold_together_is_refused(change, problem):
assert any(problem in fault for fault in found), found
def test_a_requirement_is_verified_through_the_conditions_it_comes_from():
# OR-3 comes from K-1 and K-2: the Checks carrying them verify it.
changed = graph()
assert not any(c["source"] == "OR-3" for c in changed["checks"])
assert not any("OR-3" in fault for fault in faults(changed))
def test_a_file_that_is_not_a_graph_is_named():
assert "not valid JSON" in check({GRAPH_FILE: "{"}, upstream())[0]
assert "must be an object with" in check({GRAPH_FILE: "[]"}, upstream())[0]

View File

@@ -133,7 +133,7 @@ def test_the_scenarios_verdicts_are_written_into_the_validators_file_by_code():
"OC-3",
"OC-4",
]
assert len(merged["checks"]) == 14
assert len(merged["checks"]) == 16
# Merging again replaces the scenarios' verdicts; it never adds them twice.
again = json.loads(scenarios.merge(json.dumps(merged), found))
assert again == merged
@@ -165,7 +165,7 @@ def test_a_verdict_on_every_check_is_accepted():
def test_a_check_without_a_verdict_is_refused():
missing = {"checks": VERDICTS["checks"][:-1]}
assert verdict_faults(missing) == ["verdicts.json: Check OC-4 has no verdict"]
assert verdict_faults(missing) == ["verdicts.json: Check OC-6 has no verdict"]
def test_a_check_that_waits_on_a_finding_cannot_pass():