feat: every Operational requirement verified by a Check; eval spread and report
The operating model is refused while an Operational requirement is verified by no Check, from it or from a Condition it comes from; the architect is told so, and the operating-chain Skill lists it. Six of nine earlier runs left a requirement unverified. The evaluation checks it too, runs each case N times (--repeat), reports the spread of people, cost, Roles, Checks and Capabilities (spread.json), and writes report.md to publish. Result folders are relative to the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -29,3 +29,4 @@
|
||||
| T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) |
|
||||
| T13 | Staffing and cost in code | built (`steps/step-1.md`) |
|
||||
| T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) |
|
||||
| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); runs in progress |
|
||||
|
||||
@@ -65,3 +65,14 @@ It also filed a gap in its own model as a finding: service continuity staffed on
|
||||
- **The architect is told** what a finding is and is not.
|
||||
|
||||
177 tests pass. Not yet run with real models.
|
||||
|
||||
## Case 02 after findings were kept to the solution, 2026-10-14 16:28
|
||||
|
||||
Recorded 2026-10-21 from `eval/results/20261014T162852Z/`. Case 02 passed every check in 15.1 minutes: 9 Items, 16 Operational requirements, 21 Capabilities, 11 Roles, 13 Checks, 33 of 33 scenarios, no findings, 3 business outcomes and 6 Measures, 12.8 people at 92,903 GBP a month, one Revision of the operating model.
|
||||
|
||||
Read against the runs before it, two things the checks did not show:
|
||||
|
||||
- **Cost moved 55% between two runs of the same case** (59,997 GBP at 13:26, 92,903 GBP at 16:28), with 6 Roles against 11. Staffing and cost are computed, so the spread is the model's design, not the arithmetic. Nothing measured it.
|
||||
- **Three Operational requirements were verified by no Check** (OR-14 the sourcing model, OR-15 the delivery countries, OR-16 the tooling), each served by a Capability. Six of the nine runs of 2026-10-13 and 14 left at least one so. The design required each requirement to be served by a Capability, not verified by a Check, and Checks fell from 31 to 36 a run to 13 or 14 once business outcomes came in.
|
||||
|
||||
Both are taken up in step 4.
|
||||
|
||||
21
.agents/plans/iter-01-first-slice/steps/step-4.md
Normal file
21
.agents/plans/iter-01-first-slice/steps/step-4.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Step 4: every requirement verified, and the spread of repeated runs
|
||||
|
||||
Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, after reading the run of 2026-10-14 16:28 (`step-3.md`).
|
||||
|
||||
## What was built
|
||||
|
||||
- **Every Operational requirement is verified by a Check** (`graph.py`): one whose `source` is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the `operating-chain` Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks.
|
||||
- **The evaluation:**
|
||||
- a check that every Operational requirement is verified, naming those that are not, and `requirements_unverified` among the figures;
|
||||
- `--repeat N`: each case run N times under its own Execution, `<case>-r<n>`;
|
||||
- `spread.json`: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;
|
||||
- `report.md`: the run as a report that can be published, with each run's result, the checks that failed and the spread;
|
||||
- each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.
|
||||
|
||||
## Verified
|
||||
|
||||
184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.
|
||||
|
||||
## The runs
|
||||
|
||||
Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one.
|
||||
@@ -62,6 +62,7 @@ Checks:
|
||||
|
||||
- A Check is a condition that can be shown to hold or not, on a Capability, a Role or a Responsibility. Its `source` is the Operational requirement or the Condition it comes from.
|
||||
- Every Condition in `scope.stated.solution.conditions` must be carried by at least one Check whose `source` is that Condition's id.
|
||||
- Every Operational requirement must be verified by at least one Check: one whose `source` is the requirement's id, or the id of a Condition the requirement comes from. Organisation and tooling requirements need one too: a requirement that staff work only from certain countries is verified by a Check on the Roles' countries, one that tooling is integrated by a Check on the Capability that runs it.
|
||||
- Start from the catalogue's `typical_checks` and rewrite each for this solution, with the customer's own numbers.
|
||||
- Put a Check on the Capability that does the work it describes. A Check that service is restored within four hours is on the Capability that restores it, not on the one that reports service levels.
|
||||
|
||||
|
||||
@@ -18,7 +18,8 @@
|
||||
The Capabilities are the capability model's, by id. A submission is refused while
|
||||
the graph does not hold together, so what the customer approves is complete: every
|
||||
Capability has one Role answerable for it and someone to do it at the hours it is
|
||||
needed, and everything the customer said must be kept true is carried by a Check.
|
||||
needed, everything the customer said must be kept true is carried by a Check, and
|
||||
every Operational requirement is verified by one.
|
||||
|
||||
The responsibility matrix, the organisation chart and the staffing and cost are
|
||||
written from the graph by code, as views of it.
|
||||
@@ -301,6 +302,21 @@ def check(card: RateCard, frameworks: Collection[str] = ()) -> chain.DataCheck:
|
||||
for identifier in conditions
|
||||
if identifier not in carried
|
||||
]
|
||||
# Every Operational requirement is verified: by a Check from it, or from a
|
||||
# Condition it comes from. Serving it with a Capability says who does the
|
||||
# work; a Check says how anyone would know it is done.
|
||||
for requirement in chain.requirements(upstream):
|
||||
identifier = str(requirement["id"])
|
||||
named = requirement.get("source") or []
|
||||
named = [named] if isinstance(named, str) else named
|
||||
if identifier not in carried and not any(
|
||||
str(source) in carried and str(source) in conditions for source in named
|
||||
):
|
||||
problems.append(
|
||||
f"{name}: Operational requirement {identifier} is verified by no "
|
||||
"Check: give it one, from the requirement or from a Condition "
|
||||
"it comes from"
|
||||
)
|
||||
problems += check_measures(
|
||||
graph, scope, capabilities, by_capability, roles, frameworks, name
|
||||
)
|
||||
|
||||
@@ -6,12 +6,14 @@ It costs model usage: run it deliberately.
|
||||
|
||||
Usage:
|
||||
GOOGLE_GENAI_USE_ENTERPRISE=true GOOGLE_CLOUD_PROJECT=<project> GOOGLE_CLOUD_LOCATION=global \\
|
||||
.venv/bin/python -m eval.operating_model_eval [case-id ...]
|
||||
.venv/bin/python -m eval.operating_model_eval [--repeat N] [case-id ...]
|
||||
|
||||
The model settings are read from the environment or from `.env`. With no case named,
|
||||
every case in eval/datasets/cases.json runs, one after another. Results are written
|
||||
under eval/results/<time>/: each Execution's files, and results.json. The exit code
|
||||
is non-zero when a case fails a check or runs over its time.
|
||||
every case in eval/datasets/cases.json runs, one after another; `--repeat N` runs each
|
||||
N times, so the spread of what is staffed and priced can be seen. Results are written
|
||||
under eval/results/<time>/: each Execution's files, results.json, spread.json, and
|
||||
report.md, which holds no path of the machine it ran on and can be published. The
|
||||
exit code is non-zero when a case fails a check or runs over its time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -19,6 +21,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import statistics
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Mapping, Sequence
|
||||
@@ -78,15 +81,24 @@ def _tokens(runtime: Any) -> dict[str, dict[str, int]]:
|
||||
return by_role
|
||||
|
||||
|
||||
def execution_ids(case: Mapping[str, Any], repeat: int) -> list[str]:
|
||||
"""The Execution of each run of a case: the case's id, or one per repeat."""
|
||||
if repeat <= 1:
|
||||
return [str(case["id"])]
|
||||
return [f"{case['id']}-r{n}" for n in range(1, repeat + 1)]
|
||||
|
||||
|
||||
async def run_case(
|
||||
case: Mapping[str, Any],
|
||||
output_root: Path,
|
||||
*,
|
||||
settings: Settings | None = None,
|
||||
runtime: AgentRuntime | None = None,
|
||||
execution_id: str | None = None,
|
||||
) -> CaseResult:
|
||||
"""Run one case to the end and score it. `runtime` replaces the real one, for a
|
||||
scripted run of the harness itself."""
|
||||
execution_id = execution_id or str(case["id"])
|
||||
settings = settings or get_settings()
|
||||
workflow = build(ROOT)
|
||||
started = time.monotonic()
|
||||
@@ -105,7 +117,7 @@ async def run_case(
|
||||
try:
|
||||
interaction = await asyncio.wait_for(
|
||||
assembly.engine.start(
|
||||
"", pre_approved=True, execution_id=case["id"], **inputs(case)
|
||||
"", pre_approved=True, execution_id=execution_id, **inputs(case)
|
||||
),
|
||||
timeout=limit * 60,
|
||||
)
|
||||
@@ -150,17 +162,140 @@ async def run_case(
|
||||
tokens=_tokens(used),
|
||||
figures=figures,
|
||||
warnings=list(state.get("warnings") or []),
|
||||
folder=str(
|
||||
output_root
|
||||
/ "deliverables"
|
||||
# Relative to the run's folder: results are published, and the machine
|
||||
# they ran on is no part of them.
|
||||
folder=(
|
||||
Path("deliverables")
|
||||
/ "cases"
|
||||
/ str(state.get("customer_case"))
|
||||
/ "executions"
|
||||
/ case["id"]
|
||||
),
|
||||
/ execution_id
|
||||
).as_posix(),
|
||||
)
|
||||
|
||||
|
||||
# The figures whose spread across repeated runs is reported.
|
||||
SPREAD = ("people", "cost_per_month", "roles", "checks", "capabilities")
|
||||
|
||||
|
||||
def spread(results: Sequence[CaseResult]) -> dict[str, dict[str, Any]]:
|
||||
"""For each case run more than once, how far its figures moved: least, most,
|
||||
mean, and the range as a share of the mean."""
|
||||
by_case: dict[str, list[CaseResult]] = {}
|
||||
for result in results:
|
||||
by_case.setdefault(result.case, []).append(result)
|
||||
found: dict[str, dict[str, Any]] = {}
|
||||
for case, runs in by_case.items():
|
||||
if len(runs) < 2:
|
||||
continue
|
||||
figures: dict[str, Any] = {"runs": len(runs)}
|
||||
for name in SPREAD:
|
||||
values = [
|
||||
r.figures[name]
|
||||
for r in runs
|
||||
if isinstance(r.figures.get(name), (int, float))
|
||||
]
|
||||
if len(values) < 2:
|
||||
continue
|
||||
mean = statistics.fmean(values)
|
||||
figures[name] = {
|
||||
"min": min(values),
|
||||
"max": max(values),
|
||||
"mean": round(mean, 1),
|
||||
"range_pct": round((max(values) - min(values)) / mean * 100, 1)
|
||||
if mean
|
||||
else 0.0,
|
||||
}
|
||||
found[case] = figures
|
||||
return found
|
||||
|
||||
|
||||
def markdown(
|
||||
results: Sequence[CaseResult], spreads: Mapping[str, Mapping[str, Any]], stamp: str
|
||||
) -> str:
|
||||
"""The run, as a report that can be published."""
|
||||
lines = [
|
||||
"# Target operating model agent: evaluation",
|
||||
"",
|
||||
(
|
||||
f"Run `{stamp}`. Each run is one pre-approved Execution with real models "
|
||||
"on a benchmark case. Every case is fictional; no customer's data is in it."
|
||||
),
|
||||
"",
|
||||
"## Results",
|
||||
"",
|
||||
(
|
||||
"| Case | Run | Result | Status | Minutes | Revisions "
|
||||
"| Requirements verified | Roles | People | Cost a month |"
|
||||
),
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
seen: dict[str, int] = {}
|
||||
for result in results:
|
||||
seen[result.case] = seen.get(result.case, 0) + 1
|
||||
figures = result.figures
|
||||
coverage = next(
|
||||
(
|
||||
c.detail.split(";")[0]
|
||||
for c in result.checks
|
||||
if c.name.startswith("every Operational")
|
||||
),
|
||||
"",
|
||||
)
|
||||
cost = figures.get("cost_per_month")
|
||||
lines.append(
|
||||
f"| {result.case} | {seen[result.case]} | "
|
||||
f"{'PASS' if result.passed else 'FAIL'} | {result.status} | "
|
||||
f"{result.minutes} | {sum(result.revisions.values())} | {coverage} | "
|
||||
f"{figures.get('roles', '')} | {figures.get('people', '')} | "
|
||||
+ (
|
||||
f"{cost:,} {figures.get('currency', '')} |"
|
||||
if isinstance(cost, (int, float))
|
||||
else " |"
|
||||
)
|
||||
)
|
||||
failed = [(r, c) for r in results for c in r.checks if not c.passed]
|
||||
lines += ["", "## Checks that failed", ""]
|
||||
if failed:
|
||||
lines += [
|
||||
f"- **{r.case}**: {c.name}" + (f" ({c.detail})" if c.detail else "")
|
||||
for r, c in failed
|
||||
]
|
||||
else:
|
||||
lines.append("None: every run passed every check.")
|
||||
if spreads:
|
||||
lines += [
|
||||
"",
|
||||
"## Spread across repeated runs",
|
||||
"",
|
||||
"The same case and inputs, run again: how far the operating model moved.",
|
||||
"",
|
||||
"| Case | Figure | Least | Most | Mean | Range of the mean |",
|
||||
"|---|---|---|---|---|---|",
|
||||
]
|
||||
for case, measured in spreads.items():
|
||||
for name in SPREAD:
|
||||
if name in measured:
|
||||
f = measured[name]
|
||||
lines.append(
|
||||
f"| {case} | {name} | {f['min']} | {f['max']} | {f['mean']} | "
|
||||
f"{f['range_pct']}% |"
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def parse_arguments(arguments: Sequence[str]) -> tuple[list[str], int]:
|
||||
"""The cases named, and how many times each is to run."""
|
||||
names, repeat, rest = [], 1, list(arguments)
|
||||
while rest:
|
||||
argument = rest.pop(0)
|
||||
if argument == "--repeat":
|
||||
repeat = int(rest.pop(0))
|
||||
else:
|
||||
names.append(argument)
|
||||
return names, repeat
|
||||
|
||||
|
||||
def report(results: Sequence[CaseResult]) -> str:
|
||||
lines = []
|
||||
for result in results:
|
||||
@@ -183,7 +318,8 @@ def report(results: Sequence[CaseResult]) -> str:
|
||||
|
||||
async def main(arguments: Sequence[str]) -> int:
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(message)s")
|
||||
cases = load_cases(arguments)
|
||||
wanted, repeat = parse_arguments(arguments)
|
||||
cases = load_cases(wanted)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
|
||||
output_root = RESULTS / stamp
|
||||
output_root.mkdir(parents=True, exist_ok=True)
|
||||
@@ -193,12 +329,24 @@ async def main(arguments: Sequence[str]) -> int:
|
||||
)
|
||||
results = []
|
||||
for case in cases:
|
||||
logger.info("Running %s", case["id"])
|
||||
results.append(await run_case(case, output_root, settings=settings))
|
||||
(output_root / "results.json").write_text(
|
||||
json.dumps([result.as_dict() for result in results], indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
for execution_id in execution_ids(case, repeat):
|
||||
logger.info("Running %s as %s", case["id"], execution_id)
|
||||
results.append(
|
||||
await run_case(
|
||||
case, output_root, settings=settings, execution_id=execution_id
|
||||
)
|
||||
)
|
||||
spreads = spread(results)
|
||||
(output_root / "results.json").write_text(
|
||||
json.dumps([result.as_dict() for result in results], indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(output_root / "spread.json").write_text(
|
||||
json.dumps(spreads, indent=2), encoding="utf-8"
|
||||
)
|
||||
(output_root / "report.md").write_text(
|
||||
markdown(results, spreads, stamp), encoding="utf-8"
|
||||
)
|
||||
print(report(results))
|
||||
print(f"\nWritten to {output_root}")
|
||||
return 0 if all(result.passed for result in results) else 1
|
||||
|
||||
@@ -9,7 +9,7 @@ against one, and is not built.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from typing import Any
|
||||
|
||||
@@ -61,6 +61,26 @@ def _attachment(deliverables: Mapping[str, Any], kind: str, name: str) -> Any:
|
||||
return None
|
||||
|
||||
|
||||
def unverified(
|
||||
requirements: Sequence[Mapping[str, Any]], model: Mapping[str, Any]
|
||||
) -> list[str]:
|
||||
"""The Operational requirements no Check verifies: none has the requirement, or
|
||||
a Condition it comes from, as its source."""
|
||||
carried = {str(check.get("source")) for check in model.get("checks") or []}
|
||||
missing = []
|
||||
for requirement in requirements:
|
||||
named = requirement.get("source") or []
|
||||
named = [named] if isinstance(named, str) else named
|
||||
# A Check's source is a requirement or a Condition (the graph refuses
|
||||
# anything else), so a source of the requirement that a Check carries is a
|
||||
# Condition it comes from.
|
||||
if str(requirement.get("id")) not in carried and not any(
|
||||
str(source) in carried for source in named
|
||||
):
|
||||
missing.append(str(requirement.get("id")))
|
||||
return missing
|
||||
|
||||
|
||||
def score(
|
||||
workflow: Workflow,
|
||||
state: Mapping[str, Any],
|
||||
@@ -155,6 +175,21 @@ def score(
|
||||
_attachment(deliverables, TARGET_OPERATING_MODEL, "target-operating-model.json")
|
||||
or {}
|
||||
)
|
||||
requirements = (
|
||||
_attachment(
|
||||
deliverables, "operational_requirements", "operational-requirements.json"
|
||||
)
|
||||
or []
|
||||
)
|
||||
missing = unverified(requirements, model)
|
||||
checks.append(
|
||||
Check(
|
||||
"every Operational requirement is verified by a Check",
|
||||
bool(requirements) and not missing,
|
||||
f"{len(requirements) - len(missing)} of {len(requirements)}"
|
||||
+ (f"; not verified: {', '.join(missing)}" if missing else ""),
|
||||
)
|
||||
)
|
||||
figures = {
|
||||
"items": len(spec.items(scope)),
|
||||
"conditions": len(spec.conditions(scope)),
|
||||
@@ -178,6 +213,7 @@ def score(
|
||||
"parties": len(model.get("parties") or []),
|
||||
"roles": len(model.get("roles") or []),
|
||||
"checks": len(own),
|
||||
"requirements_unverified": len(missing),
|
||||
"checks_not_verifiable": sum(
|
||||
1 for v in own if v.get("verdict") == "not_verifiable"
|
||||
),
|
||||
|
||||
@@ -33,7 +33,7 @@ verdicts.json a verdict on every OC id
|
||||
| --- | --- |
|
||||
| `operational-requirements.json` | A requirement has no `source`, or names one that is not a stated requirement, a Condition or a settled field. A stated requirement or a binding field is carried by no requirement. |
|
||||
| `capabilities.json` | A Capability names no catalogue entry and is not `custom` with a reason. It serves nothing, or names an id that does not exist. An Item is covered by no Capability. A requirement is served by none. A Capability that responds to faults is staffed for narrower hours than faults are responded to. Its effort is missing, or has no basis. |
|
||||
| `target-operating-model.json` | A Role is in no Party, or in a country or at a level the rate card lacks, or outside the countries the customer allows. A Capability has no accountable Role, or more than one, or no responsible Role, or none that works its hours. A Condition is carried by no Check. A Check is on, or sourced from, something that does not exist. |
|
||||
| `target-operating-model.json` | A Role is in no Party, or in a country or at a level the rate card lacks, or outside the countries the customer allows. A Capability has no accountable Role, or more than one, or no responsible Role, or none that works its hours. A Condition is carried by no Check. An Operational requirement is verified by no Check, from it or from a Condition it comes from. A Check is on, or sourced from, something that does not exist. |
|
||||
| measures, in `target-operating-model.json` | A business outcome the customer stated is shown by no Measure. A Measure shows an outcome that was not stated, or is taken by a Capability nobody is responsible for. Where the model measures how work flows, the outcome, guardrail and leading layers are not all measured. |
|
||||
| `findings.json` | A finding lacks `lacks`, `proposed` or `needed_by`, names no Item of the solution under `about`, or names an id that does not exist. |
|
||||
| `verdicts.json` | A Check has no verdict. A Check that depends on a finding passes: it is `not_verifiable` while the finding stands, or `fail` when the validator judges the operating model could meet it itself, such as by staffing or hours. |
|
||||
|
||||
@@ -232,6 +232,18 @@ GRAPH: dict[str, list[dict[str, Any]]] = {
|
||||
"source": "OR-2",
|
||||
"condition": "No change to the network is applied without the platform owner's approval.",
|
||||
},
|
||||
{
|
||||
"id": "OC-5",
|
||||
"on": "C-1",
|
||||
"source": "OR-4",
|
||||
"condition": "A recovery of the hub network is rehearsed each quarter within 4 hours.",
|
||||
},
|
||||
{
|
||||
"id": "OC-6",
|
||||
"on": "C-3",
|
||||
"source": "OR-5",
|
||||
"condition": "Every Role is placed in a country the customer allows.",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@@ -1,12 +1,15 @@
|
||||
"""The evaluation harness itself, run on the scripted runtime: no model is called."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from execution_engine.adk.fake_runtime import FakeAgentRuntime
|
||||
from execution_engine.config import Settings
|
||||
|
||||
from eval import operating_model_eval as harness
|
||||
from tests.support import SPEC
|
||||
from eval import scoring
|
||||
from eval.scoring import CaseResult, Check
|
||||
from tests.support import GRAPH, REQUIREMENTS, SPEC
|
||||
from tests.test_execution import turns
|
||||
|
||||
|
||||
@@ -67,3 +70,75 @@ def test_the_benchmark_spec_is_complete_enough_to_run_with_nobody_to_ask():
|
||||
|
||||
assert scope.gaps(ESSENTIAL) == []
|
||||
assert scope.customer_case == "bench-landing-zone-shared"
|
||||
|
||||
|
||||
def test_a_sound_execution_verifies_every_requirement_and_is_kept_relatively(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
result = asyncio.run(run(tmp_path, turns(), monkeypatch))
|
||||
|
||||
coverage = next(c for c in result.checks if c.name.startswith("every Operational"))
|
||||
assert coverage.passed, coverage.detail
|
||||
assert result.figures["requirements_unverified"] == 0
|
||||
# Results are published: no path of the machine they ran on.
|
||||
assert (
|
||||
result.folder == "deliverables/cases/northwind-landing-zone/executions/scripted"
|
||||
)
|
||||
assert str(tmp_path) not in json.dumps(result.as_dict())
|
||||
|
||||
|
||||
def test_a_requirement_no_check_verifies_is_named():
|
||||
model = {"checks": [c for c in GRAPH["checks"] if c["id"] != "OC-5"]}
|
||||
|
||||
assert scoring.unverified(REQUIREMENTS, model) == ["OR-4"]
|
||||
assert scoring.unverified(REQUIREMENTS, GRAPH) == []
|
||||
|
||||
|
||||
def result(case: str, people: float, cost: int, passed: bool = True) -> CaseResult:
|
||||
return CaseResult(
|
||||
case=case,
|
||||
status="completed",
|
||||
checks=[Check("completed", passed)],
|
||||
figures={
|
||||
"people": people,
|
||||
"cost_per_month": cost,
|
||||
"roles": 5,
|
||||
"checks": 9,
|
||||
"capabilities": 17,
|
||||
"currency": "GBP",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def test_the_spread_of_repeated_runs_is_reported():
|
||||
spread = harness.spread(
|
||||
[result("a", 10.0, 50000), result("a", 12.0, 70000), result("b", 5.0, 1)]
|
||||
)
|
||||
|
||||
assert set(spread) == {"a"} # one run has no spread
|
||||
assert spread["a"]["runs"] == 2
|
||||
assert spread["a"]["cost_per_month"] == {
|
||||
"min": 50000,
|
||||
"max": 70000,
|
||||
"mean": 60000.0,
|
||||
"range_pct": 33.3,
|
||||
}
|
||||
assert spread["a"]["people"]["range_pct"] == 18.2
|
||||
|
||||
|
||||
def test_the_report_can_be_published():
|
||||
results = [result("a", 10.0, 50000), result("a", 12.0, 70000, passed=False)]
|
||||
|
||||
text = harness.markdown(results, harness.spread(results), "20261021T120000Z")
|
||||
|
||||
assert text.startswith("# Target operating model agent: evaluation")
|
||||
assert "| a | 1 | PASS |" in text and "| a | 2 | FAIL |" in text
|
||||
assert "Spread across repeated runs" in text
|
||||
assert "/home/" not in text
|
||||
|
||||
|
||||
def test_runs_are_repeated_under_their_own_ids():
|
||||
assert harness.parse_arguments(["--repeat", "3", "case-x"]) == (["case-x"], 3)
|
||||
assert harness.parse_arguments([]) == ([], 1)
|
||||
assert harness.execution_ids({"id": "case-x"}, 1) == ["case-x"]
|
||||
assert harness.execution_ids({"id": "case-x"}, 2) == ["case-x-r1", "case-x-r2"]
|
||||
|
||||
@@ -92,6 +92,10 @@ def test_a_graph_that_holds_together_is_accepted():
|
||||
"Capability C-1 is needed 24x7, and no Role responsible for it works those hours",
|
||||
),
|
||||
(lambda g: g["checks"].pop(1), "Condition K-1 is carried by no Check"),
|
||||
(
|
||||
lambda g: g["checks"].pop(0),
|
||||
"Operational requirement OR-1 is verified by no Check",
|
||||
),
|
||||
(
|
||||
lambda g: g["checks"][0].update(on="C-9"),
|
||||
"Check OC-1 is on C-9, which is not",
|
||||
@@ -119,6 +123,14 @@ def test_a_graph_that_does_not_hold_together_is_refused(change, problem):
|
||||
assert any(problem in fault for fault in found), found
|
||||
|
||||
|
||||
def test_a_requirement_is_verified_through_the_conditions_it_comes_from():
|
||||
# OR-3 comes from K-1 and K-2: the Checks carrying them verify it.
|
||||
changed = graph()
|
||||
assert not any(c["source"] == "OR-3" for c in changed["checks"])
|
||||
|
||||
assert not any("OR-3" in fault for fault in faults(changed))
|
||||
|
||||
|
||||
def test_a_file_that_is_not_a_graph_is_named():
|
||||
assert "not valid JSON" in check({GRAPH_FILE: "{"}, upstream())[0]
|
||||
assert "must be an object with" in check({GRAPH_FILE: "[]"}, upstream())[0]
|
||||
|
||||
@@ -133,7 +133,7 @@ def test_the_scenarios_verdicts_are_written_into_the_validators_file_by_code():
|
||||
"OC-3",
|
||||
"OC-4",
|
||||
]
|
||||
assert len(merged["checks"]) == 14
|
||||
assert len(merged["checks"]) == 16
|
||||
# Merging again replaces the scenarios' verdicts; it never adds them twice.
|
||||
again = json.loads(scenarios.merge(json.dumps(merged), found))
|
||||
assert again == merged
|
||||
@@ -165,7 +165,7 @@ def test_a_verdict_on_every_check_is_accepted():
|
||||
def test_a_check_without_a_verdict_is_refused():
|
||||
missing = {"checks": VERDICTS["checks"][:-1]}
|
||||
|
||||
assert verdict_faults(missing) == ["verdicts.json: Check OC-4 has no verdict"]
|
||||
assert verdict_faults(missing) == ["verdicts.json: Check OC-6 has no verdict"]
|
||||
|
||||
|
||||
def test_a_check_that_waits_on_a_finding_cannot_pass():
|
||||
|
||||
Reference in New Issue
Block a user