Files
tom_analysis_agent/.agents/plans/iter-01-first-slice/steps/step-4.md
Jonathan Boniface 6fdac22876 feat: every Operational requirement verified by a Check; eval spread and report
The operating model is refused while an Operational requirement is verified
by no Check, from it or from a Condition it comes from; the architect is
told so, and the operating-chain Skill lists it. Six of nine earlier runs
left a requirement unverified.

The evaluation checks it too, runs each case N times (--repeat), reports
the spread of people, cost, Roles, Checks and Capabilities (spread.json),
and writes report.md to publish. Result folders are relative to the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 21:32:05 +01:00

1.6 KiB

Step 4: every requirement verified, and the spread of repeated runs

Built 2026-10-21 on feat/eval-coverage-and-spread, at the owner's request, after reading the run of 2026-10-14 16:28 (step-3.md).

What was built

  • Every Operational requirement is verified by a Check (graph.py): one whose source is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the operating-chain Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks.
  • The evaluation:
    • a check that every Operational requirement is verified, naming those that are not, and requirements_unverified among the figures;
    • --repeat N: each case run N times under its own Execution, <case>-r<n>;
    • spread.json: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;
    • report.md: the run as a report that can be published, with each run's result, the checks that failed and the spread;
    • each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.

Verified

184 tests pass; ruff and mypy clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.

The runs

Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one.