Files
tom_analysis_agent/.agents/plans/iter-01-first-slice/steps/step-4.md

4.4 KiB

Step 4: every requirement verified, and the spread of repeated runs

Built 2026-10-21 on feat/eval-coverage-and-spread, at the owner's request, after reading the run of 2026-10-14 16:28 (step-3.md).

What was built

  • Every Operational requirement is verified by a Check (graph.py): one whose source is the requirement, or a Condition it comes from. A Target operating model that leaves one unverified is refused, naming it. The architect's instructions say so, with how organisation and tooling requirements are verified; the operating-chain Skill lists the refusal. The test fixture had the same gap (OR-4, OR-5) and gained the two Checks.
  • The evaluation:
    • a check that every Operational requirement is verified, naming those that are not, and requirements_unverified among the figures;
    • --repeat N: each case run N times under its own Execution, <case>-r<n>;
    • spread.json: for each case run more than once, least, most, mean and range of the mean, of people, cost a month, Roles, Checks and Capabilities. Reported, not failed;
    • report.md: the run as a report that can be published, with each run's result, the checks that failed and the spread;
    • each result's folder relative to the run's: results are published, and hold no path of the machine they ran on.

Verified

184 tests pass; ruff and mypy clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.

The runs, 2026-10-21

Both cases three times (eval/results/20261021T203624Z/)

Case Run Result Minutes Revisions Requirements verified Roles People Cost a month
01, written spec 1 pass 18.8 1 10 of 10 5 6.8 46,252 GBP
01 2 pass 17.1 1 9 of 9 7 11.0 44,720 GBP
01 3 pass 20.2 1 10 of 10 6 11.4 47,603 GBP
02, digested architecture 1 fail 50.4 5 17 of 17 8 11.7 73,863 GBP
02 2 pass 20.9 1 15 of 15 6 10.8 83,846 GBP
02 3 pass 24.4 2 15 of 15 8 16.9 116,409 GBP
  • Every requirement was verified in every run. Checks rose to 15 to 20 a run, from 13 or 14.
  • Case 02, run 1 ended with warnings at the operating model's revision limit. The validator failed five Checks whose words the Capability they are on does not bear out: "continuously maintained" and "continuously reviewed" on Capabilities staffed in business hours, a 10-week validation on the Capability that manages the roadmap rather than the one that delivers, sign-off the Capability does not mention, and alignment with DORA, PCI DSS and ISO 27001 on a Capability covering PCI DSS alone. Four of the five come from Conditions, not requirements: the weakness of step 3 ("a Check whose hours or Capability did not fit"), not the new rule.
  • The spread: case 01's cost moved 6.2% across its runs while its people moved 47.3%: more people, cheaper. Case 02's cost moved 46.6% (73,863 to 116,409 GBP) and its people 46.4%; Roles 27 to 33%, Checks 17 to 25%; Capabilities not at all (21 each time).

Through the Orchestration engine from the solution architecture agent (eval/results/20261021T203623Z-handoff/)

One pre-approved request for architecture_and_operating_model, Customer case case-08-landing-zone, from the benchmark landing-zone spec, with case 02's operating fields under operating_model. Completed in 74.9 minutes: the architecture in about 50, its 15 kept files given to this agent as sources, which digested them (metadata.source: architecture:6c316f05…) and completed: 24 Operational requirements, all verified, 21 Capabilities, 8 Roles, 26 Checks, 44 of 44 scenarios, 12.2 people at 87,959 GBP a month. Both Executions' files are kept in the Orchestration engine's store (MinIO, 28 objects) and were fetched through their links. The signatures were taken off the links in the results before publishing.

What remains

  • The spread is the larger finding. The same architecture costs between 74 and 116 thousand GBP a month across three runs. The figures are computed; the design is not stable. What moves is the Roles and how work is shared between them.
  • Checks that the Capability they are on cannot bear out end a run at its revision limit about one time in three for case 02. A data check that refuses a "continuous" or 24x7 Check on a Capability whose hours are narrower would catch the commonest of them before validation.