Files
tom_analysis_agent/eval/results/20261021T203624Z/report.md
Jonathan Boniface 628b6ee170 docs: publish the runs of 2026-10-21
The evaluation (both cases three times: report, results, spread, each
run's Deliverables and lesson candidates) and the hand-off through the
Orchestration engine (report, the document of links, both agents' kept
files and records). Benchmark cases only, all fictional; no machine paths,
and the links carry no signatures. Other runs stay ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 09:25:03 +01:00

2.2 KiB

Target operating model agent: evaluation

Run 20261021T203624Z. Each run is one pre-approved Execution with real models on a benchmark case. Every case is fictional; no customer's data is in it.

Results

Case Run Result Status Minutes Revisions Requirements verified Roles People Cost a month
case-01-landing-zone-shared 1 PASS completed 18.8 1 10 of 10 5 6.8 46,252 GBP
case-01-landing-zone-shared 2 PASS completed 17.1 1 9 of 9 7 11.0 44,720 GBP
case-01-landing-zone-shared 3 PASS completed 20.2 1 10 of 10 6 11.4 47,603 GBP
case-02-from-architecture 1 FAIL completed_with_warnings 50.4 5 17 of 17 8 11.7 73,863 GBP
case-02-from-architecture 2 PASS completed 20.9 1 15 of 15 6 10.8 83,846 GBP
case-02-from-architecture 3 PASS completed 24.4 2 15 of 15 8 16.9 116,409 GBP

Checks that failed

  • case-02-from-architecture: completed (completed_with_warnings)
  • case-02-from-architecture: every Deliverable was submitted (operating_model_guide)
  • case-02-from-architecture: no Check failed (OC-2, OC-4, OC-9, OC-12, OC-17)
  • case-02-from-architecture: the guide has its sections
  • case-02-from-architecture: the packager wrote a summary (0 words)

Spread across repeated runs

The same case and inputs, run again: how far the operating model moved.

Case Figure Least Most Mean Range of the mean
case-01-landing-zone-shared people 6.8 11.4 9.7 47.3%
case-01-landing-zone-shared cost_per_month 44720 47603 46191.7 6.2%
case-01-landing-zone-shared roles 5 7 6.0 33.3%
case-01-landing-zone-shared checks 15 19 16.3 24.5%
case-01-landing-zone-shared capabilities 21 21 21.0 0.0%
case-02-from-architecture people 10.8 16.9 13.1 46.4%
case-02-from-architecture cost_per_month 73863 116409 91372.7 46.6%
case-02-from-architecture roles 6 8 7.3 27.3%
case-02-from-architecture checks 17 20 18.0 16.7%
case-02-from-architecture capabilities 21 21 21.0 0.0%