Files
Jonathan Boniface 628b6ee170 docs: publish the runs of 2026-10-21
The evaluation (both cases three times: report, results, spread, each
run's Deliverables and lesson candidates) and the hand-off through the
Orchestration engine (report, the document of links, both agents' kept
files and records). Benchmark cases only, all fictional; no machine paths,
and the links carry no signatures. Other runs stay ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 09:25:03 +01:00

40 lines
2.2 KiB
Markdown

# Target operating model agent: evaluation
Run `20261021T203624Z`. Each run is one pre-approved Execution with real models on a benchmark case. Every case is fictional; no customer's data is in it.
## Results
| Case | Run | Result | Status | Minutes | Revisions | Requirements verified | Roles | People | Cost a month |
|---|---|---|---|---|---|---|---|---|---|
| case-01-landing-zone-shared | 1 | PASS | completed | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP |
| case-01-landing-zone-shared | 2 | PASS | completed | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP |
| case-01-landing-zone-shared | 3 | PASS | completed | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP |
| case-02-from-architecture | 1 | FAIL | completed_with_warnings | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP |
| case-02-from-architecture | 2 | PASS | completed | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP |
| case-02-from-architecture | 3 | PASS | completed | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP |
## Checks that failed
- **case-02-from-architecture**: completed (completed_with_warnings)
- **case-02-from-architecture**: every Deliverable was submitted (operating_model_guide)
- **case-02-from-architecture**: no Check failed (OC-2, OC-4, OC-9, OC-12, OC-17)
- **case-02-from-architecture**: the guide has its sections
- **case-02-from-architecture**: the packager wrote a summary (0 words)
## Spread across repeated runs
The same case and inputs, run again: how far the operating model moved.
| Case | Figure | Least | Most | Mean | Range of the mean |
|---|---|---|---|---|---|
| case-01-landing-zone-shared | people | 6.8 | 11.4 | 9.7 | 47.3% |
| case-01-landing-zone-shared | cost_per_month | 44720 | 47603 | 46191.7 | 6.2% |
| case-01-landing-zone-shared | roles | 5 | 7 | 6.0 | 33.3% |
| case-01-landing-zone-shared | checks | 15 | 19 | 16.3 | 24.5% |
| case-01-landing-zone-shared | capabilities | 21 | 21 | 21.0 | 0.0% |
| case-02-from-architecture | people | 10.8 | 16.9 | 13.1 | 46.4% |
| case-02-from-architecture | cost_per_month | 73863 | 116409 | 91372.7 | 46.6% |
| case-02-from-architecture | roles | 6 | 8 | 7.3 | 27.3% |
| case-02-from-architecture | checks | 17 | 20 | 18.0 | 16.7% |
| case-02-from-architecture | capabilities | 21 | 21 | 21.0 | 0.0% |