The evaluation (both cases three times: report, results, spread, each run's Deliverables and lesson candidates) and the hand-off through the Orchestration engine (report, the document of links, both agents' kept files and records). Benchmark cases only, all fictional; no machine paths, and the links carry no signatures. Other runs stay ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.2 KiB
2.2 KiB
Target operating model agent: evaluation
Run 20261021T203624Z. Each run is one pre-approved Execution with real models on a benchmark case. Every case is fictional; no customer's data is in it.
Results
| Case | Run | Result | Status | Minutes | Revisions | Requirements verified | Roles | People | Cost a month |
|---|---|---|---|---|---|---|---|---|---|
| case-01-landing-zone-shared | 1 | PASS | completed | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP |
| case-01-landing-zone-shared | 2 | PASS | completed | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP |
| case-01-landing-zone-shared | 3 | PASS | completed | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP |
| case-02-from-architecture | 1 | FAIL | completed_with_warnings | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP |
| case-02-from-architecture | 2 | PASS | completed | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP |
| case-02-from-architecture | 3 | PASS | completed | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP |
Checks that failed
- case-02-from-architecture: completed (completed_with_warnings)
- case-02-from-architecture: every Deliverable was submitted (operating_model_guide)
- case-02-from-architecture: no Check failed (OC-2, OC-4, OC-9, OC-12, OC-17)
- case-02-from-architecture: the guide has its sections
- case-02-from-architecture: the packager wrote a summary (0 words)
Spread across repeated runs
The same case and inputs, run again: how far the operating model moved.
| Case | Figure | Least | Most | Mean | Range of the mean |
|---|---|---|---|---|---|
| case-01-landing-zone-shared | people | 6.8 | 11.4 | 9.7 | 47.3% |
| case-01-landing-zone-shared | cost_per_month | 44720 | 47603 | 46191.7 | 6.2% |
| case-01-landing-zone-shared | roles | 5 | 7 | 6.0 | 33.3% |
| case-01-landing-zone-shared | checks | 15 | 19 | 16.3 | 24.5% |
| case-01-landing-zone-shared | capabilities | 21 | 21 | 21.0 | 0.0% |
| case-02-from-architecture | people | 10.8 | 16.9 | 13.1 | 46.4% |
| case-02-from-architecture | cost_per_month | 73863 | 116409 | 91372.7 | 46.6% |
| case-02-from-architecture | roles | 6 | 8 | 7.3 | 27.3% |
| case-02-from-architecture | checks | 17 | 20 | 18.0 | 16.7% |
| case-02-from-architecture | capabilities | 21 | 21 | 21.0 | 0.0% |