docs: publish the runs of 2026-10-21
The evaluation (both cases three times: report, results, spread, each run's Deliverables and lesson candidates) and the hand-off through the Orchestration engine (report, the document of links, both agents' kept files and records). Benchmark cases only, all fictional; no machine paths, and the links carry no signatures. Other runs stay ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
39
eval/results/20261021T203624Z/report.md
Normal file
39
eval/results/20261021T203624Z/report.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# Target operating model agent: evaluation
|
||||
|
||||
Run `20261021T203624Z`. Each run is one pre-approved Execution with real models on a benchmark case. Every case is fictional; no customer's data is in it.
|
||||
|
||||
## Results
|
||||
|
||||
| Case | Run | Result | Status | Minutes | Revisions | Requirements verified | Roles | People | Cost a month |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| case-01-landing-zone-shared | 1 | PASS | completed | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP |
|
||||
| case-01-landing-zone-shared | 2 | PASS | completed | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP |
|
||||
| case-01-landing-zone-shared | 3 | PASS | completed | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP |
|
||||
| case-02-from-architecture | 1 | FAIL | completed_with_warnings | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP |
|
||||
| case-02-from-architecture | 2 | PASS | completed | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP |
|
||||
| case-02-from-architecture | 3 | PASS | completed | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP |
|
||||
|
||||
## Checks that failed
|
||||
|
||||
- **case-02-from-architecture**: completed (completed_with_warnings)
|
||||
- **case-02-from-architecture**: every Deliverable was submitted (operating_model_guide)
|
||||
- **case-02-from-architecture**: no Check failed (OC-2, OC-4, OC-9, OC-12, OC-17)
|
||||
- **case-02-from-architecture**: the guide has its sections
|
||||
- **case-02-from-architecture**: the packager wrote a summary (0 words)
|
||||
|
||||
## Spread across repeated runs
|
||||
|
||||
The same case and inputs, run again: how far the operating model moved.
|
||||
|
||||
| Case | Figure | Least | Most | Mean | Range of the mean |
|
||||
|---|---|---|---|---|---|
|
||||
| case-01-landing-zone-shared | people | 6.8 | 11.4 | 9.7 | 47.3% |
|
||||
| case-01-landing-zone-shared | cost_per_month | 44720 | 47603 | 46191.7 | 6.2% |
|
||||
| case-01-landing-zone-shared | roles | 5 | 7 | 6.0 | 33.3% |
|
||||
| case-01-landing-zone-shared | checks | 15 | 19 | 16.3 | 24.5% |
|
||||
| case-01-landing-zone-shared | capabilities | 21 | 21 | 21.0 | 0.0% |
|
||||
| case-02-from-architecture | people | 10.8 | 16.9 | 13.1 | 46.4% |
|
||||
| case-02-from-architecture | cost_per_month | 73863 | 116409 | 91372.7 | 46.6% |
|
||||
| case-02-from-architecture | roles | 6 | 8 | 7.3 | 27.3% |
|
||||
| case-02-from-architecture | checks | 17 | 20 | 18.0 | 16.7% |
|
||||
| case-02-from-architecture | capabilities | 21 | 21 | 21.0 | 0.0% |
|
||||
Reference in New Issue
Block a user