docs: record the runs of 2026-10-21: three of each case, and the hand-off

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-22 00:04:13 +01:00
parent 6fdac22876
commit 1fed1cb538
2 changed files with 25 additions and 3 deletions

View File

@@ -29,4 +29,4 @@
| T12 | The rate card Skill's format and checks | built (`steps/step-1.md`) |
| T13 | Staffing and cost in code | built (`steps/step-1.md`) |
| T14 | Evaluation with real models | built; two cases passed (`steps/step-2.md`) |
| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); runs in progress |
| T15 | Every Operational requirement verified by a Check; the spread of repeated runs; a report to publish | A requirement no Check verifies is refused; three runs of each case report their spread | built (`steps/step-4.md`); run 2026-10-21: five of six passed, every requirement verified; through the Orchestration engine from the solution architecture agent, completed |

View File

@@ -16,6 +16,28 @@ Built 2026-10-21 on `feat/eval-coverage-and-spread`, at the owner's request, aft
184 tests pass; `ruff` and `mypy` clean. With no model: the benchmark architecture from the fixture, with case 02's operating fields, starts a pre-approved Execution.
## The runs
## The runs, 2026-10-21
Recorded below when they finish: both cases three times each, and one request through the Orchestration engine from the solution architecture agent to this one.
### Both cases three times (`eval/results/20261021T203624Z/`)
| Case | Run | Result | Minutes | Revisions | Requirements verified | Roles | People | Cost a month |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 01, written spec | 1 | pass | 18.8 | 1 | 10 of 10 | 5 | 6.8 | 46,252 GBP |
| 01 | 2 | pass | 17.1 | 1 | 9 of 9 | 7 | 11.0 | 44,720 GBP |
| 01 | 3 | pass | 20.2 | 1 | 10 of 10 | 6 | 11.4 | 47,603 GBP |
| 02, digested architecture | 1 | **fail** | 50.4 | 5 | 17 of 17 | 8 | 11.7 | 73,863 GBP |
| 02 | 2 | pass | 20.9 | 1 | 15 of 15 | 6 | 10.8 | 83,846 GBP |
| 02 | 3 | pass | 24.4 | 2 | 15 of 15 | 8 | 16.9 | 116,409 GBP |
- **Every requirement was verified in every run.** Checks rose to 15 to 20 a run, from 13 or 14.
- **Case 02, run 1 ended with warnings** at the operating model's revision limit. The validator failed five Checks whose words the Capability they are on does not bear out: "continuously maintained" and "continuously reviewed" on Capabilities staffed in business hours, a 10-week validation on the Capability that manages the roadmap rather than the one that delivers, sign-off the Capability does not mention, and alignment with DORA, PCI DSS and ISO 27001 on a Capability covering PCI DSS alone. Four of the five come from Conditions, not requirements: the weakness of step 3 ("a Check whose hours or Capability did not fit"), not the new rule.
- **The spread:** case 01's cost moved 6.2% across its runs while its people moved 47.3%: more people, cheaper. Case 02's cost moved 46.6% (73,863 to 116,409 GBP) and its people 46.4%; Roles 27 to 33%, Checks 17 to 25%; Capabilities not at all (21 each time).
### Through the Orchestration engine from the solution architecture agent (`eval/results/20261021T203623Z-handoff/`)
One pre-approved request for `architecture_and_operating_model`, Customer case `case-08-landing-zone`, from the benchmark landing-zone spec, with case 02's operating fields under `operating_model`. Completed in 74.9 minutes: the architecture in about 50, its 15 kept files given to this agent as `sources`, which digested them (`metadata.source: architecture:6c316f05…`) and completed: 24 Operational requirements, all verified, 21 Capabilities, 8 Roles, 26 Checks, 44 of 44 scenarios, 12.2 people at 87,959 GBP a month. Both Executions' files are kept in the Orchestration engine's store (MinIO, 28 objects) and were fetched through their links. The signatures were taken off the links in the results before publishing.
## What remains
- **The spread is the larger finding.** The same architecture costs between 74 and 116 thousand GBP a month across three runs. The figures are computed; the design is not stable. What moves is the Roles and how work is shared between them.
- **Checks that the Capability they are on cannot bear out** end a run at its revision limit about one time in three for case 02. A data check that refuses a "continuous" or 24x7 Check on a Capability whose hours are narrower would catch the commonest of them before validation.