feat: every Operational requirement verified by a Check; eval spread, report and published runs #1

Merged
demo-bot merged 4 commits from feat/eval-coverage-and-spread into main 2026-10-23 17:30:06 +00:00
Owner

Summary

Every Operational requirement must now be verified by a Check, the evaluation measures it and the spread of repeated runs, and the runs of 2026-10-21 are published.

 Target operating model refused while
   a Capability has no accountable Role, …
   a Condition is carried by no Check
+  an Operational requirement is verified by no Check
+    (none from the requirement, or from a Condition it comes from)
 eval/
 ├── operating_model_eval.py   # --repeat N; spread.json; report.md; folders relative to the run
 ├── scoring.py                # + "every Operational requirement is verified by a Check"
 └── results/                  # still ignored, but for the two runs published:
+    ├── 20261021T203624Z/          # both cases three times
+    └── 20261021T203623Z-handoff/  # from the solution architecture agent through the Orchestration engine

The architect's instructions and the operating-chain Skill state the new rule; the test fixture gained the two Checks it lacked.

Evidence

  • Before: six of the nine runs of 2026-10-13 and 14 left at least one Operational requirement (sourcing model, delivery countries, tooling) verified by no Check, and the evaluation passed them; the 16:28 run had 13 Checks for 16 requirements.
    After: test_a_graph_that_does_not_hold_together_is_refused[…OR-1 is verified by no Check] fails before the rule and passes after; 184 tests pass, ruff and mypy clean.
  • Runs with real models, 2026-10-21: every requirement verified in all six evaluation runs (Checks 15 to 20 a run) and in the hand-off (24 of 24). Five of six passed; case 02 run 1 stopped at the operating model's revision limit on Checks its Capabilities could not bear out (four of five from Conditions: the weakness recorded in step 3, not this rule). Spread: case 01 cost 6.2%, case 02 cost 46.6% (73,863 to 116,409 GBP a month).

Merge Danger

Door: two-way

Blast Radius: operating-models

Executions will be refused, and revised, where an Operational requirement has no Check: more Checks a model, possibly a Revision more a run, more for the validator to judge. The published results add 193 files (about 2.2 MB) of fictional benchmark data; the hand-off's links carry no signatures.

🤖 Generated with Claude Code

## Summary Every Operational requirement must now be verified by a Check, the evaluation measures it and the spread of repeated runs, and the runs of 2026-10-21 are published. ```diff Target operating model refused while a Capability has no accountable Role, … a Condition is carried by no Check + an Operational requirement is verified by no Check + (none from the requirement, or from a Condition it comes from) ``` ```diff eval/ ├── operating_model_eval.py # --repeat N; spread.json; report.md; folders relative to the run ├── scoring.py # + "every Operational requirement is verified by a Check" └── results/ # still ignored, but for the two runs published: + ├── 20261021T203624Z/ # both cases three times + └── 20261021T203623Z-handoff/ # from the solution architecture agent through the Orchestration engine ``` The architect's instructions and the `operating-chain` Skill state the new rule; the test fixture gained the two Checks it lacked. ## Evidence - **Before:** six of the nine runs of 2026-10-13 and 14 left at least one Operational requirement (sourcing model, delivery countries, tooling) verified by no Check, and the evaluation passed them; the 16:28 run had 13 Checks for 16 requirements. **After:** `test_a_graph_that_does_not_hold_together_is_refused[…OR-1 is verified by no Check]` fails before the rule and passes after; 184 tests pass, `ruff` and `mypy` clean. - **Runs with real models, 2026-10-21:** every requirement verified in all six evaluation runs (Checks 15 to 20 a run) and in the hand-off (24 of 24). Five of six passed; case 02 run 1 stopped at the operating model's revision limit on Checks its Capabilities could not bear out (four of five from Conditions: the weakness recorded in step 3, not this rule). Spread: case 01 cost 6.2%, case 02 cost 46.6% (73,863 to 116,409 GBP a month). ## Merge Danger **Door:** two-way **Blast Radius:** operating-models Executions will be refused, and revised, where an Operational requirement has no Check: more Checks a model, possibly a Revision more a run, more for the validator to judge. The published results add 193 files (about 2.2 MB) of fictional benchmark data; the hand-off's links carry no signatures. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
demo-bot added 4 commits 2026-10-23 17:22:03 +00:00
The operating model is refused while an Operational requirement is verified
by no Check, from it or from a Condition it comes from; the architect is
told so, and the operating-chain Skill lists it. Six of nine earlier runs
left a requirement unverified.

The evaluation checks it too, runs each case N times (--repeat), reports
the spread of people, cost, Roles, Checks and Capabilities (spread.json),
and writes report.md to publish. Result folders are relative to the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The evaluation (both cases three times: report, results, spread, each
run's Deliverables and lesson candidates) and the hand-off through the
Orchestration engine (report, the document of links, both agents' kept
files and records). Benchmark cases only, all fictional; no machine paths,
and the links carry no signatures. Other runs stay ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The general `deliverables/` ignore matched the folder inside each run's
Execution too, so the Deliverables as markdown (operating scope,
requirements, capability model, operating model, validation, guide) were
left out. They are now included for the published run only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
demo-bot merged commit ceddc1e554 into main 2026-10-23 17:30:06 +00:00
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: agents/tom_analysis_agent#1