fix: a changed functional requirement is reached through its components

The change-case check asked for a changed Check whose source is the changed
requirement. A functional requirement is carried by the components that
serve it, so it is reached through a Check on an Element realising one.

Plans record the first Change request with real models: a retention change
made in 18 minutes for a third of its Baseline's tokens, touching seven
Checks on one Element and nothing else, and the regression runs of cases
02, 03 and 04, all at 100%. Iteration 7 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 13:50:18 +01:00
parent 5ef434d00e
commit 59a641fded
6 changed files with 89 additions and 11 deletions

View File

@@ -11,6 +11,26 @@ Built on 2026-10-09: the case and its checks. **The run is outstanding.**
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
4. The report records the Baseline's id, seconds and tokens.
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
## Run of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/`)
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5).
The first Change request with real models. Completed, no warnings, no Revision.
| | Baseline (the request) | Change request |
|---|---|---|
| Time | 36 minutes | 18 minutes |
| Tokens | 4.32 million | 1.47 million (34%) |
| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks |
- **The requirements analyst marked two requirements changed**, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact.
- **Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else.** Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14).
- **Nothing changed without being reached:** 0 of 129 items unrequested.
- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation.
- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change.
Score as run: 93%. `changed_requirements_reach_checks` failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass.
Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens.
One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried.
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (6).