fix: a changed functional requirement is reached through its components
The change-case check asked for a changed Check whose source is the changed requirement. A functional requirement is carried by the components that serve it, so it is reached through a Check on an Element realising one. Plans record the first Change request with real models: a retention change made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else, and the regression runs of cases 02, 03 and 04, all at 100%. Iteration 7 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,26 @@ Built on 2026-10-09: the case and its checks. **The run is outstanding.**
|
||||
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
|
||||
4. The report records the Baseline's id, seconds and tokens.
|
||||
|
||||
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
|
||||
## Run of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/`)
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5).
|
||||
The first Change request with real models. Completed, no warnings, no Revision.
|
||||
|
||||
| | Baseline (the request) | Change request |
|
||||
|---|---|---|
|
||||
| Time | 36 minutes | 18 minutes |
|
||||
| Tokens | 4.32 million | 1.47 million (34%) |
|
||||
| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks |
|
||||
|
||||
- **The requirements analyst marked two requirements changed**, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact.
|
||||
- **Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else.** Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14).
|
||||
- **Nothing changed without being reached:** 0 of 129 items unrequested.
|
||||
- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation.
|
||||
- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change.
|
||||
|
||||
Score as run: 93%. `changed_requirements_reach_checks` failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass.
|
||||
|
||||
Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens.
|
||||
|
||||
One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (6).
|
||||
|
||||
Reference in New Issue
Block a user