The change-case check asked for a changed Check whose source is the changed requirement. A functional requirement is carried by the components that serve it, so it is reached through a Check on an Element realising one. Plans record the first Change request with real models: a retention change made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else, and the regression runs of cases 02, 03 and 04, all at 100%. Iteration 7 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3.1 KiB
Step 7 — Implementation Detail
Built on 2026-10-09: the case and its checks. The run is outstanding.
- A case may carry
change, a statement of what has changed. The evaluation then runs the case's request to completion, pre-approved, and runschangeas a Change request against it in the same Customer case. The Change request is what is scored. case-05-change-retention: case 01's request, then "raw events must now be kept for 7 years instead of 30 days ... and must not be deletable during that time. Everything else stays as it is."score_changeadds four checks to the usual ones:change_is_recorded: at least one requirement is marked added, changed or removed;changed_requirements_reach_checks: each added or changed requirement is the source of a Check that was added or changed, and a removed one's Checks are gone;change_is_contained: no more than two items, or a tenth of the Baseline's graph, changed without being reached;change_costs_less_than_baseline: the Change request used fewer tokens than the Execution it changed.
- The report records the Baseline's id, seconds and tokens.
Run of 2026-10-09, 10:38 UTC (eval/results/20261009T103822Z/)
The first Change request with real models. Completed, no warnings, no Revision.
| Baseline (the request) | Change request | |
|---|---|---|
| Time | 36 minutes | 18 minutes |
| Tokens | 4.32 million | 1.47 million (34%) |
| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks |
- The requirements analyst marked two requirements changed, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact.
- Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else. Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14).
- Nothing changed without being reached: 0 of 129 items unrequested.
- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation.
- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change.
Score as run: 93%. changed_requirements_reach_checks failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass.
Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens.
One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried.
Tests: tests/eval/test_scoring.py, "a Change request stays contained" (6).