A caller restates a request to say how requirements have changed, naming a completed Execution of the same Customer case as the Baseline. - app/baseline.py reads a Baseline from its files and refuses one that does not exist, did not complete or belongs to another case, before anything runs or is recorded. A Baseline is never modified. - Requirements carry a change (unchanged, added, changed, removed) and a reason. Submission is refused when a Baseline requirement disappears, an id is reused, or an unchanged requirement's text differs. A removed requirement stays on record and asks nothing downstream. - Each Specialist starts from its delivered version; the graph and the code are its working copy, and the description is carried over when the design's shape is the same. - app/workflows/changes.py compares versions by id and traces what each changed requirement reaches; anything else that changed is listed as unrequested. Approvals show the change for their Phase, and changes.json is written with the full Deliverables. - Intake no longer starts a message read as a change: it names the Execution to give as the Baseline. - A fifth evaluation case runs a request and then a change to it, with four checks that the change is recorded, reaches Checks, stays contained and costs less than its Baseline. Tested with scripted agents only; no Change request has run with real models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1.3 KiB
1.3 KiB
Step 7 — Implementation Detail
Built on 2026-10-09: the case and its checks. The run is outstanding.
- A case may carry
change, a statement of what has changed. The evaluation then runs the case's request to completion, pre-approved, and runschangeas a Change request against it in the same Customer case. The Change request is what is scored. case-05-change-retention: case 01's request, then "raw events must now be kept for 7 years instead of 30 days ... and must not be deletable during that time. Everything else stays as it is."score_changeadds four checks to the usual ones:change_is_recorded: at least one requirement is marked added, changed or removed;changed_requirements_reach_checks: each added or changed requirement is the source of a Check that was added or changed, and a removed one's Checks are gone;change_is_contained: no more than two items, or a tenth of the Baseline's graph, changed without being reached;change_costs_less_than_baseline: the Change request used fewer tokens than the Execution it changed.
- The report records the Baseline's id, seconds and tokens.
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
Tests: tests/eval/test_scoring.py, "a Change request stays contained" (5).