fix: a changed functional requirement is reached through its components

The change-case check asked for a changed Check whose source is the changed
requirement. A functional requirement is carried by the components that
serve it, so it is reached through a Check on an Element realising one.

Plans record the first Change request with real models: a retention change
made in 18 minutes for a third of its Baseline's tokens, touching seven
Checks on one Element and nothing else, and the regression runs of cases
02, 03 and 04, all at 100%. Iteration 7 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 13:50:18 +01:00
parent 5ef434d00e
commit 59a641fded
6 changed files with 89 additions and 11 deletions

View File

@@ -12,7 +12,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | | 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. | | 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. | | 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | steps 1–6 and 8 built (2026-10-09), tested with scripted agents only; step 7's evaluation case is built and its run is outstanding | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | | 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps. Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.

View File

@@ -220,3 +220,16 @@ To verify the batched schema lookup (iteration 4 step 18) and the corrected conf
This is the first run of case 04 at 100%. This is the first run of case 04 at 100%.
## Runs of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/` and `...103827Z/`)
A regression check after Change requests (iteration 7) changed what every Specialist is told. All pass with no warnings.
| Case | Score | Time | Tokens | Revisions |
|---|---|---|---|---|
| 02 | 1.00 | 31 min | 3.93 M | 1 |
| 03 | 1.00 | 22 min | 2.32 M | 1 |
| 04 | 1.00 (12 of 12) | 40 min | 3.20 M | 1 |
| 05, the request that the change case then changes (case 01's) | completed | 36 min | 4.32 M | 1 |
Case 03 was its fastest and cheapest yet. The change case itself is recorded in `../../iter-07-change-requests/steps/step-7.md`.

View File

@@ -2,7 +2,7 @@
**Iteration:** 7, Change requests **Iteration:** 7, Change requests
**Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why. **Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why.
**Status:** steps 1 to 6 and 8 built (2026-10-09), covered by the deterministic suite; no Change request has run with real models. Step 7's evaluation case and checks are built and its run is outstanding **Status:** complete (2026-10-09). The change case ran with real models: a retention change was made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else (`steps/step-7.md`)
**This directory covers:** steps 1 to 8 **This directory covers:** steps 1 to 8
## Sources of truth ## Sources of truth
@@ -17,12 +17,12 @@
| Step | What | Size | Status | | Step | What | Size | Status |
|---|---|---|---| |---|---|---|---|
| 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 | | 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 |
| 2 | Start an Execution against a Baseline | M | built; not run with real models | | 2 | Start an Execution against a Baseline | M | complete |
| 3 | Requirements revised with a change record | M | built; not run with real models | | 3 | Requirements revised with a change record | M | complete |
| 4 | Later Specialists revise from the Baseline | M | built; not run with real models | | 4 | Later Specialists revise from the Baseline | M | complete |
| 5 | Approvals show the change | S | built; not run with real models | | 5 | Approvals show the change | S | complete |
| 6 | Change summary and version line in the case | S | built; not run with real models | | 6 | Change summary and version line in the case | S | complete |
| 7 | Evaluation: a change case | M | case and checks built; run outstanding | | 7 | Evaluation: a change case | M | complete: one run, all change checks pass |
| 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 | | 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 |
## Dependency graph ## Dependency graph

View File

@@ -11,6 +11,26 @@ Built on 2026-10-09: the case and its checks. **The run is outstanding.**
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed. - `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
4. The report records the Baseline's id, seconds and tokens. 4. The report records the Baseline's id, seconds and tokens.
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10. ## Run of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/`)
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5). The first Change request with real models. Completed, no warnings, no Revision.
| | Baseline (the request) | Change request |
|---|---|---|
| Time | 36 minutes | 18 minutes |
| Tokens | 4.32 million | 1.47 million (34%) |
| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks |
- **The requirements analyst marked two requirements changed**, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact.
- **Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else.** Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14).
- **Nothing changed without being reached:** 0 of 129 items unrequested.
- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation.
- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change.
Score as run: 93%. `changed_requirements_reach_checks` failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass.
Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens.
One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried.
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (6).

View File

@@ -476,6 +476,12 @@ def score_change(
old_sources = { old_sources = {
item.get("id"): item.get("source") for item in _graph_data(baseline)["checks"] item.get("id"): item.get("source") for item in _graph_data(baseline)["checks"]
} }
serving: dict[str, set[str]] = {}
for component in read_list(
deliverables, "technical_decomposition", COMPONENTS_FILE
):
for requirement in component.get("requirements") or ():
serving.setdefault(str(requirement), set()).add(str(component.get("id")))
unreached = [] unreached = []
for item in requirements: for item in requirements:
identifier = item["id"] identifier = item["id"]
@@ -484,8 +490,20 @@ def score_change(
reaches = any(old_sources.get(c) == identifier for c in removed_checks) reaches = any(old_sources.get(c) == identifier for c in removed_checks)
reaches = reaches or identifier not in old_sources.values() reaches = reaches or identifier not in old_sources.values()
else: else:
# A constraint is the source of its Checks. A functional requirement is
# carried by the components that serve it, so it is reached through a
# Check on an Element that realises one of them.
realising = {
element.get("id")
for element in graph["elements"]
if element.get("component") in serving.get(identifier, ())
}
reaches = any( reaches = any(
check.get("id") in touched and check.get("source") == identifier check.get("id") in touched
and (
check.get("source") == identifier
or check.get("applies_to") in realising
)
for check in graph["checks"] for check in graph["checks"]
) )
if not reaches: if not reaches:

View File

@@ -745,3 +745,30 @@ def test_the_change_case_loads_with_its_statement_of_change():
assert "7 years" in case.change assert "7 years" in case.change
assert all(not c.change for c in cases if c.id != case.id) assert all(not c.change for c in cases if c.id != case.id)
def test_a_functional_requirement_is_reached_through_its_components_elements():
"""The change case of 2026-10-09: a changed functional requirement whose Checks
are sourced to a constraint, on the Element that realises its component."""
from eval.scoring import score_change
def tighten(graph):
# K-1 is on E-1, which realises C-1, which serves FR-1. Its source is NFR-1.
graph["checks"][0]["statement"] = "The service runs at least three instances."
functional = change_interaction(tighten, requirement=False)
requirements = json.loads(
interaction_requirements := functional.state["deliverables"]["requirements"][
"attachments"
]["requirements.json"]
)
assert interaction_requirements
requirements[0].update(change="changed", text="Accept events.", reason="Restated.")
functional.state["deliverables"]["requirements"]["attachments"] = {
"requirements.json": json.dumps(requirements)
}
by_name = {check.name: check for check in score_change(functional, 4, 1)}
assert by_name["change_is_recorded"].detail == "FR-1 changed"
assert by_name["changed_requirements_reach_checks"].passed