fix: a changed functional requirement is reached through its components
The change-case check asked for a changed Check whose source is the changed requirement. A functional requirement is carried by the components that serve it, so it is reached through a Check on an Element realising one. Plans record the first Change request with real models: a retention change made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else, and the regression runs of cases 02, 03 and 04, all at 100%. Iteration 7 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | steps 1–6 and 8 built (2026-10-09), tested with scripted agents only; step 7's evaluation case is built and its run is outstanding | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
|
||||
@@ -220,3 +220,16 @@ To verify the batched schema lookup (iteration 4 step 18) and the corrected conf
|
||||
|
||||
This is the first run of case 04 at 100%.
|
||||
|
||||
## Runs of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/` and `...103827Z/`)
|
||||
|
||||
A regression check after Change requests (iteration 7) changed what every Specialist is told. All pass with no warnings.
|
||||
|
||||
| Case | Score | Time | Tokens | Revisions |
|
||||
|---|---|---|---|---|
|
||||
| 02 | 1.00 | 31 min | 3.93 M | 1 |
|
||||
| 03 | 1.00 | 22 min | 2.32 M | 1 |
|
||||
| 04 | 1.00 (12 of 12) | 40 min | 3.20 M | 1 |
|
||||
| 05, the request that the change case then changes (case 01's) | completed | 36 min | 4.32 M | 1 |
|
||||
|
||||
Case 03 was its fastest and cheapest yet. The change case itself is recorded in `../../iter-07-change-requests/steps/step-7.md`.
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 7, Change requests
|
||||
**Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why.
|
||||
**Status:** steps 1 to 6 and 8 built (2026-10-09), covered by the deterministic suite; no Change request has run with real models. Step 7's evaluation case and checks are built and its run is outstanding
|
||||
**Status:** complete (2026-10-09). The change case ran with real models: a retention change was made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else (`steps/step-7.md`)
|
||||
**This directory covers:** steps 1 to 8
|
||||
|
||||
## Sources of truth
|
||||
@@ -17,12 +17,12 @@
|
||||
| Step | What | Size | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 |
|
||||
| 2 | Start an Execution against a Baseline | M | built; not run with real models |
|
||||
| 3 | Requirements revised with a change record | M | built; not run with real models |
|
||||
| 4 | Later Specialists revise from the Baseline | M | built; not run with real models |
|
||||
| 5 | Approvals show the change | S | built; not run with real models |
|
||||
| 6 | Change summary and version line in the case | S | built; not run with real models |
|
||||
| 7 | Evaluation: a change case | M | case and checks built; run outstanding |
|
||||
| 2 | Start an Execution against a Baseline | M | complete |
|
||||
| 3 | Requirements revised with a change record | M | complete |
|
||||
| 4 | Later Specialists revise from the Baseline | M | complete |
|
||||
| 5 | Approvals show the change | S | complete |
|
||||
| 6 | Change summary and version line in the case | S | complete |
|
||||
| 7 | Evaluation: a change case | M | complete: one run, all change checks pass |
|
||||
| 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
@@ -11,6 +11,26 @@ Built on 2026-10-09: the case and its checks. **The run is outstanding.**
|
||||
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
|
||||
4. The report records the Baseline's id, seconds and tokens.
|
||||
|
||||
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
|
||||
## Run of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/`)
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5).
|
||||
The first Change request with real models. Completed, no warnings, no Revision.
|
||||
|
||||
| | Baseline (the request) | Change request |
|
||||
|---|---|---|
|
||||
| Time | 36 minutes | 18 minutes |
|
||||
| Tokens | 4.32 million | 1.47 million (34%) |
|
||||
| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks |
|
||||
|
||||
- **The requirements analyst marked two requirements changed**, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact.
|
||||
- **Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else.** Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14).
|
||||
- **Nothing changed without being reached:** 0 of 129 items unrequested.
|
||||
- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation.
|
||||
- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change.
|
||||
|
||||
Score as run: 93%. `changed_requirements_reach_checks` failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass.
|
||||
|
||||
Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens.
|
||||
|
||||
One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (6).
|
||||
|
||||
@@ -476,6 +476,12 @@ def score_change(
|
||||
old_sources = {
|
||||
item.get("id"): item.get("source") for item in _graph_data(baseline)["checks"]
|
||||
}
|
||||
serving: dict[str, set[str]] = {}
|
||||
for component in read_list(
|
||||
deliverables, "technical_decomposition", COMPONENTS_FILE
|
||||
):
|
||||
for requirement in component.get("requirements") or ():
|
||||
serving.setdefault(str(requirement), set()).add(str(component.get("id")))
|
||||
unreached = []
|
||||
for item in requirements:
|
||||
identifier = item["id"]
|
||||
@@ -484,8 +490,20 @@ def score_change(
|
||||
reaches = any(old_sources.get(c) == identifier for c in removed_checks)
|
||||
reaches = reaches or identifier not in old_sources.values()
|
||||
else:
|
||||
# A constraint is the source of its Checks. A functional requirement is
|
||||
# carried by the components that serve it, so it is reached through a
|
||||
# Check on an Element that realises one of them.
|
||||
realising = {
|
||||
element.get("id")
|
||||
for element in graph["elements"]
|
||||
if element.get("component") in serving.get(identifier, ())
|
||||
}
|
||||
reaches = any(
|
||||
check.get("id") in touched and check.get("source") == identifier
|
||||
check.get("id") in touched
|
||||
and (
|
||||
check.get("source") == identifier
|
||||
or check.get("applies_to") in realising
|
||||
)
|
||||
for check in graph["checks"]
|
||||
)
|
||||
if not reaches:
|
||||
|
||||
@@ -745,3 +745,30 @@ def test_the_change_case_loads_with_its_statement_of_change():
|
||||
|
||||
assert "7 years" in case.change
|
||||
assert all(not c.change for c in cases if c.id != case.id)
|
||||
|
||||
|
||||
def test_a_functional_requirement_is_reached_through_its_components_elements():
|
||||
"""The change case of 2026-10-09: a changed functional requirement whose Checks
|
||||
are sourced to a constraint, on the Element that realises its component."""
|
||||
from eval.scoring import score_change
|
||||
|
||||
def tighten(graph):
|
||||
# K-1 is on E-1, which realises C-1, which serves FR-1. Its source is NFR-1.
|
||||
graph["checks"][0]["statement"] = "The service runs at least three instances."
|
||||
|
||||
functional = change_interaction(tighten, requirement=False)
|
||||
requirements = json.loads(
|
||||
interaction_requirements := functional.state["deliverables"]["requirements"][
|
||||
"attachments"
|
||||
]["requirements.json"]
|
||||
)
|
||||
assert interaction_requirements
|
||||
requirements[0].update(change="changed", text="Accept events.", reason="Restated.")
|
||||
functional.state["deliverables"]["requirements"]["attachments"] = {
|
||||
"requirements.json": json.dumps(requirements)
|
||||
}
|
||||
|
||||
by_name = {check.name: check for check in score_change(functional, 4, 1)}
|
||||
|
||||
assert by_name["change_is_recorded"].detail == "FR-1 changed"
|
||||
assert by_name["changed_requirements_reach_checks"].passed
|
||||
|
||||
Reference in New Issue
Block a user