diff --git a/.agents/plans/README.md b/.agents/plans/README.md index d0fdeeb..60d2ee4 100644 --- a/.agents/plans/README.md +++ b/.agents/plans/README.md @@ -12,7 +12,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio | 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. | | 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. | | 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. | -| 7 | `iter-07-change-requests/` | steps 1–6 and 8 built (2026-10-09), tested with scripted agents only; step 7's evaluation case is built and its run is outstanding | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | +| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. | Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps. diff --git a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md index 35b607a..cd278a3 100644 --- a/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md +++ b/.agents/plans/iter-06-target-architecture-graph/steps/step-11.md @@ -220,3 +220,16 @@ To verify the batched schema lookup (iteration 4 step 18) and the corrected conf This is the first run of case 04 at 100%. +## Runs of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/` and `...103827Z/`) + +A regression check after Change requests (iteration 7) changed what every Specialist is told. All pass with no warnings. + +| Case | Score | Time | Tokens | Revisions | +|---|---|---|---|---| +| 02 | 1.00 | 31 min | 3.93 M | 1 | +| 03 | 1.00 | 22 min | 2.32 M | 1 | +| 04 | 1.00 (12 of 12) | 40 min | 3.20 M | 1 | +| 05, the request that the change case then changes (case 01's) | completed | 36 min | 4.32 M | 1 | + +Case 03 was its fastest and cheapest yet. The change case itself is recorded in `../../iter-07-change-requests/steps/step-7.md`. + diff --git a/.agents/plans/iter-07-change-requests/MAP.md b/.agents/plans/iter-07-change-requests/MAP.md index a5ac468..2eb1973 100644 --- a/.agents/plans/iter-07-change-requests/MAP.md +++ b/.agents/plans/iter-07-change-requests/MAP.md @@ -2,7 +2,7 @@ **Iteration:** 7, Change requests **Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why. -**Status:** steps 1 to 6 and 8 built (2026-10-09), covered by the deterministic suite; no Change request has run with real models. Step 7's evaluation case and checks are built and its run is outstanding +**Status:** complete (2026-10-09). The change case ran with real models: a retention change was made in 18 minutes for a third of its Baseline's tokens, touching seven Checks on one Element and nothing else (`steps/step-7.md`) **This directory covers:** steps 1 to 8 ## Sources of truth @@ -17,12 +17,12 @@ | Step | What | Size | Status | |---|---|---|---| | 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 | -| 2 | Start an Execution against a Baseline | M | built; not run with real models | -| 3 | Requirements revised with a change record | M | built; not run with real models | -| 4 | Later Specialists revise from the Baseline | M | built; not run with real models | -| 5 | Approvals show the change | S | built; not run with real models | -| 6 | Change summary and version line in the case | S | built; not run with real models | -| 7 | Evaluation: a change case | M | case and checks built; run outstanding | +| 2 | Start an Execution against a Baseline | M | complete | +| 3 | Requirements revised with a change record | M | complete | +| 4 | Later Specialists revise from the Baseline | M | complete | +| 5 | Approvals show the change | S | complete | +| 6 | Change summary and version line in the case | S | complete | +| 7 | Evaluation: a change case | M | complete: one run, all change checks pass | | 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 | ## Dependency graph diff --git a/.agents/plans/iter-07-change-requests/steps/step-7.md b/.agents/plans/iter-07-change-requests/steps/step-7.md index 5ab231a..af42658 100644 --- a/.agents/plans/iter-07-change-requests/steps/step-7.md +++ b/.agents/plans/iter-07-change-requests/steps/step-7.md @@ -11,6 +11,26 @@ Built on 2026-10-09: the case and its checks. **The run is outstanding.** - `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed. 4. The report records the Baseline's id, seconds and tokens. -Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10. +## Run of 2026-10-09, 10:38 UTC (`eval/results/20261009T103822Z/`) -Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5). +The first Change request with real models. Completed, no warnings, no Revision. + +| | Baseline (the request) | Change request | +|---|---|---| +| Time | 36 minutes | 18 minutes | +| Tokens | 4.32 million | 1.47 million (34%) | +| Graph | 22 Elements, 19 Relationships, 88 Checks | the same Elements and Relationships, 91 Checks | + +- **The requirements analyst marked two requirements changed**, FR-6 and NFR-5, both with the audit obligation as the reason, and left the rest unchanged with their text intact. +- **Seven Checks were touched, all on the raw-events bucket (E-6), and nothing else.** Two lifecycle ages went from 30 days to 2,555 (K-41, K-63); the storage class became ARCHIVE (K-40, K-70); a retention policy of 220,752,000 seconds and its lock were added (K-89, K-90, K-91). Two recommendations changed (DR-5, DR-14). +- **Nothing changed without being reached:** 0 of 129 items unrequested. +- The description was carried over from the Baseline without running the describer. The code was revised in 102 seconds and passed all 91 Checks on its first validation. +- The design advisor still consulted all six pillar advisors afresh, 267 seconds, the longest step of the change. + +Score as run: 93%. `changed_requirements_reach_checks` failed for FR-6, wrongly: it asked for a Check whose source is the requirement, and a functional requirement is carried by its components, not by Checks. Corrected to accept a changed Check on an Element that realises a component serving it, and re-scored on the saved files: all four change checks pass. + +Acceptance criteria: met. The change case scores above 0.8; every changed requirement reaches a changed or new Check; nothing outside the reach changed; the Change request used a third of its Baseline's tokens. + +One run, one kind of change. A change that adds or removes an Element, or removes a requirement, has not been tried. + +Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (6). diff --git a/eval/scoring.py b/eval/scoring.py index f31eba9..ce3b956 100644 --- a/eval/scoring.py +++ b/eval/scoring.py @@ -476,6 +476,12 @@ def score_change( old_sources = { item.get("id"): item.get("source") for item in _graph_data(baseline)["checks"] } + serving: dict[str, set[str]] = {} + for component in read_list( + deliverables, "technical_decomposition", COMPONENTS_FILE + ): + for requirement in component.get("requirements") or (): + serving.setdefault(str(requirement), set()).add(str(component.get("id"))) unreached = [] for item in requirements: identifier = item["id"] @@ -484,8 +490,20 @@ def score_change( reaches = any(old_sources.get(c) == identifier for c in removed_checks) reaches = reaches or identifier not in old_sources.values() else: + # A constraint is the source of its Checks. A functional requirement is + # carried by the components that serve it, so it is reached through a + # Check on an Element that realises one of them. + realising = { + element.get("id") + for element in graph["elements"] + if element.get("component") in serving.get(identifier, ()) + } reaches = any( - check.get("id") in touched and check.get("source") == identifier + check.get("id") in touched + and ( + check.get("source") == identifier + or check.get("applies_to") in realising + ) for check in graph["checks"] ) if not reaches: diff --git a/tests/eval/test_scoring.py b/tests/eval/test_scoring.py index 4b5ac72..62e5335 100644 --- a/tests/eval/test_scoring.py +++ b/tests/eval/test_scoring.py @@ -745,3 +745,30 @@ def test_the_change_case_loads_with_its_statement_of_change(): assert "7 years" in case.change assert all(not c.change for c in cases if c.id != case.id) + + +def test_a_functional_requirement_is_reached_through_its_components_elements(): + """The change case of 2026-10-09: a changed functional requirement whose Checks + are sourced to a constraint, on the Element that realises its component.""" + from eval.scoring import score_change + + def tighten(graph): + # K-1 is on E-1, which realises C-1, which serves FR-1. Its source is NFR-1. + graph["checks"][0]["statement"] = "The service runs at least three instances." + + functional = change_interaction(tighten, requirement=False) + requirements = json.loads( + interaction_requirements := functional.state["deliverables"]["requirements"][ + "attachments" + ]["requirements.json"] + ) + assert interaction_requirements + requirements[0].update(change="changed", text="Accept events.", reason="Restated.") + functional.state["deliverables"]["requirements"]["attachments"] = { + "requirements.json": json.dumps(requirements) + } + + by_name = {check.name: check for check in score_change(functional, 4, 1)} + + assert by_name["change_is_recorded"].detail == "FR-1 changed" + assert by_name["changed_requirements_reach_checks"].passed