feat: Change requests against a Baseline (ADR-0006)
A caller restates a request to say how requirements have changed, naming a completed Execution of the same Customer case as the Baseline. - app/baseline.py reads a Baseline from its files and refuses one that does not exist, did not complete or belongs to another case, before anything runs or is recorded. A Baseline is never modified. - Requirements carry a change (unchanged, added, changed, removed) and a reason. Submission is refused when a Baseline requirement disappears, an id is reused, or an unchanged requirement's text differs. A removed requirement stays on record and asks nothing downstream. - Each Specialist starts from its delivered version; the graph and the code are its working copy, and the description is carried over when the design's shape is the same. - app/workflows/changes.py compares versions by id and traces what each changed requirement reaches; anything else that changed is listed as unrequested. Approvals show the change for their Phase, and changes.json is written with the full Deliverables. - Intake no longer starts a message read as a change: it names the Execution to give as the Baseline. - A fifth evaluation case runs a request and then a change to it, with four checks that the change is recorded, reaches Checks, stays contained and costs less than its Baseline. Tested with scripted agents only; no Change request has run with real models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
| 7 | `iter-07-change-requests/` | steps 1–6 and 8 built (2026-10-09), tested with scripted agents only; step 7's evaluation case is built and its run is outstanding | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
||||
| 13 | Cloud-read call guard | J7 | planned |
|
||||
| 14 | Input guardrail | J11 | planned |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
||||
| 15 | Changes of meaning in a Change request | none | planned; iteration 7 now compares by id and text, so a reworded but equivalent Check still reads as changed |
|
||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||
| 18 | Readings as their own graph step | none | planned |
|
||||
|
||||
@@ -2,12 +2,12 @@
|
||||
|
||||
**Iteration:** 7, Change requests
|
||||
**Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why.
|
||||
**Status:** planned. ADR-0006 was accepted on 2026-10-07 and the glossary updated (step 1). Build after iteration 6 step 11 has a passing evaluation run
|
||||
**Status:** steps 1 to 6 and 8 built (2026-10-09), covered by the deterministic suite; no Change request has run with real models. Step 7's evaluation case and checks are built and its run is outstanding
|
||||
**This directory covers:** steps 1 to 8
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md`: the decision and its four points to confirm. Status `proposed`.
|
||||
- `docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md`: the decision and its four points. Accepted 2026-10-07.
|
||||
- `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` and `../iter-06-target-architecture-graph/`: the ids along the Requirements chain that a change is traced through. Build iteration 6 first.
|
||||
- `CONTEXT.md`: today it says an Execution starts clean and a Customer case never feeds one Execution into another. Step 1 changes those entries; until then they stand.
|
||||
- `iter-04-learning-and-skills/steps/step-15.md`: the Customer case layout, `execution.json` and the Intent history (`intent/history.json`) that a Baseline is found through. A Change request is the next statement in that history.
|
||||
@@ -17,12 +17,12 @@
|
||||
| Step | What | Size | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 |
|
||||
| 2 | Start an Execution against a Baseline | M | planned |
|
||||
| 3 | Requirements revised with a change record | M | planned |
|
||||
| 4 | Later Specialists revise from the Baseline | M | planned |
|
||||
| 5 | Approvals show the change | S | planned |
|
||||
| 6 | Change summary and version line in the case | S | planned |
|
||||
| 7 | Evaluation: a change case | M | planned |
|
||||
| 2 | Start an Execution against a Baseline | M | built; not run with real models |
|
||||
| 3 | Requirements revised with a change record | M | built; not run with real models |
|
||||
| 4 | Later Specialists revise from the Baseline | M | built; not run with real models |
|
||||
| 5 | Approvals show the change | S | built; not run with real models |
|
||||
| 6 | Change summary and version line in the case | S | built; not run with real models |
|
||||
| 7 | Evaluation: a change case | M | case and checks built; run outstanding |
|
||||
| 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
14
.agents/plans/iter-07-change-requests/steps/step-2.md
Normal file
14
.agents/plans/iter-07-change-requests/steps/step-2.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 2 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09.
|
||||
|
||||
1. `app/baseline.py`: `load_baseline(output_root, customer_case, execution_id)` reads a Baseline from its files under the Customer case, through `execution.json`: its id, its request, the Baseline it was itself changed from, and every Deliverable with its attachments. It never needs the Baseline's checkpoint.
|
||||
2. It refuses, with `BaselineError` (a `ValueError`), an id that is not an Execution id, an Execution with no files in that case, one that belongs to another Customer case (without saying which), one whose manifest status is not `completed`, and a Change request with no Customer case.
|
||||
3. `ExecutionEngine.start(baseline=...)` loads the Baseline before the Execution's id is taken or its intent recorded, so a refusal leaves nothing behind. The loaded Baseline is in the state as `baseline`.
|
||||
4. `record_intent` puts the Baseline on the new statement in `intent/history.json` and in its `NNN.md`; `execution.json` records it too. An Execution started without one has `"baseline": null` in both.
|
||||
5. Callers: `baseline` in the `/generate` body and in the A2A message metadata. A bad Baseline is a 400 from `/generate` and a failed task over A2A.
|
||||
6. Intake (iteration 5 step 12) no longer starts a message read as a change. It answers with the latest completed Execution of the case and how to name it as the Baseline, and starts nothing; `"new_request": true` in the metadata says it is new work and skips the reading. A caller who names a Baseline is not read at all.
|
||||
|
||||
Differs from the plan: nothing.
|
||||
|
||||
Tests: `tests/execution/test_change_requests.py` "step 2" (10), `tests/a2a/test_server.py` (3), `tests/execution/test_intake.py`.
|
||||
17
.agents/plans/iter-07-change-requests/steps/step-3.md
Normal file
17
.agents/plans/iter-07-change-requests/steps/step-3.md
Normal file
@@ -0,0 +1,17 @@
|
||||
# Step 3 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09.
|
||||
|
||||
1. In a Change request every requirement in `requirements.json` carries a `change`: `unchanged`, `added`, `changed` or `removed`. Everything but `unchanged` carries a `reason`.
|
||||
2. `check_requirement_changes` in `app/workflows/changes.py`, run by `check_submission` when the Deliverable has a Baseline, refuses:
|
||||
- a requirement with no `change`, or a change with no reason;
|
||||
- a Baseline requirement missing from the list (it must stay, marked `removed`);
|
||||
- a requirement marked `added` whose id the Baseline already has, or marked anything else whose id it does not;
|
||||
- a requirement marked `unchanged` whose text differs from the Baseline's;
|
||||
- a list in which nothing is added, changed or removed.
|
||||
3. A removed requirement stays in the file as the record of what was dropped. `read_list` leaves it out, so nothing downstream is asked to serve or check it.
|
||||
4. `agents/requirements-analyst.md` says how to mark each requirement and to add a `Changes` heading. With a user present the draft is confirmed before submission by the rule of iteration 4 step 27.
|
||||
|
||||
Differs from the plan: the check that an `unchanged` requirement keeps its text was added. Without it a requirement could change quietly.
|
||||
|
||||
Tests: `tests/execution/test_change_requests.py` "step 3" (6).
|
||||
14
.agents/plans/iter-07-change-requests/steps/step-4.md
Normal file
14
.agents/plans/iter-07-change-requests/steps/step-4.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# Step 4 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09.
|
||||
|
||||
1. Each Specialist's context has `baseline` in a Change request: the Baseline's id and original request, the Baseline's version of its own Deliverable (not for the solution guide, which is assembled afresh), and `changes`, what has changed so far upstream.
|
||||
2. Working copies start from the Baseline (`_start_from` in `app/nodes/execution_nodes.py`): the target architect's working graph is the delivered graph, and the IaC author's working copy is the delivered code. A Deliverable with `written_from` is seeded as standing, so the architecture description is carried over without running the describer when the Elements and Relationships are the same as the Baseline's.
|
||||
3. `agents/partials/specialist-protocol.md` tells every Specialist how to revise: start from the delivered version, change only what the changed requirements reach, keep every id that still holds, never renumber.
|
||||
4. `app/workflows/changes.py` compares two versions by id (`change_summary`) and traces `reach`: a changed requirement reaches the components that serve it, the Elements that realise them, the Relationships at those Elements, and every Check on any of them or with the requirement as its source; a changed recommendation reaches the Checks that come from it. Reach is traced through both versions. Anything that changed outside it is listed under `unrequested`.
|
||||
|
||||
Differs from the plan: the plan's test was that an unreached Element or Check is "identical to the Baseline's". The engine cannot make an agent leave something alone; it reports what changed outside the reach, and step 7's check fails a run where much did.
|
||||
|
||||
**Not run with real models**, so whether Specialists do keep ids and leave unreached parts alone is unknown.
|
||||
|
||||
Tests: `tests/execution/test_change_requests.py` "step 4" and "what changed" (6).
|
||||
9
.agents/plans/iter-07-change-requests/steps/step-5.md
Normal file
9
.agents/plans/iter-07-change-requests/steps/step-5.md
Normal file
@@ -0,0 +1,9 @@
|
||||
# Step 5 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09.
|
||||
|
||||
1. `OrchestratorToolbox` is given the Baseline. In a Change request `request_approval` adds `baseline` and `changes` to the Approval: the part of the change summary that belongs to that Phase's Deliverables (`changes_for`), computed from the ids and never written by an agent.
|
||||
2. `describe_prompt` in `app/presentation.py` renders it: each requirement added, changed or removed with its reason; ids added, changed and removed for components, recommendations, Elements, Relationships and Checks; and, apart, anything changed that no changed requirement reaches. A Phase that changed nothing says so.
|
||||
3. An Approval outside a Change request has no `changes` key.
|
||||
|
||||
Tests: `tests/execution/test_change_requests.py` "step 5" (2).
|
||||
12
.agents/plans/iter-07-change-requests/steps/step-6.md
Normal file
12
.agents/plans/iter-07-change-requests/steps/step-6.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Step 6 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09.
|
||||
|
||||
1. A Change request writes `changes.json` beside its Deliverables: the requirements' own account, then components, recommendations, Elements, Relationships and Checks added, changed and removed by id, and `unrequested`. `execution.json` names it under `changes` and names the Baseline.
|
||||
2. Every Deliverable is written in full, as for any Execution, so the latest Execution of a case stands on its own.
|
||||
3. `intent/history.json` lists each statement with its Baseline (step 2), so a case's Executions read as a line of versions.
|
||||
4. `agents/packager.md`: the guide's executive summary opens with what changed and why, and names unrequested changes separately.
|
||||
|
||||
Differs from the plan: `changes.json` records ids, and reasons for requirements only. The plan asked for a reason on every item; a reason for a changed Check is whichever changed requirement or recommendation is its source, which a reader can follow from the ids.
|
||||
|
||||
Tests: `tests/execution/test_change_requests.py` "step 6" (1).
|
||||
16
.agents/plans/iter-07-change-requests/steps/step-7.md
Normal file
16
.agents/plans/iter-07-change-requests/steps/step-7.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# Step 7 — Implementation Detail
|
||||
|
||||
Built on 2026-10-09: the case and its checks. **The run is outstanding.**
|
||||
|
||||
1. A case may carry `change`, a statement of what has changed. The evaluation then runs the case's request to completion, pre-approved, and runs `change` as a Change request against it in the same Customer case. The Change request is what is scored.
|
||||
2. `case-05-change-retention`: case 01's request, then "raw events must now be kept for 7 years instead of 30 days ... and must not be deletable during that time. Everything else stays as it is."
|
||||
3. `score_change` adds four checks to the usual ones:
|
||||
- `change_is_recorded`: at least one requirement is marked added, changed or removed;
|
||||
- `changed_requirements_reach_checks`: each added or changed requirement is the source of a Check that was added or changed, and a removed one's Checks are gone;
|
||||
- `change_is_contained`: no more than two items, or a tenth of the Baseline's graph, changed without being reached;
|
||||
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
|
||||
4. The report records the Baseline's id, seconds and tokens.
|
||||
|
||||
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
|
||||
|
||||
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5).
|
||||
Reference in New Issue
Block a user