feat: Change requests against a Baseline (ADR-0006)
A caller restates a request to say how requirements have changed, naming a completed Execution of the same Customer case as the Baseline. - app/baseline.py reads a Baseline from its files and refuses one that does not exist, did not complete or belongs to another case, before anything runs or is recorded. A Baseline is never modified. - Requirements carry a change (unchanged, added, changed, removed) and a reason. Submission is refused when a Baseline requirement disappears, an id is reused, or an unchanged requirement's text differs. A removed requirement stays on record and asks nothing downstream. - Each Specialist starts from its delivered version; the graph and the code are its working copy, and the description is carried over when the design's shape is the same. - app/workflows/changes.py compares versions by id and traces what each changed requirement reaches; anything else that changed is listed as unrequested. Approvals show the change for their Phase, and changes.json is written with the full Deliverables. - Intake no longer starts a message read as a change: it names the Execution to give as the Baseline. - A fifth evaluation case runs a request and then a change to it, with four checks that the change is recorded, reaches Checks, stays contained and costs less than its Baseline. Tested with scripted agents only; no Change request has run with real models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
|||||||
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
| 4 | `iter-04-learning-and-skills/` | steps 1–8, 11, 13 and 15–27 built; cases 01, 02 and 03 completed at 100% on them on 2026-10-07 and 08; ADR-0004 accepted 2026-10-08 (step 9); 10, 12 and 14 planned | The enhancement backlog from the 2026-10-06 review: cheap fixes, the learning loop through Skills, overlay Skills, provider neutrality. Steps 17–20 are the speed-ups from the 2026-10-07 run: a working copy for files and the graph, a provider schema lookup, routing a Check the code cannot meet, and a model timeout per role. |
|
||||||
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
| 5 | `iter-05-jev-decisions/` | the Jev client and the Orchestrator router are built (steps 2, 11, 12), unverified against the live service; the rest planned | Jev (a hosted decision model) for the agent's closed decisions: checks, routing, context selection. |
|
||||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||||
| 7 | `iter-07-change-requests/` | planned; ADR-0006 accepted 2026-10-07; follows iteration 6; its step 8 (removing `deliverables/cases/legacy-baseline/`) was done early on the owner's instruction | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
| 7 | `iter-07-change-requests/` | steps 1–6 and 8 built (2026-10-09), tested with scripted agents only; step 7's evaluation case is built and its run is outstanding | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||||
|
|
||||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||||
|
|
||||||
|
|||||||
@@ -89,7 +89,7 @@ Jev answers closed questions. Writing Deliverables, grounding product choices an
|
|||||||
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
| 12 | The Orchestrator routes by readings and a playbook; a model only writes replies | J1 | built, including intake of a message that starts something (`steps/step-12.md`); the three faults case 04 found are fixed; a completed interactive run is outstanding |
|
||||||
| 13 | Cloud-read call guard | J7 | planned |
|
| 13 | Cloud-read call guard | J7 | planned |
|
||||||
| 14 | Input guardrail | J11 | planned |
|
| 14 | Input guardrail | J11 | planned |
|
||||||
| 15 | Changes of meaning in a Change request | none | planned; needs iteration 7 |
|
| 15 | Changes of meaning in a Change request | none | planned; iteration 7 now compares by id and text, so a reworded but equivalent Check still reads as changed |
|
||||||
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
| 16 | Jev as the evaluation judge, calibrated | none | judge and calibration built; stability and agreement between judges measured (`steps/step-16.md`); agreement with a person's labels awaits the owner's 32 labels |
|
||||||
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
| 17 | Trace every decision | none | open, owner's decision on LangSmith |
|
||||||
| 18 | Readings as their own graph step | none | planned |
|
| 18 | Readings as their own graph step | none | planned |
|
||||||
|
|||||||
@@ -2,12 +2,12 @@
|
|||||||
|
|
||||||
**Iteration:** 7, Change requests
|
**Iteration:** 7, Change requests
|
||||||
**Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why.
|
**Intent:** Let a user restate a request to say how their requirements have changed, at a high level, and receive the delivered architecture updated from where it stood, with an account of what changed and why.
|
||||||
**Status:** planned. ADR-0006 was accepted on 2026-10-07 and the glossary updated (step 1). Build after iteration 6 step 11 has a passing evaluation run
|
**Status:** steps 1 to 6 and 8 built (2026-10-09), covered by the deterministic suite; no Change request has run with real models. Step 7's evaluation case and checks are built and its run is outstanding
|
||||||
**This directory covers:** steps 1 to 8
|
**This directory covers:** steps 1 to 8
|
||||||
|
|
||||||
## Sources of truth
|
## Sources of truth
|
||||||
|
|
||||||
- `docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md`: the decision and its four points to confirm. Status `proposed`.
|
- `docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md`: the decision and its four points. Accepted 2026-10-07.
|
||||||
- `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` and `../iter-06-target-architecture-graph/`: the ids along the Requirements chain that a change is traced through. Build iteration 6 first.
|
- `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` and `../iter-06-target-architecture-graph/`: the ids along the Requirements chain that a change is traced through. Build iteration 6 first.
|
||||||
- `CONTEXT.md`: today it says an Execution starts clean and a Customer case never feeds one Execution into another. Step 1 changes those entries; until then they stand.
|
- `CONTEXT.md`: today it says an Execution starts clean and a Customer case never feeds one Execution into another. Step 1 changes those entries; until then they stand.
|
||||||
- `iter-04-learning-and-skills/steps/step-15.md`: the Customer case layout, `execution.json` and the Intent history (`intent/history.json`) that a Baseline is found through. A Change request is the next statement in that history.
|
- `iter-04-learning-and-skills/steps/step-15.md`: the Customer case layout, `execution.json` and the Intent history (`intent/history.json`) that a Baseline is found through. A Change request is the next statement in that history.
|
||||||
@@ -17,12 +17,12 @@
|
|||||||
| Step | What | Size | Status |
|
| Step | What | Size | Status |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 |
|
| 1 | Accept ADR-0006 and update the glossary | S | complete: accepted 2026-10-07 |
|
||||||
| 2 | Start an Execution against a Baseline | M | planned |
|
| 2 | Start an Execution against a Baseline | M | built; not run with real models |
|
||||||
| 3 | Requirements revised with a change record | M | planned |
|
| 3 | Requirements revised with a change record | M | built; not run with real models |
|
||||||
| 4 | Later Specialists revise from the Baseline | M | planned |
|
| 4 | Later Specialists revise from the Baseline | M | built; not run with real models |
|
||||||
| 5 | Approvals show the change | S | planned |
|
| 5 | Approvals show the change | S | built; not run with real models |
|
||||||
| 6 | Change summary and version line in the case | S | planned |
|
| 6 | Change summary and version line in the case | S | built; not run with real models |
|
||||||
| 7 | Evaluation: a change case | M | planned |
|
| 7 | Evaluation: a change case | M | case and checks built; run outstanding |
|
||||||
| 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 |
|
| 8 | Remove the `legacy-baseline` case | S | complete: removed early on the owner's instruction, 2026-10-07 |
|
||||||
|
|
||||||
## Dependency graph
|
## Dependency graph
|
||||||
|
|||||||
14
.agents/plans/iter-07-change-requests/steps/step-2.md
Normal file
14
.agents/plans/iter-07-change-requests/steps/step-2.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Step 2 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09.
|
||||||
|
|
||||||
|
1. `app/baseline.py`: `load_baseline(output_root, customer_case, execution_id)` reads a Baseline from its files under the Customer case, through `execution.json`: its id, its request, the Baseline it was itself changed from, and every Deliverable with its attachments. It never needs the Baseline's checkpoint.
|
||||||
|
2. It refuses, with `BaselineError` (a `ValueError`), an id that is not an Execution id, an Execution with no files in that case, one that belongs to another Customer case (without saying which), one whose manifest status is not `completed`, and a Change request with no Customer case.
|
||||||
|
3. `ExecutionEngine.start(baseline=...)` loads the Baseline before the Execution's id is taken or its intent recorded, so a refusal leaves nothing behind. The loaded Baseline is in the state as `baseline`.
|
||||||
|
4. `record_intent` puts the Baseline on the new statement in `intent/history.json` and in its `NNN.md`; `execution.json` records it too. An Execution started without one has `"baseline": null` in both.
|
||||||
|
5. Callers: `baseline` in the `/generate` body and in the A2A message metadata. A bad Baseline is a 400 from `/generate` and a failed task over A2A.
|
||||||
|
6. Intake (iteration 5 step 12) no longer starts a message read as a change. It answers with the latest completed Execution of the case and how to name it as the Baseline, and starts nothing; `"new_request": true` in the metadata says it is new work and skips the reading. A caller who names a Baseline is not read at all.
|
||||||
|
|
||||||
|
Differs from the plan: nothing.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_change_requests.py` "step 2" (10), `tests/a2a/test_server.py` (3), `tests/execution/test_intake.py`.
|
||||||
17
.agents/plans/iter-07-change-requests/steps/step-3.md
Normal file
17
.agents/plans/iter-07-change-requests/steps/step-3.md
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
# Step 3 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09.
|
||||||
|
|
||||||
|
1. In a Change request every requirement in `requirements.json` carries a `change`: `unchanged`, `added`, `changed` or `removed`. Everything but `unchanged` carries a `reason`.
|
||||||
|
2. `check_requirement_changes` in `app/workflows/changes.py`, run by `check_submission` when the Deliverable has a Baseline, refuses:
|
||||||
|
- a requirement with no `change`, or a change with no reason;
|
||||||
|
- a Baseline requirement missing from the list (it must stay, marked `removed`);
|
||||||
|
- a requirement marked `added` whose id the Baseline already has, or marked anything else whose id it does not;
|
||||||
|
- a requirement marked `unchanged` whose text differs from the Baseline's;
|
||||||
|
- a list in which nothing is added, changed or removed.
|
||||||
|
3. A removed requirement stays in the file as the record of what was dropped. `read_list` leaves it out, so nothing downstream is asked to serve or check it.
|
||||||
|
4. `agents/requirements-analyst.md` says how to mark each requirement and to add a `Changes` heading. With a user present the draft is confirmed before submission by the rule of iteration 4 step 27.
|
||||||
|
|
||||||
|
Differs from the plan: the check that an `unchanged` requirement keeps its text was added. Without it a requirement could change quietly.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_change_requests.py` "step 3" (6).
|
||||||
14
.agents/plans/iter-07-change-requests/steps/step-4.md
Normal file
14
.agents/plans/iter-07-change-requests/steps/step-4.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Step 4 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09.
|
||||||
|
|
||||||
|
1. Each Specialist's context has `baseline` in a Change request: the Baseline's id and original request, the Baseline's version of its own Deliverable (not for the solution guide, which is assembled afresh), and `changes`, what has changed so far upstream.
|
||||||
|
2. Working copies start from the Baseline (`_start_from` in `app/nodes/execution_nodes.py`): the target architect's working graph is the delivered graph, and the IaC author's working copy is the delivered code. A Deliverable with `written_from` is seeded as standing, so the architecture description is carried over without running the describer when the Elements and Relationships are the same as the Baseline's.
|
||||||
|
3. `agents/partials/specialist-protocol.md` tells every Specialist how to revise: start from the delivered version, change only what the changed requirements reach, keep every id that still holds, never renumber.
|
||||||
|
4. `app/workflows/changes.py` compares two versions by id (`change_summary`) and traces `reach`: a changed requirement reaches the components that serve it, the Elements that realise them, the Relationships at those Elements, and every Check on any of them or with the requirement as its source; a changed recommendation reaches the Checks that come from it. Reach is traced through both versions. Anything that changed outside it is listed under `unrequested`.
|
||||||
|
|
||||||
|
Differs from the plan: the plan's test was that an unreached Element or Check is "identical to the Baseline's". The engine cannot make an agent leave something alone; it reports what changed outside the reach, and step 7's check fails a run where much did.
|
||||||
|
|
||||||
|
**Not run with real models**, so whether Specialists do keep ids and leave unreached parts alone is unknown.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_change_requests.py` "step 4" and "what changed" (6).
|
||||||
9
.agents/plans/iter-07-change-requests/steps/step-5.md
Normal file
9
.agents/plans/iter-07-change-requests/steps/step-5.md
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
# Step 5 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09.
|
||||||
|
|
||||||
|
1. `OrchestratorToolbox` is given the Baseline. In a Change request `request_approval` adds `baseline` and `changes` to the Approval: the part of the change summary that belongs to that Phase's Deliverables (`changes_for`), computed from the ids and never written by an agent.
|
||||||
|
2. `describe_prompt` in `app/presentation.py` renders it: each requirement added, changed or removed with its reason; ids added, changed and removed for components, recommendations, Elements, Relationships and Checks; and, apart, anything changed that no changed requirement reaches. A Phase that changed nothing says so.
|
||||||
|
3. An Approval outside a Change request has no `changes` key.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_change_requests.py` "step 5" (2).
|
||||||
12
.agents/plans/iter-07-change-requests/steps/step-6.md
Normal file
12
.agents/plans/iter-07-change-requests/steps/step-6.md
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
# Step 6 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09.
|
||||||
|
|
||||||
|
1. A Change request writes `changes.json` beside its Deliverables: the requirements' own account, then components, recommendations, Elements, Relationships and Checks added, changed and removed by id, and `unrequested`. `execution.json` names it under `changes` and names the Baseline.
|
||||||
|
2. Every Deliverable is written in full, as for any Execution, so the latest Execution of a case stands on its own.
|
||||||
|
3. `intent/history.json` lists each statement with its Baseline (step 2), so a case's Executions read as a line of versions.
|
||||||
|
4. `agents/packager.md`: the guide's executive summary opens with what changed and why, and names unrequested changes separately.
|
||||||
|
|
||||||
|
Differs from the plan: `changes.json` records ids, and reasons for requirements only. The plan asked for a reason on every item; a reason for a changed Check is whichever changed requirement or recommendation is its source, which a reader can follow from the ids.
|
||||||
|
|
||||||
|
Tests: `tests/execution/test_change_requests.py` "step 6" (1).
|
||||||
16
.agents/plans/iter-07-change-requests/steps/step-7.md
Normal file
16
.agents/plans/iter-07-change-requests/steps/step-7.md
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
# Step 7 — Implementation Detail
|
||||||
|
|
||||||
|
Built on 2026-10-09: the case and its checks. **The run is outstanding.**
|
||||||
|
|
||||||
|
1. A case may carry `change`, a statement of what has changed. The evaluation then runs the case's request to completion, pre-approved, and runs `change` as a Change request against it in the same Customer case. The Change request is what is scored.
|
||||||
|
2. `case-05-change-retention`: case 01's request, then "raw events must now be kept for 7 years instead of 30 days ... and must not be deletable during that time. Everything else stays as it is."
|
||||||
|
3. `score_change` adds four checks to the usual ones:
|
||||||
|
- `change_is_recorded`: at least one requirement is marked added, changed or removed;
|
||||||
|
- `changed_requirements_reach_checks`: each added or changed requirement is the source of a Check that was added or changed, and a removed one's Checks are gone;
|
||||||
|
- `change_is_contained`: no more than two items, or a tenth of the Baseline's graph, changed without being reached;
|
||||||
|
- `change_costs_less_than_baseline`: the Change request used fewer tokens than the Execution it changed.
|
||||||
|
4. The report records the Baseline's id, seconds and tokens.
|
||||||
|
|
||||||
|
Acceptance criteria: not yet tested. The case needs two Executions, roughly an hour and $8 to $10.
|
||||||
|
|
||||||
|
Tests: `tests/eval/test_scoring.py`, "a Change request stays contained" (5).
|
||||||
@@ -11,4 +11,6 @@ You produce the **solution guide** Deliverable: the Skill's packaging of every a
|
|||||||
- Include the architecture diagram, which is the `architecture.mmd` attachment of the Target architecture, unchanged as a `mermaid` code block, and the Terraform as code blocks.
|
- Include the architecture diagram, which is the `architecture.mmd` attachment of the Target architecture, unchanged as a `mermaid` code block, and the Terraform as code blocks.
|
||||||
- List the Target architecture's Checks with their validation verdicts in the validation section.
|
- List the Target architecture's Checks with their validation verdicts in the validation section.
|
||||||
|
|
||||||
|
In a Change request (`baseline` is set), open the executive summary with what changed and why, from `baseline.changes`: the requirements added, changed and removed with their reasons, then what that changed in the architecture. Name separately anything listed under `unrequested`: it changed without a changed requirement reaching it.
|
||||||
|
|
||||||
Your contract's sections are the template's eight top-level sections.
|
Your contract's sections are the template's eight top-level sections.
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ Your task message carries JSON context:
|
|||||||
- `contract`: what `submit_deliverable` checks. `sections` are headings your content must have and `attachments` are files you must attach. A data file among them must also keep every id traceable to the Deliverables in `dependencies`; the requirements-chain Skill gives each file's format.
|
- `contract`: what `submit_deliverable` checks. `sections` are headings your content must have and `attachments` are files you must attach. A data file among them must also keep every id traceable to the Deliverables in `dependencies`; the requirements-chain Skill gives each file's format.
|
||||||
- `draft`: the draft you sent the user with your last questions, with its `attachments`, or `null`.
|
- `draft`: the draft you sent the user with your last questions, with its `attachments`, or `null`.
|
||||||
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
|
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
|
||||||
|
- `baseline`: set only in a Change request, where the user has said how their requirements changed since an architecture was delivered. `deliverable` is that delivered version of your own Deliverable, and `changes` is what has changed so far: the requirements added, changed and removed, with the reason, and everything already revised from them. `null` otherwise.
|
||||||
- `pre_approved`: `true` when no user is available.
|
- `pre_approved`: `true` when no user is available.
|
||||||
|
|
||||||
Steps:
|
Steps:
|
||||||
@@ -20,4 +21,13 @@ Steps:
|
|||||||
3. Draft the Deliverable to its `contract`. When `pre_approved` is `false`, confirm the draft with the user before submitting: call `needs_input` with a short summary of the draft, asking them to approve it or request changes, and pass the complete draft as `draft` with its `attachments`. When `draft` is set and the latest entry in `answers` approves it, submit that draft as it is; when the user asked for changes, apply them to that draft. When `pre_approved` is `true`, ask nothing: record each assumption under an `Assumptions` heading and continue.
|
3. Draft the Deliverable to its `contract`. When `pre_approved` is `false`, confirm the draft with the user before submitting: call `needs_input` with a short summary of the draft, asking them to approve it or request changes, and pass the complete draft as `draft` with its `attachments`. When `draft` is set and the latest entry in `answers` approves it, submit that draft as it is; when the user asked for changes, apply them to that draft. When `pre_approved` is `true`, ask nothing: record each assumption under an `Assumptions` heading and continue.
|
||||||
4. Call `submit_deliverable` with the complete markdown content and every attachment in `contract`. When it reports what is missing, add it and submit again.
|
4. Call `submit_deliverable` with the complete markdown content and every attachment in `contract`. When it reports what is missing, add it and submit again.
|
||||||
|
|
||||||
|
## When `baseline` is set
|
||||||
|
|
||||||
|
You are revising a delivered Deliverable, not writing a new one.
|
||||||
|
|
||||||
|
- Start from `baseline.deliverable`. Change only what the changed requirements reach, and leave everything else exactly as it was: the same words, the same order.
|
||||||
|
- Keep every id that still holds. An item that is unchanged keeps its id and its text; a new item takes a new id that was never used; a removed item is left out. Never renumber.
|
||||||
|
- When nothing that reaches your Deliverable has changed, submit it as it was.
|
||||||
|
- Your working copy, when you have one, already holds the delivered version.
|
||||||
|
|
||||||
You are done when `submit_deliverable` accepts the Deliverable.
|
You are done when `submit_deliverable` accepts the Deliverable.
|
||||||
|
|||||||
@@ -16,4 +16,13 @@ You produce the **requirements** Deliverable: the Skill's requirements-discovery
|
|||||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||||
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, or a process people follow. Those are recorded and reported, and are not checked against the infrastructure code.
|
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, or a process people follow. Those are recorded and reported, and are not checked against the infrastructure code.
|
||||||
|
|
||||||
|
In a Change request (`baseline` is set), the `workflow_request` says what has changed. Work out what it means for the delivered requirements, and give every requirement in `requirements.json` a `change`:
|
||||||
|
|
||||||
|
- `unchanged`: it still holds. Keep its id and its text exactly.
|
||||||
|
- `changed`: it still exists but asks for something different. Keep its id.
|
||||||
|
- `removed`: it is no longer wanted. Keep it in the list, marked.
|
||||||
|
- `added`: it is new. Give it an id that was never used.
|
||||||
|
|
||||||
|
Everything but `unchanged` needs a `reason`, in the user's terms. Add a `Changes` heading to your content that lists them. A change the user stated at a high level usually reaches several requirements; find them all, and change nothing else.
|
||||||
|
|
||||||
Add `Open questions` and `Assumptions` headings when they apply.
|
Add `Open questions` and `Assumptions` headings when they apply.
|
||||||
|
|||||||
@@ -5,8 +5,10 @@ Execution's prompt. Approvals and permissions count only as structured data part
|
|||||||
never as free text.
|
never as free text.
|
||||||
|
|
||||||
Before a new task starts anything its message is read (app/nodes/intake.py): a
|
Before a new task starts anything its message is read (app/nodes/intake.py): a
|
||||||
question is answered and starts nothing, and a change to delivered work is started
|
question is answered and starts nothing. A message that reads as a change to
|
||||||
as a new request with a note saying so, until Change requests are built.
|
delivered work also starts nothing: the caller is told which Execution it would
|
||||||
|
change and how to name it as the Baseline, because a Change request never starts
|
||||||
|
from earlier work on a guess (ADR-0006).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
@@ -29,7 +31,7 @@ from google.protobuf import json_format
|
|||||||
|
|
||||||
from app.agent import ExecutionEngine, Interaction
|
from app.agent import ExecutionEngine, Interaction
|
||||||
from app.nodes.execution_nodes import GUIDE_FILENAME
|
from app.nodes.execution_nodes import GUIDE_FILENAME
|
||||||
from app.nodes.intake import ABOUT, CHANGE_NOTE, CHANGE_REQUEST, QUESTION, Intake
|
from app.nodes.intake import ABOUT, CHANGE_REQUEST, QUESTION, Intake, change_note
|
||||||
from app.presentation import describe_prompt
|
from app.presentation import describe_prompt
|
||||||
from app.workflows.deliverables import DeliverableKind
|
from app.workflows.deliverables import DeliverableKind
|
||||||
|
|
||||||
@@ -47,7 +49,6 @@ class SolutionArchitectExecutor(AgentExecutor):
|
|||||||
task_id = context.task_id or ""
|
task_id = context.task_id or ""
|
||||||
updater = TaskUpdater(event_queue, task_id, context.context_id or "")
|
updater = TaskUpdater(event_queue, task_id, context.context_id or "")
|
||||||
new = context.current_task is None
|
new = context.current_task is None
|
||||||
note: str | None = None
|
|
||||||
try:
|
try:
|
||||||
if new:
|
if new:
|
||||||
history = [context.message] if context.message is not None else []
|
history = [context.message] if context.message is not None else []
|
||||||
@@ -62,7 +63,12 @@ class SolutionArchitectExecutor(AgentExecutor):
|
|||||||
await updater.start_work()
|
await updater.start_work()
|
||||||
options = _options(context)
|
options = _options(context)
|
||||||
# A pre-approved caller is a batch job: it sends work, not questions.
|
# A pre-approved caller is a batch job: it sends work, not questions.
|
||||||
if self._intake and not options.get("pre_approved"):
|
if (
|
||||||
|
self._intake
|
||||||
|
and not options.get("pre_approved")
|
||||||
|
and not options.get("baseline")
|
||||||
|
and not options.get("new_request")
|
||||||
|
):
|
||||||
reading = await self._intake.read(
|
reading = await self._intake.read(
|
||||||
context.get_user_input(),
|
context.get_user_input(),
|
||||||
options.get("customer_case"),
|
options.get("customer_case"),
|
||||||
@@ -80,12 +86,27 @@ class SolutionArchitectExecutor(AgentExecutor):
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
if reading.kind == CHANGE_REQUEST:
|
if reading.kind == CHANGE_REQUEST and reading.latest:
|
||||||
note = CHANGE_NOTE
|
await updater.complete(
|
||||||
|
updater.new_agent_message(
|
||||||
|
[
|
||||||
|
new_text_part(change_note(reading.latest)),
|
||||||
|
new_data_part(
|
||||||
|
{
|
||||||
|
"status": "answered",
|
||||||
|
"intake": CHANGE_REQUEST,
|
||||||
|
"baseline": reading.latest,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
interaction = await self._engine.start(
|
interaction = await self._engine.start(
|
||||||
context.get_user_input(),
|
context.get_user_input(),
|
||||||
session_id=context.context_id,
|
session_id=context.context_id,
|
||||||
customer_case=options.get("customer_case"),
|
customer_case=options.get("customer_case"),
|
||||||
|
baseline=options.get("baseline"),
|
||||||
execution_id=task_id,
|
execution_id=task_id,
|
||||||
pre_approved=bool(options.get("pre_approved", False)),
|
pre_approved=bool(options.get("pre_approved", False)),
|
||||||
allow_cloud_read=options.get("allow_cloud_read"),
|
allow_cloud_read=options.get("allow_cloud_read"),
|
||||||
@@ -105,7 +126,7 @@ class SolutionArchitectExecutor(AgentExecutor):
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
await self._publish(interaction, updater, note if new else None)
|
await self._publish(interaction, updater)
|
||||||
|
|
||||||
async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None:
|
async def cancel(self, context: RequestContext, event_queue: EventQueue) -> None:
|
||||||
# The checkpoint stays; cancelling only ends the A2A task.
|
# The checkpoint stays; cancelling only ends the A2A task.
|
||||||
@@ -114,15 +135,12 @@ class SolutionArchitectExecutor(AgentExecutor):
|
|||||||
)
|
)
|
||||||
await updater.cancel()
|
await updater.cancel()
|
||||||
|
|
||||||
async def _publish(
|
async def _publish(self, interaction: Interaction, updater: TaskUpdater) -> None:
|
||||||
self, interaction: Interaction, updater: TaskUpdater, note: str | None = None
|
|
||||||
) -> None:
|
|
||||||
if interaction.status == "awaiting_user" and interaction.prompt is not None:
|
if interaction.status == "awaiting_user" and interaction.prompt is not None:
|
||||||
prompt = dict(interaction.prompt)
|
prompt = dict(interaction.prompt)
|
||||||
await updater.requires_input(
|
await updater.requires_input(
|
||||||
updater.new_agent_message(
|
updater.new_agent_message(
|
||||||
[
|
[
|
||||||
*([new_text_part(note)] if note else []),
|
|
||||||
new_text_part(describe_prompt(prompt)),
|
new_text_part(describe_prompt(prompt)),
|
||||||
new_data_part({"prompt": prompt}),
|
new_data_part({"prompt": prompt}),
|
||||||
]
|
]
|
||||||
|
|||||||
18
app/agent.py
18
app/agent.py
@@ -32,6 +32,7 @@ from app.adk.runtime import (
|
|||||||
SubmissionReview,
|
SubmissionReview,
|
||||||
ToolFactory,
|
ToolFactory,
|
||||||
)
|
)
|
||||||
|
from app.baseline import load_baseline
|
||||||
from app.checkpointing import open_checkpointer
|
from app.checkpointing import open_checkpointer
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.customer_case import record_intent
|
from app.customer_case import record_intent
|
||||||
@@ -125,13 +126,25 @@ class ExecutionEngine:
|
|||||||
allow_cloud_read: bool | None = None,
|
allow_cloud_read: bool | None = None,
|
||||||
execution_id: str | None = None,
|
execution_id: str | None = None,
|
||||||
scripted_answers: Sequence[Sequence[str]] = (),
|
scripted_answers: Sequence[Sequence[str]] = (),
|
||||||
|
baseline: str | None = None,
|
||||||
) -> Interaction:
|
) -> Interaction:
|
||||||
"""Start an Execution. `scripted_answers` are (regex, answer[, asker]) entries a
|
"""Start an Execution. `scripted_answers` are (regex, answer[, asker]) entries a
|
||||||
pre-approved Execution uses to answer questions, for evaluation runs."""
|
pre-approved Execution uses to answer questions, for evaluation runs.
|
||||||
|
|
||||||
|
`baseline` names a completed Execution of the same Customer case and makes
|
||||||
|
this a Change request (ADR-0006): the request says what has changed, and the
|
||||||
|
Execution starts from that Baseline's Deliverables. It is refused, before
|
||||||
|
anything runs or is recorded, when the Baseline cannot be one.
|
||||||
|
"""
|
||||||
customer_case = check_customer_case(customer_case)
|
customer_case = check_customer_case(customer_case)
|
||||||
execution_id = execution_id or str(uuid.uuid4())
|
execution_id = execution_id or str(uuid.uuid4())
|
||||||
if await self._exists(execution_id):
|
if await self._exists(execution_id):
|
||||||
raise ValueError(f"Execution {execution_id} already exists")
|
raise ValueError(f"Execution {execution_id} already exists")
|
||||||
|
loaded = (
|
||||||
|
load_baseline(self._output_root, customer_case, baseline)
|
||||||
|
if baseline
|
||||||
|
else None
|
||||||
|
)
|
||||||
if pre_approved and allow_cloud_read is None:
|
if pre_approved and allow_cloud_read is None:
|
||||||
allow_cloud_read = False
|
allow_cloud_read = False
|
||||||
session_id = session_id or str(uuid.uuid4())
|
session_id = session_id or str(uuid.uuid4())
|
||||||
@@ -141,6 +154,7 @@ class ExecutionEngine:
|
|||||||
execution_id=execution_id,
|
execution_id=execution_id,
|
||||||
session_id=session_id,
|
session_id=session_id,
|
||||||
workflow_request=workflow_request,
|
workflow_request=workflow_request,
|
||||||
|
baseline=baseline or None,
|
||||||
)
|
)
|
||||||
initial: dict[str, Any] = {
|
initial: dict[str, Any] = {
|
||||||
"execution_id": execution_id,
|
"execution_id": execution_id,
|
||||||
@@ -148,6 +162,8 @@ class ExecutionEngine:
|
|||||||
"customer_case": customer_case,
|
"customer_case": customer_case,
|
||||||
"intent_version": intent_version,
|
"intent_version": intent_version,
|
||||||
"workflow_request": workflow_request,
|
"workflow_request": workflow_request,
|
||||||
|
# The Baseline of a Change request, as read from its files; else None.
|
||||||
|
"baseline": loaded,
|
||||||
"pre_approved": pre_approved,
|
"pre_approved": pre_approved,
|
||||||
"allow_cloud_read": allow_cloud_read,
|
"allow_cloud_read": allow_cloud_read,
|
||||||
"ledger": Ledger(
|
"ledger": Ledger(
|
||||||
|
|||||||
95
app/baseline.py
Normal file
95
app/baseline.py
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
"""The Baseline of a Change request (ADR-0006).
|
||||||
|
|
||||||
|
A Change request is an Execution that starts from the Deliverables of a completed
|
||||||
|
Execution of the same Customer case. That Execution is its Baseline. It is read from
|
||||||
|
its files under the Customer case, through its manifest, so it does not depend on a
|
||||||
|
checkpoint and survives a restart. It is only ever read: a Baseline is never modified.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.states.execution_state import UNASSIGNED_CASE, case_dir, execution_dir
|
||||||
|
|
||||||
|
MANIFEST_FILENAME = "execution.json"
|
||||||
|
COMPLETED = "completed"
|
||||||
|
_EXECUTION_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$")
|
||||||
|
|
||||||
|
|
||||||
|
class BaselineError(ValueError):
|
||||||
|
"""The Execution named as a Baseline cannot be one; the message says why."""
|
||||||
|
|
||||||
|
|
||||||
|
def load_baseline(
|
||||||
|
output_root: Path, customer_case: str, execution_id: str
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""The Baseline as plain data: its id, its request and every Deliverable.
|
||||||
|
|
||||||
|
Raises BaselineError when the Execution does not exist, belongs to another
|
||||||
|
Customer case, or did not complete.
|
||||||
|
"""
|
||||||
|
if not isinstance(execution_id, str) or not _EXECUTION_ID.match(execution_id):
|
||||||
|
raise BaselineError(f"baseline {execution_id!r} is not an Execution id")
|
||||||
|
if customer_case == UNASSIGNED_CASE:
|
||||||
|
raise BaselineError(
|
||||||
|
"a Change request needs a customer_case: a Baseline is a completed "
|
||||||
|
"Execution of the same Customer case"
|
||||||
|
)
|
||||||
|
folder = execution_dir(output_root, execution_id, customer_case)
|
||||||
|
manifest_path = folder / MANIFEST_FILENAME
|
||||||
|
if not manifest_path.is_file():
|
||||||
|
elsewhere = [
|
||||||
|
path.parent.parent.parent.name
|
||||||
|
for path in case_dir(output_root, "x").parent.glob(
|
||||||
|
f"*/executions/{execution_id}/{MANIFEST_FILENAME}"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
if elsewhere:
|
||||||
|
# Never say which: one case must learn nothing about another.
|
||||||
|
raise BaselineError(
|
||||||
|
f"baseline {execution_id} belongs to another Customer case"
|
||||||
|
)
|
||||||
|
raise BaselineError(
|
||||||
|
f"baseline {execution_id} is not an Execution of Customer case "
|
||||||
|
f"{customer_case} with files on record"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise BaselineError(f"baseline {execution_id} has no readable record") from exc
|
||||||
|
if manifest.get("customer_case") not in (None, customer_case):
|
||||||
|
raise BaselineError(f"baseline {execution_id} belongs to another Customer case")
|
||||||
|
if manifest.get("status", COMPLETED) != COMPLETED:
|
||||||
|
raise BaselineError(
|
||||||
|
f"baseline {execution_id} did not complete ({manifest.get('status')}); "
|
||||||
|
"a Change request starts from a completed Execution"
|
||||||
|
)
|
||||||
|
|
||||||
|
deliverables: dict[str, dict[str, Any]] = {}
|
||||||
|
for kind, entry in (manifest.get("deliverables") or {}).items():
|
||||||
|
document = folder / str(entry.get("document", ""))
|
||||||
|
if not document.is_file():
|
||||||
|
raise BaselineError(f"baseline {execution_id} is missing its {kind} file")
|
||||||
|
attachments = {}
|
||||||
|
for name in entry.get("attachments") or ():
|
||||||
|
path = folder / name
|
||||||
|
if not path.is_file():
|
||||||
|
raise BaselineError(f"baseline {execution_id} is missing {name}")
|
||||||
|
attachments[name] = path.read_text(encoding="utf-8")
|
||||||
|
deliverables[kind] = {
|
||||||
|
"content": document.read_text(encoding="utf-8"),
|
||||||
|
"attachments": attachments,
|
||||||
|
}
|
||||||
|
if not deliverables:
|
||||||
|
raise BaselineError(f"baseline {execution_id} holds no Deliverables")
|
||||||
|
return {
|
||||||
|
"execution_id": execution_id,
|
||||||
|
"customer_case": customer_case,
|
||||||
|
"workflow_request": manifest.get("workflow_request", ""),
|
||||||
|
"baseline": manifest.get("baseline"), # what it was itself changed from
|
||||||
|
"deliverables": deliverables,
|
||||||
|
}
|
||||||
@@ -49,8 +49,13 @@ def record_intent(
|
|||||||
execution_id: str,
|
execution_id: str,
|
||||||
session_id: str,
|
session_id: str,
|
||||||
workflow_request: str,
|
workflow_request: str,
|
||||||
|
baseline: str | None = None,
|
||||||
) -> int:
|
) -> int:
|
||||||
"""Record a new statement of the customer's intent and return its version."""
|
"""Record a new statement of the customer's intent and return its version.
|
||||||
|
|
||||||
|
`baseline` is the Execution a Change request starts from; with it the history
|
||||||
|
shows the case's line of versions.
|
||||||
|
"""
|
||||||
folder = intent_dir(output_root, customer_case)
|
folder = intent_dir(output_root, customer_case)
|
||||||
folder.mkdir(parents=True, exist_ok=True)
|
folder.mkdir(parents=True, exist_ok=True)
|
||||||
history = read_history(output_root, customer_case)
|
history = read_history(output_root, customer_case)
|
||||||
@@ -60,7 +65,9 @@ def record_intent(
|
|||||||
(folder / statement).write_text(
|
(folder / statement).write_text(
|
||||||
f"# Intent {version}: {customer_case}\n\n"
|
f"# Intent {version}: {customer_case}\n\n"
|
||||||
f"- Recorded: {recorded_at}\n"
|
f"- Recorded: {recorded_at}\n"
|
||||||
f"- Execution: `{execution_id}`\n\n"
|
f"- Execution: `{execution_id}`\n"
|
||||||
|
+ (f"- Changes: `{baseline}`\n" if baseline else "")
|
||||||
|
+ "\n"
|
||||||
f"## What the customer wants\n\n{workflow_request.strip()}\n",
|
f"## What the customer wants\n\n{workflow_request.strip()}\n",
|
||||||
encoding="utf-8",
|
encoding="utf-8",
|
||||||
)
|
)
|
||||||
@@ -74,6 +81,7 @@ def record_intent(
|
|||||||
"session_id": session_id,
|
"session_id": session_id,
|
||||||
"status": IN_PROGRESS,
|
"status": IN_PROGRESS,
|
||||||
"execution": None,
|
"execution": None,
|
||||||
|
"baseline": baseline,
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
_write_history(folder, history)
|
_write_history(folder, history)
|
||||||
|
|||||||
@@ -69,6 +69,7 @@ def create_app(
|
|||||||
workflow_request,
|
workflow_request,
|
||||||
session_id=body.get("session_id"),
|
session_id=body.get("session_id"),
|
||||||
customer_case=body.get("customer_case"),
|
customer_case=body.get("customer_case"),
|
||||||
|
baseline=body.get("baseline"),
|
||||||
pre_approved=True,
|
pre_approved=True,
|
||||||
allow_cloud_read=bool(body.get("allow_cloud_read", False)),
|
allow_cloud_read=bool(body.get("allow_cloud_read", False)),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -53,12 +53,14 @@ from app.workflows.chain import (
|
|||||||
unimplementable_checks,
|
unimplementable_checks,
|
||||||
validation_outcome,
|
validation_outcome,
|
||||||
)
|
)
|
||||||
|
from app.workflows.changes import CHANGES_FILE, change_summary, is_empty
|
||||||
from app.workflows.deliverables import (
|
from app.workflows.deliverables import (
|
||||||
DEPENDENCY_GRAPH,
|
DEPENDENCY_GRAPH,
|
||||||
PHASE_ORDER,
|
PHASE_ORDER,
|
||||||
DeliverableKind,
|
DeliverableKind,
|
||||||
dependencies_for,
|
dependencies_for,
|
||||||
)
|
)
|
||||||
|
from app.workflows.target_architecture import GRAPH_ATTACHMENT
|
||||||
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
@@ -260,7 +262,9 @@ def make_execution_nodes(
|
|||||||
)
|
)
|
||||||
return finish_with("end_with_warnings", warnings=warnings)
|
return finish_with("end_with_warnings", warnings=warnings)
|
||||||
|
|
||||||
toolbox = OrchestratorToolbox(ledger, deliverables, pending_questions)
|
toolbox = OrchestratorToolbox(
|
||||||
|
ledger, deliverables, pending_questions, state.get("baseline")
|
||||||
|
)
|
||||||
refusals = list(state.get("refusals", []))
|
refusals = list(state.get("refusals", []))
|
||||||
update["refusals"] = refusals
|
update["refusals"] = refusals
|
||||||
view = TurnView(
|
view = TurnView(
|
||||||
@@ -382,6 +386,11 @@ def make_execution_nodes(
|
|||||||
kind = DeliverableKind(task_input["kind"])
|
kind = DeliverableKind(task_input["kind"])
|
||||||
working: dict[str, Any] = dict(task_input.get("working") or {})
|
working: dict[str, Any] = dict(task_input.get("working") or {})
|
||||||
upstream = task_input["context"].get("dependencies") or {}
|
upstream = task_input["context"].get("dependencies") or {}
|
||||||
|
before = task_input.get("baseline")
|
||||||
|
if before and not working:
|
||||||
|
# A Change request starts from the Baseline's version, not from nothing:
|
||||||
|
# its graph, its files and its description are the working copy.
|
||||||
|
_start_from(kind, before, working, task_input.get("baseline_basis"))
|
||||||
source = DEPENDENCY_GRAPH[kind].working_copy
|
source = DEPENDENCY_GRAPH[kind].working_copy
|
||||||
if source and source is not kind:
|
if source and source is not kind:
|
||||||
# The tools work on the files as they were last submitted.
|
# The tools work on the files as they were last submitted.
|
||||||
@@ -440,6 +449,7 @@ def make_execution_nodes(
|
|||||||
working=working,
|
working=working,
|
||||||
library=task_input.get("library"),
|
library=task_input.get("library"),
|
||||||
review=review if submission_review else None,
|
review=review if submission_review else None,
|
||||||
|
baseline=before,
|
||||||
# With a user present, their first sight of a Deliverable comes
|
# With a user present, their first sight of a Deliverable comes
|
||||||
# before it is submitted, whatever the Specialist intends.
|
# before it is submitted, whatever the Specialist intends.
|
||||||
confirm_first=confirm_drafts
|
confirm_first=confirm_drafts
|
||||||
@@ -707,6 +717,8 @@ def _specialist_input(
|
|||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
spec = DEPENDENCY_GRAPH[kind]
|
spec = DEPENDENCY_GRAPH[kind]
|
||||||
given = dependencies_for(kind, deliverables)
|
given = dependencies_for(kind, deliverables)
|
||||||
|
baseline = state.get("baseline") or None
|
||||||
|
before = (baseline or {}).get("deliverables", {}).get(kind.value)
|
||||||
return {
|
return {
|
||||||
"execution_id": state["execution_id"],
|
"execution_id": state["execution_id"],
|
||||||
# The Deliverables given as data files alone, whole, for reading on demand.
|
# The Deliverables given as data files alone, whole, for reading on demand.
|
||||||
@@ -723,6 +735,12 @@ def _specialist_input(
|
|||||||
# Whether the user has been shown a draft of this Deliverable. Answering some
|
# Whether the user has been shown a draft of this Deliverable. Answering some
|
||||||
# other question about it, such as what to do after it stopped, is not that.
|
# other question about it, such as what to do after it stopped, is not that.
|
||||||
"shown": kind.value in state.get("drafts_shown", []),
|
"shown": kind.value in state.get("drafts_shown", []),
|
||||||
|
# The Baseline's version of this Deliverable, in a Change request.
|
||||||
|
"baseline": before,
|
||||||
|
# What the Baseline's version was written from, to tell whether it still stands.
|
||||||
|
"baseline_basis": spec.written_from((baseline or {}).get("deliverables"))
|
||||||
|
if baseline and spec.written_from
|
||||||
|
else None,
|
||||||
"context": {
|
"context": {
|
||||||
"workflow_request": state["workflow_request"],
|
"workflow_request": state["workflow_request"],
|
||||||
"pre_approved": bool(state.get("pre_approved")),
|
"pre_approved": bool(state.get("pre_approved")),
|
||||||
@@ -736,10 +754,60 @@ def _specialist_input(
|
|||||||
state.get("working", {}).get(kind.value, {})
|
state.get("working", {}).get(kind.value, {})
|
||||||
),
|
),
|
||||||
"attempt": attempt,
|
"attempt": attempt,
|
||||||
|
# Set only in a Change request: what this Execution changes, and from what.
|
||||||
|
"baseline": _baseline_context(kind, baseline, before, deliverables)
|
||||||
|
if baseline
|
||||||
|
else None,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# The solution guide is assembled afresh from the revised Deliverables, so the
|
||||||
|
# Baseline's guide, the longest document there is, is not handed back to its author.
|
||||||
|
_NOT_HANDED_BACK = {DeliverableKind.SOLUTION_GUIDE}
|
||||||
|
|
||||||
|
|
||||||
|
def _baseline_context(
|
||||||
|
kind: DeliverableKind,
|
||||||
|
baseline: Mapping[str, Any],
|
||||||
|
before: Mapping[str, Any] | None,
|
||||||
|
deliverables: Mapping[str, Mapping[str, Any]],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
summary = change_summary(baseline.get("deliverables"), deliverables)
|
||||||
|
return {
|
||||||
|
"execution_id": baseline.get("execution_id"),
|
||||||
|
"original_request": baseline.get("workflow_request"),
|
||||||
|
"deliverable": dict(before)
|
||||||
|
if before and kind not in _NOT_HANDED_BACK
|
||||||
|
else None,
|
||||||
|
# What has changed so far, upstream of this Deliverable.
|
||||||
|
"changes": None if is_empty(summary) else summary,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _start_from(
|
||||||
|
kind: DeliverableKind,
|
||||||
|
before: Mapping[str, Any],
|
||||||
|
working: dict[str, Any],
|
||||||
|
basis: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
"""Make the Baseline's version of a Deliverable its working copy."""
|
||||||
|
attachments = dict(before.get("attachments", {}))
|
||||||
|
spec = DEPENDENCY_GRAPH[kind]
|
||||||
|
if kind is DeliverableKind.TARGET_ARCHITECTURE and GRAPH_ATTACHMENT in attachments:
|
||||||
|
workspace.keep_graph(working, attachments[GRAPH_ATTACHMENT])
|
||||||
|
if spec.working_copy is kind:
|
||||||
|
working[workspace.FILES] = attachments
|
||||||
|
if basis:
|
||||||
|
# Kept as it stands when what it was written from is the same as before.
|
||||||
|
working[STANDING] = {
|
||||||
|
"basis": basis,
|
||||||
|
"feedback": [],
|
||||||
|
"content": before.get("content", ""),
|
||||||
|
"attachments": attachments,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def _outcome_record(
|
def _outcome_record(
|
||||||
role: str, kind: DeliverableKind, attempt: int, outcome: Outcome
|
role: str, kind: DeliverableKind, attempt: int, outcome: Outcome
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
@@ -896,6 +964,15 @@ def _write_outputs(
|
|||||||
# submitted, or back in draft after something it depends on was reopened
|
# submitted, or back in draft after something it depends on was reopened
|
||||||
"status": ledger.status(DeliverableKind(kind)).value,
|
"status": ledger.status(DeliverableKind(kind)).value,
|
||||||
}
|
}
|
||||||
|
baseline = state.get("baseline") or None
|
||||||
|
if baseline:
|
||||||
|
# What this Change request changed against its Baseline, worked out from
|
||||||
|
# the ids in the two versions' data files.
|
||||||
|
files[CHANGES_FILE] = json.dumps(
|
||||||
|
change_summary(baseline.get("deliverables"), deliverables),
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
)
|
||||||
for name, content in files.items():
|
for name, content in files.items():
|
||||||
path = out_dir / name
|
path = out_dir / name
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
@@ -907,12 +984,15 @@ def _write_outputs(
|
|||||||
"session_id": state.get("session_id"),
|
"session_id": state.get("session_id"),
|
||||||
"intent_version": state.get("intent_version"),
|
"intent_version": state.get("intent_version"),
|
||||||
"workflow_request": state["workflow_request"],
|
"workflow_request": state["workflow_request"],
|
||||||
|
# The Execution this one changes, for a Change request.
|
||||||
|
"baseline": (state.get("baseline") or {}).get("execution_id"),
|
||||||
"status": status,
|
"status": status,
|
||||||
"warnings": list(state.get("warnings", [])),
|
"warnings": list(state.get("warnings", [])),
|
||||||
"phases": {
|
"phases": {
|
||||||
phase.value: ledger.phase_status(phase).value for phase in PHASE_ORDER
|
phase.value: ledger.phase_status(phase).value for phase in PHASE_ORDER
|
||||||
},
|
},
|
||||||
"solution_guide": GUIDE_FILENAME if guide else None,
|
"solution_guide": GUIDE_FILENAME if guide else None,
|
||||||
|
"changes": CHANGES_FILE if baseline else None,
|
||||||
"deliverables": by_kind,
|
"deliverables": by_kind,
|
||||||
}
|
}
|
||||||
out_dir.mkdir(parents=True, exist_ok=True)
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|||||||
@@ -4,9 +4,13 @@ Inside an Execution the router reads the user's replies. This is the same readin
|
|||||||
one step earlier, for a message that arrives with no Execution in progress:
|
one step earlier, for a message that arrives with no Execution in progress:
|
||||||
|
|
||||||
new request design this workload start an Execution
|
new request design this workload start an Execution
|
||||||
change request change what was delivered (ADR-0006; not built yet)
|
change request change what was delivered name the Baseline to change (ADR-0006)
|
||||||
question what can you do, how long... answer; start nothing
|
question what can you do, how long... answer; start nothing
|
||||||
|
|
||||||
|
A change is never started from earlier work on a guess: the caller chooses the
|
||||||
|
Baseline. So a message read as a change is answered with the Execution it would
|
||||||
|
change and how to name it, and starts nothing.
|
||||||
|
|
||||||
Without a decision, and whenever the reading is not confident, the message is taken
|
Without a decision, and whenever the reading is not confident, the message is taken
|
||||||
as a new request: that is what every message was before, and starting work nobody
|
as a new request: that is what every message was before, and starting work nobody
|
||||||
asked for is cheaper to put right than refusing work somebody did.
|
asked for is cheaper to put right than refusing work somebody did.
|
||||||
@@ -36,11 +40,18 @@ ABOUT = (
|
|||||||
"infrastructure code, validation and a packaged guide, asking for your approval "
|
"infrastructure code, validation and a packaged guide, asking for your approval "
|
||||||
"at each Phase. Nothing was started for this message."
|
"at each Phase. Nothing was started for this message."
|
||||||
)
|
)
|
||||||
CHANGE_NOTE = (
|
|
||||||
"This reads as a change to work already delivered for this Customer case. "
|
|
||||||
"Changing an earlier architecture in place is not available yet, so this is "
|
def change_note(baseline: str) -> str:
|
||||||
"being designed as a new request."
|
"""What a caller is told when their message reads as a change to delivered work."""
|
||||||
)
|
return (
|
||||||
|
"This reads as a change to the architecture already delivered for this "
|
||||||
|
f"Customer case, in Execution {baseline}. Nothing was started. To change "
|
||||||
|
f'that architecture from where it stood, send this again with "baseline" '
|
||||||
|
f'set to "{baseline}". To design it as new work instead, send it again '
|
||||||
|
'with "new_request" set to true.'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
_MESSAGE_LIMIT = 8_000
|
_MESSAGE_LIMIT = 8_000
|
||||||
_EARLIER_SHOWN = 5
|
_EARLIER_SHOWN = 5
|
||||||
@@ -52,6 +63,7 @@ class IntakeReading:
|
|||||||
confidence: float = 0.0
|
confidence: float = 0.0
|
||||||
judged: bool = False # False when no decision was available
|
judged: bool = False # False when no decision was available
|
||||||
earlier: int = 0 # finished Executions of the same Customer case
|
earlier: int = 0 # finished Executions of the same Customer case
|
||||||
|
latest: str | None = None # the most recent of them: what a change would change
|
||||||
|
|
||||||
|
|
||||||
def finished_intents(output_root: Path, customer_case: str) -> list[Mapping[str, Any]]:
|
def finished_intents(output_root: Path, customer_case: str) -> list[Mapping[str, Any]]:
|
||||||
@@ -110,7 +122,10 @@ class Intake:
|
|||||||
if kind == CHANGE_REQUEST and not earlier:
|
if kind == CHANGE_REQUEST and not earlier:
|
||||||
kind = NEW_REQUEST # there is nothing delivered to change
|
kind = NEW_REQUEST # there is nothing delivered to change
|
||||||
logger.info("[%s] intake: %s (%.2f)", scope, kind, confidence)
|
logger.info("[%s] intake: %s (%.2f)", scope, kind, confidence)
|
||||||
return IntakeReading(kind, confidence, judged=True, earlier=len(earlier))
|
latest = str(earlier[-1].get("execution_id")) if earlier else None
|
||||||
|
return IntakeReading(
|
||||||
|
kind, confidence, judged=True, earlier=len(earlier), latest=latest
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
|
def _state(message: str, earlier: Sequence[Mapping[str, Any]]) -> str:
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ def describe_prompt(prompt: Mapping[str, Any]) -> str:
|
|||||||
return (
|
return (
|
||||||
f"Please review the {approval['phase']} Phase: {approval['summary']}\n\n"
|
f"Please review the {approval['phase']} Phase: {approval['summary']}\n\n"
|
||||||
f"Deliverables: {', '.join(approval['deliverables'])}.\n\n"
|
f"Deliverables: {', '.join(approval['deliverables'])}.\n\n"
|
||||||
|
f"{_describe_changes(approval)}"
|
||||||
f"{_describe_target_architecture(approval.get('target_architecture'))}"
|
f"{_describe_target_architecture(approval.get('target_architecture'))}"
|
||||||
f"To approve, reply with the data {json.dumps(reply)}. "
|
f"To approve, reply with the data {json.dumps(reply)}. "
|
||||||
f"To request changes, reply with {json.dumps(changes)}. "
|
f"To request changes, reply with {json.dumps(changes)}. "
|
||||||
@@ -47,6 +48,42 @@ def describe_prompt(prompt: Mapping[str, Any]) -> str:
|
|||||||
return "\n".join(lines) + "\n\nReply in text; answer every question you can."
|
return "\n".join(lines) + "\n\nReply in text; answer every question you can."
|
||||||
|
|
||||||
|
|
||||||
|
_CHANGE_WORDS = {"added": "Added", "changed": "Changed", "removed": "Removed"}
|
||||||
|
|
||||||
|
|
||||||
|
def _describe_changes(approval: Mapping[str, Any]) -> str:
|
||||||
|
"""In a Change request: what this Phase changes against the Baseline. It comes
|
||||||
|
from the ids in the two versions, not from an agent's account."""
|
||||||
|
if "changes" not in approval:
|
||||||
|
return ""
|
||||||
|
changes = approval["changes"]
|
||||||
|
lines = [f"This changes the architecture delivered in {approval.get('baseline')}."]
|
||||||
|
if not changes:
|
||||||
|
lines.append("Nothing in this Phase differs from it.")
|
||||||
|
for item in changes.get("requirements", []):
|
||||||
|
lines.append(
|
||||||
|
f"- Requirement {item['id']} {item['change']}: {item['text']} "
|
||||||
|
f"({item['reason']})"
|
||||||
|
)
|
||||||
|
for part in (
|
||||||
|
"components",
|
||||||
|
"recommendations",
|
||||||
|
"elements",
|
||||||
|
"relationships",
|
||||||
|
"checks",
|
||||||
|
):
|
||||||
|
for change, word in _CHANGE_WORDS.items():
|
||||||
|
ids = changes.get(part, {}).get(change)
|
||||||
|
if ids:
|
||||||
|
lines.append(f"- {word} {part}: {', '.join(ids)}")
|
||||||
|
for part, ids in changes.get("unrequested", {}).items():
|
||||||
|
lines.append(
|
||||||
|
f"- Changed without being asked for ({part}): {', '.join(ids)}. No "
|
||||||
|
"changed requirement reaches these."
|
||||||
|
)
|
||||||
|
return "\n".join(lines) + "\n\n"
|
||||||
|
|
||||||
|
|
||||||
def _describe_target_architecture(summary: Mapping[str, Any] | None) -> str:
|
def _describe_target_architecture(summary: Mapping[str, Any] | None) -> str:
|
||||||
"""What the design Approval commits to: no code is written until it is granted."""
|
"""What the design Approval commits to: no code is written until it is granted."""
|
||||||
if not summary:
|
if not summary:
|
||||||
|
|||||||
@@ -56,6 +56,7 @@ class ExecutionState(TypedDict, total=False):
|
|||||||
customer_case: str # the Customer case this Execution belongs to
|
customer_case: str # the Customer case this Execution belongs to
|
||||||
intent_version: int # which statement of the case's intent started this Execution
|
intent_version: int # which statement of the case's intent started this Execution
|
||||||
workflow_request: str
|
workflow_request: str
|
||||||
|
baseline: dict[str, Any] | None # a Change request's Baseline, read from its files
|
||||||
pre_approved: bool
|
pre_approved: bool
|
||||||
allow_cloud_read: bool | None
|
allow_cloud_read: bool | None
|
||||||
|
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ from app.tools.workspace import (
|
|||||||
working_verdicts,
|
working_verdicts,
|
||||||
)
|
)
|
||||||
from app.workflows.chain import VERDICTS_FILE, read_graph
|
from app.workflows.chain import VERDICTS_FILE, read_graph
|
||||||
|
from app.workflows.changes import change_summary, changes_for
|
||||||
from app.workflows.deliverables import (
|
from app.workflows.deliverables import (
|
||||||
DEPENDENCY_GRAPH,
|
DEPENDENCY_GRAPH,
|
||||||
DeliverableKind,
|
DeliverableKind,
|
||||||
@@ -82,10 +83,13 @@ class OrchestratorToolbox:
|
|||||||
ledger: Ledger,
|
ledger: Ledger,
|
||||||
deliverables: Mapping[str, Mapping[str, Any]],
|
deliverables: Mapping[str, Mapping[str, Any]],
|
||||||
pending_questions: Sequence[Mapping[str, Any]],
|
pending_questions: Sequence[Mapping[str, Any]],
|
||||||
|
baseline: Mapping[str, Any] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.ledger = ledger
|
self.ledger = ledger
|
||||||
self._deliverables = deliverables
|
self._deliverables = deliverables
|
||||||
self._pending_questions = pending_questions
|
self._pending_questions = pending_questions
|
||||||
|
# A Change request's Baseline: with it, an Approval shows what changed.
|
||||||
|
self._baseline = baseline
|
||||||
self.started: dict[DeliverableKind, int] = {}
|
self.started: dict[DeliverableKind, int] = {}
|
||||||
self.questions: list[dict[str, str]] = []
|
self.questions: list[dict[str, str]] = []
|
||||||
self.approval: dict[str, Any] | None = None
|
self.approval: dict[str, Any] | None = None
|
||||||
@@ -213,6 +217,14 @@ class OrchestratorToolbox:
|
|||||||
"summary": summary,
|
"summary": summary,
|
||||||
"deliverables": [kind.value for kind in deliverables_in(phase_id)],
|
"deliverables": [kind.value for kind in deliverables_in(phase_id)],
|
||||||
}
|
}
|
||||||
|
if self._baseline:
|
||||||
|
# Worked out from the ids, not written by an agent: what this Phase's
|
||||||
|
# Deliverables add, change and remove against the Baseline.
|
||||||
|
self.approval["baseline"] = self._baseline.get("execution_id")
|
||||||
|
self.approval["changes"] = changes_for(
|
||||||
|
change_summary(self._baseline.get("deliverables"), self._deliverables),
|
||||||
|
self.approval["deliverables"],
|
||||||
|
)
|
||||||
graph = self._target_architecture()
|
graph = self._target_architecture()
|
||||||
if graph and phase_id is DEPENDENCY_GRAPH[_TARGET].phase:
|
if graph and phase_id is DEPENDENCY_GRAPH[_TARGET].phase:
|
||||||
# What the user is approving before any code is written.
|
# What the user is approving before any code is written.
|
||||||
@@ -271,6 +283,7 @@ def specialist_tools(
|
|||||||
library: Mapping[str, Mapping[str, Any]] | None = None,
|
library: Mapping[str, Mapping[str, Any]] | None = None,
|
||||||
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
|
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
|
||||||
confirm_first: bool = False,
|
confirm_first: bool = False,
|
||||||
|
baseline: Mapping[str, Any] | None = None,
|
||||||
) -> list[ToolSpec]:
|
) -> list[ToolSpec]:
|
||||||
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
|
||||||
the cloud-read gate.
|
the cloud-read gate.
|
||||||
@@ -284,6 +297,7 @@ def specialist_tools(
|
|||||||
`confirm_first` is set while a user is present and has not yet been asked about
|
`confirm_first` is set while a user is present and has not yet been asked about
|
||||||
this Deliverable: a submission is then refused until the Specialist has put its
|
this Deliverable: a submission is then refused until the Specialist has put its
|
||||||
draft to them. A rule, because an instruction alone was not always followed.
|
draft to them. A rule, because an instruction alone was not always followed.
|
||||||
|
`baseline` is the Baseline's version of this Deliverable in a Change request.
|
||||||
"""
|
"""
|
||||||
working = {} if working is None else working
|
working = {} if working is None else working
|
||||||
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
|
||||||
@@ -319,7 +333,9 @@ def specialist_tools(
|
|||||||
"attachment names must be relative paths without '..': "
|
"attachment names must be relative paths without '..': "
|
||||||
+ ", ".join(unsafe)
|
+ ", ".join(unsafe)
|
||||||
)
|
)
|
||||||
missing = check_submission(kind, content, attachments, upstream)
|
missing = check_submission(
|
||||||
|
kind, content, attachments, upstream, baseline=baseline
|
||||||
|
)
|
||||||
if missing:
|
if missing:
|
||||||
return ToolResult(
|
return ToolResult(
|
||||||
"Not submitted. The Deliverable is missing: " + "; ".join(missing)
|
"Not submitted. The Deliverable is missing: " + "; ".join(missing)
|
||||||
|
|||||||
@@ -42,14 +42,22 @@ Upstream = Mapping[str, Mapping[str, Any]]
|
|||||||
|
|
||||||
|
|
||||||
def read_list(upstream: Upstream | None, kind: str, name: str) -> list[dict[str, Any]]:
|
def read_list(upstream: Upstream | None, kind: str, name: str) -> list[dict[str, Any]]:
|
||||||
"""An upstream data file as a list of objects; empty when absent or unreadable."""
|
"""An upstream data file as a list of objects; empty when absent or unreadable.
|
||||||
|
|
||||||
|
An entry a Change request marked removed is left out: it stays in the file as
|
||||||
|
the record of what was dropped, and asks nothing of what follows.
|
||||||
|
"""
|
||||||
text = ((upstream or {}).get(kind) or {}).get("attachments", {}).get(name, "")
|
text = ((upstream or {}).get(kind) or {}).get("attachments", {}).get(name, "")
|
||||||
try:
|
try:
|
||||||
data = json.loads(text)
|
data = json.loads(text)
|
||||||
except ValueError:
|
except ValueError:
|
||||||
return []
|
return []
|
||||||
return (
|
return (
|
||||||
[item for item in data if isinstance(item, dict)]
|
[
|
||||||
|
item
|
||||||
|
for item in data
|
||||||
|
if isinstance(item, dict) and item.get("change") != "removed"
|
||||||
|
]
|
||||||
if isinstance(data, list)
|
if isinstance(data, list)
|
||||||
else []
|
else []
|
||||||
)
|
)
|
||||||
|
|||||||
326
app/workflows/changes.py
Normal file
326
app/workflows/changes.py
Normal file
@@ -0,0 +1,326 @@
|
|||||||
|
"""What a Change request changed against its Baseline (ADR-0006).
|
||||||
|
|
||||||
|
Ids are stable from one version of a Customer case's architecture to the next: a
|
||||||
|
requirement, component, recommendation, Element, Relationship or Check that still
|
||||||
|
holds keeps its id. That is what lets code, with no model, say what was added, what
|
||||||
|
changed and what was removed; trace from each changed requirement to everything it
|
||||||
|
reaches; and list apart anything that changed without being reached. Nobody asked
|
||||||
|
for those, so a reader should see them as such.
|
||||||
|
|
||||||
|
This module is pure: data in, data out.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.workflows.chain import (
|
||||||
|
COMPONENTS_FILE,
|
||||||
|
RECOMMENDATIONS_FILE,
|
||||||
|
REQUIREMENTS_FILE,
|
||||||
|
)
|
||||||
|
from app.workflows.target_architecture import GRAPH_ATTACHMENT
|
||||||
|
|
||||||
|
CHANGES_FILE = "changes.json"
|
||||||
|
UNCHANGED, ADDED, CHANGED, REMOVED = "unchanged", "added", "changed", "removed"
|
||||||
|
CHANGE_KINDS = (UNCHANGED, ADDED, CHANGED, REMOVED)
|
||||||
|
|
||||||
|
Deliverables = Mapping[str, Mapping[str, Any]]
|
||||||
|
|
||||||
|
|
||||||
|
def _items(deliverables: Deliverables | None, kind: str, name: str) -> list[dict]:
|
||||||
|
"""A data file as a list of objects, removed requirements included."""
|
||||||
|
text = ((deliverables or {}).get(kind) or {}).get("attachments", {}).get(name, "")
|
||||||
|
try:
|
||||||
|
data = json.loads(text)
|
||||||
|
except ValueError:
|
||||||
|
return []
|
||||||
|
return (
|
||||||
|
[item for item in data if isinstance(item, dict)]
|
||||||
|
if isinstance(data, list)
|
||||||
|
else []
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _graph(deliverables: Deliverables | None) -> dict[str, list[dict]]:
|
||||||
|
text = (
|
||||||
|
((deliverables or {}).get("target_architecture") or {})
|
||||||
|
.get("attachments", {})
|
||||||
|
.get(GRAPH_ATTACHMENT, "")
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
data = json.loads(text)
|
||||||
|
except ValueError:
|
||||||
|
data = {}
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
data = {}
|
||||||
|
return {
|
||||||
|
key: [item for item in data.get(key, []) if isinstance(item, dict)]
|
||||||
|
for key in ("elements", "relationships", "checks")
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# --- the requirements say what changed --------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def check_requirement_changes(
|
||||||
|
attachments: Mapping[str, str], baseline: Mapping[str, Any] | None
|
||||||
|
) -> list[str]:
|
||||||
|
"""What is wrong with a Change request's requirements.json against the Baseline's.
|
||||||
|
|
||||||
|
Every requirement carries a `change`. One that still holds keeps its id and is
|
||||||
|
`unchanged`, with its text as it was. One that differs is `changed`; one no
|
||||||
|
longer wanted stays in the list as `removed`; a new one takes a new id and is
|
||||||
|
`added`. Everything but `unchanged` says why.
|
||||||
|
"""
|
||||||
|
name = REQUIREMENTS_FILE
|
||||||
|
before = {
|
||||||
|
item["id"]: item
|
||||||
|
for item in _items({"requirements": baseline or {}}, "requirements", name)
|
||||||
|
if isinstance(item.get("id"), str) and item.get("change") != REMOVED
|
||||||
|
}
|
||||||
|
revised = _items(
|
||||||
|
{"requirements": {"attachments": attachments}}, "requirements", name
|
||||||
|
)
|
||||||
|
problems: list[str] = []
|
||||||
|
seen = set()
|
||||||
|
for item in revised:
|
||||||
|
identifier = item.get("id")
|
||||||
|
if not isinstance(identifier, str):
|
||||||
|
continue
|
||||||
|
seen.add(identifier)
|
||||||
|
change = item.get("change")
|
||||||
|
if change not in CHANGE_KINDS:
|
||||||
|
problems.append(
|
||||||
|
f"{name}: {identifier} needs a `change`: one of "
|
||||||
|
f"{', '.join(CHANGE_KINDS)}"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
reason = item.get("reason")
|
||||||
|
if change != UNCHANGED and not (isinstance(reason, str) and reason.strip()):
|
||||||
|
problems.append(f"{name}: {identifier} is {change} and needs a `reason`")
|
||||||
|
if change == ADDED and identifier in before:
|
||||||
|
problems.append(
|
||||||
|
f"{name}: {identifier} is marked added, but the Baseline has that "
|
||||||
|
"id; a new requirement takes a new id"
|
||||||
|
)
|
||||||
|
if change != ADDED and identifier not in before:
|
||||||
|
problems.append(
|
||||||
|
f"{name}: {identifier} is marked {change}, but the Baseline has no "
|
||||||
|
"such requirement; mark a new one added"
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
change == UNCHANGED
|
||||||
|
and identifier in before
|
||||||
|
and _text(item) != _text(before[identifier])
|
||||||
|
):
|
||||||
|
problems.append(
|
||||||
|
f"{name}: {identifier} is marked unchanged, but its text differs "
|
||||||
|
"from the Baseline's; restore it or mark it changed"
|
||||||
|
)
|
||||||
|
problems += [
|
||||||
|
f"{name}: Baseline requirement {identifier} is missing; keep it, or keep it "
|
||||||
|
"marked removed with the reason"
|
||||||
|
for identifier in before
|
||||||
|
if identifier not in seen
|
||||||
|
]
|
||||||
|
if revised and all(item.get("change") == UNCHANGED for item in revised):
|
||||||
|
problems.append(
|
||||||
|
f"{name}: nothing is marked added, changed or removed; a Change request "
|
||||||
|
"changes at least one requirement"
|
||||||
|
)
|
||||||
|
return problems
|
||||||
|
|
||||||
|
|
||||||
|
def _text(item: Mapping[str, Any]) -> str:
|
||||||
|
return " ".join(str(item.get("text", "")).split())
|
||||||
|
|
||||||
|
|
||||||
|
def requirement_changes(deliverables: Deliverables | None) -> list[dict[str, str]]:
|
||||||
|
"""The requirements a Change request added, changed or removed, with the reason."""
|
||||||
|
return [
|
||||||
|
{
|
||||||
|
"id": str(item.get("id")),
|
||||||
|
"change": str(item["change"]),
|
||||||
|
"reason": str(item.get("reason", "")),
|
||||||
|
"text": str(item.get("text", "")),
|
||||||
|
}
|
||||||
|
for item in _items(deliverables, "requirements", REQUIREMENTS_FILE)
|
||||||
|
if item.get("change") in (ADDED, CHANGED, REMOVED)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# --- comparing two versions -------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _compare(
|
||||||
|
before: Sequence[Mapping[str, Any]],
|
||||||
|
after: Sequence[Mapping[str, Any]] | None,
|
||||||
|
key: str = "id",
|
||||||
|
ignore: Sequence[str] = ("change", "reason"),
|
||||||
|
) -> dict[str, list[str]]:
|
||||||
|
"""Ids added, changed and removed between two lists of objects.
|
||||||
|
|
||||||
|
`after` is None while the revised Deliverable has not been produced: nothing
|
||||||
|
has changed yet, which is not the same as everything having been removed.
|
||||||
|
"""
|
||||||
|
if after is None:
|
||||||
|
return {ADDED: [], CHANGED: [], REMOVED: []}
|
||||||
|
|
||||||
|
def body(item: Mapping[str, Any]) -> dict[str, Any]:
|
||||||
|
return {name: value for name, value in item.items() if name not in ignore}
|
||||||
|
|
||||||
|
old = {str(item.get(key)): body(item) for item in before if item.get(key)}
|
||||||
|
new = {str(item.get(key)): body(item) for item in after if item.get(key)}
|
||||||
|
return {
|
||||||
|
ADDED: [identifier for identifier in new if identifier not in old],
|
||||||
|
CHANGED: [
|
||||||
|
identifier
|
||||||
|
for identifier in new
|
||||||
|
if identifier in old and new[identifier] != old[identifier]
|
||||||
|
],
|
||||||
|
REMOVED: [identifier for identifier in old if identifier not in new],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def reach(
|
||||||
|
changed: Sequence[str],
|
||||||
|
components: Sequence[Mapping[str, Any]],
|
||||||
|
recommendations_changed: Sequence[str],
|
||||||
|
graph: Mapping[str, Sequence[Mapping[str, Any]]],
|
||||||
|
) -> set[str]:
|
||||||
|
"""The Elements, Relationships and Checks a set of changed requirements reaches.
|
||||||
|
|
||||||
|
A changed requirement reaches the components that serve it, the Elements that
|
||||||
|
realise those components, the Relationships at those Elements, and every Check
|
||||||
|
on any of them or with the requirement as its source. A recommendation that
|
||||||
|
changed reaches the Checks that come from it and what they apply to.
|
||||||
|
"""
|
||||||
|
sources = set(changed) | set(recommendations_changed)
|
||||||
|
served = {
|
||||||
|
str(component.get("id"))
|
||||||
|
for component in components
|
||||||
|
if set(component.get("requirements") or ()) & set(changed)
|
||||||
|
}
|
||||||
|
reached = {
|
||||||
|
str(element.get("id"))
|
||||||
|
for element in graph.get("elements", ())
|
||||||
|
if element.get("component") in served
|
||||||
|
}
|
||||||
|
for check in graph.get("checks", ()):
|
||||||
|
if check.get("source") in sources:
|
||||||
|
reached.add(str(check.get("id")))
|
||||||
|
reached.add(str(check.get("applies_to")))
|
||||||
|
reached |= {
|
||||||
|
str(relationship.get("id"))
|
||||||
|
for relationship in graph.get("relationships", ())
|
||||||
|
if relationship.get("from") in reached or relationship.get("to") in reached
|
||||||
|
}
|
||||||
|
reached |= {
|
||||||
|
str(check.get("id"))
|
||||||
|
for check in graph.get("checks", ())
|
||||||
|
if check.get("applies_to") in reached
|
||||||
|
}
|
||||||
|
return reached
|
||||||
|
|
||||||
|
|
||||||
|
def change_summary(
|
||||||
|
baseline: Deliverables | None, current: Deliverables | None
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""What a Change request changed, as changes.json holds it.
|
||||||
|
|
||||||
|
Requirements carry their own account. Everything else is compared by id. What
|
||||||
|
changed inside the reach of the changed requirements is listed as requested;
|
||||||
|
anything else that changed is listed apart as unrequested.
|
||||||
|
"""
|
||||||
|
requirements = requirement_changes(current)
|
||||||
|
changed_ids = [item["id"] for item in requirements]
|
||||||
|
produced = current or {}
|
||||||
|
recommendations = _compare(
|
||||||
|
_items(baseline, "design_recommendations", RECOMMENDATIONS_FILE),
|
||||||
|
_items(current, "design_recommendations", RECOMMENDATIONS_FILE)
|
||||||
|
if "design_recommendations" in produced
|
||||||
|
else None,
|
||||||
|
)
|
||||||
|
components = _compare(
|
||||||
|
_items(baseline, "technical_decomposition", COMPONENTS_FILE),
|
||||||
|
_items(current, "technical_decomposition", COMPONENTS_FILE)
|
||||||
|
if "technical_decomposition" in produced
|
||||||
|
else None,
|
||||||
|
)
|
||||||
|
old_graph, new_graph = _graph(baseline), _graph(current)
|
||||||
|
designed = "target_architecture" in produced
|
||||||
|
# Reach is traced through both versions: what a requirement used to touch can
|
||||||
|
# change or go, as well as what it touches now.
|
||||||
|
touched_recommendations = recommendations[ADDED] + recommendations[CHANGED]
|
||||||
|
reached = reach(
|
||||||
|
changed_ids,
|
||||||
|
_items(current, "technical_decomposition", COMPONENTS_FILE),
|
||||||
|
touched_recommendations,
|
||||||
|
new_graph,
|
||||||
|
) | reach(
|
||||||
|
changed_ids,
|
||||||
|
_items(baseline, "technical_decomposition", COMPONENTS_FILE),
|
||||||
|
touched_recommendations + recommendations[REMOVED],
|
||||||
|
old_graph,
|
||||||
|
)
|
||||||
|
graph: dict[str, dict[str, list[str]]] = {}
|
||||||
|
unrequested: dict[str, list[str]] = {}
|
||||||
|
for part in ("elements", "relationships", "checks"):
|
||||||
|
compared = _compare(old_graph[part], new_graph[part] if designed else None)
|
||||||
|
graph[part] = compared
|
||||||
|
outside = [
|
||||||
|
identifier
|
||||||
|
for change in (ADDED, CHANGED, REMOVED)
|
||||||
|
for identifier in compared[change]
|
||||||
|
if identifier not in reached
|
||||||
|
]
|
||||||
|
if outside:
|
||||||
|
unrequested[part] = outside
|
||||||
|
return {
|
||||||
|
"requirements": requirements,
|
||||||
|
"components": components,
|
||||||
|
"recommendations": recommendations,
|
||||||
|
**graph,
|
||||||
|
# Changed without being reached by any changed requirement.
|
||||||
|
"unrequested": unrequested,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def is_empty(summary: Mapping[str, Any]) -> bool:
|
||||||
|
return not summary.get("requirements") and not any(
|
||||||
|
summary.get(part, {}).get(change)
|
||||||
|
for part in (
|
||||||
|
"components",
|
||||||
|
"recommendations",
|
||||||
|
"elements",
|
||||||
|
"relationships",
|
||||||
|
"checks",
|
||||||
|
)
|
||||||
|
for change in (ADDED, CHANGED, REMOVED)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# What an Approval of each kind of Deliverable shows of the change.
|
||||||
|
_SHOWN_WITH = {
|
||||||
|
"requirements": ("requirements",),
|
||||||
|
"technical_decomposition": ("components",),
|
||||||
|
"design_recommendations": ("recommendations",),
|
||||||
|
"target_architecture": ("elements", "relationships", "checks", "unrequested"),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def changes_for(summary: Mapping[str, Any], kinds: Sequence[str]) -> dict[str, Any]:
|
||||||
|
"""The part of a change summary that belongs to the Deliverables being approved,
|
||||||
|
with the parts that hold nothing left out."""
|
||||||
|
shown: dict[str, Any] = {}
|
||||||
|
for kind in kinds:
|
||||||
|
for part in _SHOWN_WITH.get(kind, ()):
|
||||||
|
value = summary.get(part)
|
||||||
|
if isinstance(value, Mapping):
|
||||||
|
value = {name: ids for name, ids in value.items() if ids}
|
||||||
|
if value:
|
||||||
|
shown[part] = value
|
||||||
|
return shown
|
||||||
@@ -30,6 +30,7 @@ from app.workflows.chain import (
|
|||||||
check_verdicts,
|
check_verdicts,
|
||||||
graph_structure,
|
graph_structure,
|
||||||
)
|
)
|
||||||
|
from app.workflows.changes import check_requirement_changes
|
||||||
from app.workflows.target_architecture import GRAPH_ATTACHMENT
|
from app.workflows.target_architecture import GRAPH_ATTACHMENT
|
||||||
|
|
||||||
|
|
||||||
@@ -286,11 +287,14 @@ def check_submission(
|
|||||||
content: str,
|
content: str,
|
||||||
attachments: Mapping[str, str],
|
attachments: Mapping[str, str],
|
||||||
upstream: Upstream | None = None,
|
upstream: Upstream | None = None,
|
||||||
|
baseline: Mapping[str, object] | None = None,
|
||||||
) -> tuple[str, ...]:
|
) -> tuple[str, ...]:
|
||||||
"""Return what a submission lacks against its contract; empty means it's complete.
|
"""Return what a submission lacks against its contract; empty means it's complete.
|
||||||
|
|
||||||
`upstream` is the Deliverables this one builds on, by kind. With it, the links
|
`upstream` is the Deliverables this one builds on, by kind. With it, the links
|
||||||
along the Requirements chain are checked too.
|
along the Requirements chain are checked too. `baseline` is the Baseline's
|
||||||
|
version of this Deliverable in a Change request; with it, the account of what
|
||||||
|
changed is checked.
|
||||||
"""
|
"""
|
||||||
if not content.strip():
|
if not content.strip():
|
||||||
return ("content: must not be empty",)
|
return ("content: must not be empty",)
|
||||||
@@ -311,4 +315,11 @@ def check_submission(
|
|||||||
name, check = DATA_CHECKS[kind]
|
name, check = DATA_CHECKS[kind]
|
||||||
if attachments.get(name, "").strip():
|
if attachments.get(name, "").strip():
|
||||||
missing.extend(check(attachments, upstream))
|
missing.extend(check(attachments, upstream))
|
||||||
|
if (
|
||||||
|
baseline is not None
|
||||||
|
and kind is DeliverableKind.REQUIREMENTS
|
||||||
|
and attachments.get(REQUIREMENTS_FILE, "").strip()
|
||||||
|
and not missing
|
||||||
|
):
|
||||||
|
missing.extend(check_requirement_changes(attachments, baseline))
|
||||||
return tuple(missing)
|
return tuple(missing)
|
||||||
|
|||||||
@@ -306,5 +306,66 @@
|
|||||||
"seconds": 3400,
|
"seconds": 3400,
|
||||||
"tokens": 7500000
|
"tokens": 7500000
|
||||||
}
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "case-05-change-retention",
|
||||||
|
"name": "A change to a delivered event-driven service: retention goes from 30 days to 7 years",
|
||||||
|
"workflow_request": "Design an event-driven regional HTTP application on Google Cloud that accepts incoming webhooks, validates their signatures, durably enqueues payloads for asynchronous processing, and retains raw events for 30 days for replay and audit.",
|
||||||
|
"change": "One thing has changed since this was delivered. Raw events must now be kept for 7 years instead of 30 days, to meet a new audit obligation, and must not be deletable during that time. Everything else stays as it is.",
|
||||||
|
"required_products": [
|
||||||
|
"Cloud Run",
|
||||||
|
"Pub/Sub",
|
||||||
|
"Cloud Storage"
|
||||||
|
],
|
||||||
|
"scripted_answers": [
|
||||||
|
[
|
||||||
|
"region|location|geograph",
|
||||||
|
"A single region, europe-west2 (London). Data must stay in the UK."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"availability|uptime|sla|rto|rpo|disaster|recover",
|
||||||
|
"99.9% monthly availability. RTO 1 hour, RPO 5 minutes for accepted events. No multi-region DR in phase one."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"traffic|volume|throughput|scale|peak|requests per|rps|payload",
|
||||||
|
"Average 50 webhooks per second, peaks of 500 per second for 10 minutes. Payloads up to 256 KB."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"latency|response time|performance",
|
||||||
|
"Acknowledge each webhook within 300 ms at p95."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"security|compliance|encrypt|privacy|gdpr|pci|access|identity|auth",
|
||||||
|
"UK GDPR applies. Google-managed encryption keys are acceptable. Senders sign requests with HMAC-SHA256 shared secrets; only processing services may read events."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"cost|budget|spend",
|
||||||
|
"Under 1,500 GBP a month at average load; prefer pay-per-use."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"current|existing|today|on-prem|migrat|legacy",
|
||||||
|
"This is a new workload; nothing runs today."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"depend|integrat|downstream|upstream|consum",
|
||||||
|
"Processed events go to an existing internal REST API that is reachable over HTTPS; it can be slow or unavailable for minutes."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"operat|monitor|logging|alert|team|support",
|
||||||
|
"A small platform team of three uses Terraform and wants managed services and Cloud Monitoring alerts."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"sustainab|carbon",
|
||||||
|
"Prefer low-carbon regions when there is no other constraint, but the UK residency requirement wins."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"retention|replay|audit|archive",
|
||||||
|
"Keep raw events 30 days, then delete. Replay must be possible for any event in that window."
|
||||||
|
],
|
||||||
|
[
|
||||||
|
"approve|confirm|proceed|look right|any changes|happy",
|
||||||
|
"Approved. Please proceed."
|
||||||
|
]
|
||||||
|
]
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|||||||
108
eval/scoring.py
108
eval/scoring.py
@@ -77,6 +77,9 @@ class CaseSpec:
|
|||||||
interactive: bool = False
|
interactive: bool = False
|
||||||
# Upper bounds for "revisions", "seconds" and "tokens"; exceeding one is a regression.
|
# Upper bounds for "revisions", "seconds" and "tokens"; exceeding one is a regression.
|
||||||
thresholds: Mapping[str, float] = field(default_factory=dict)
|
thresholds: Mapping[str, float] = field(default_factory=dict)
|
||||||
|
# A change case: the request is run to completion, then this statement of what
|
||||||
|
# has changed is run as a Change request against it (ADR-0006).
|
||||||
|
change: str = ""
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -118,6 +121,7 @@ def load_cases(path: Path) -> list[CaseSpec]:
|
|||||||
),
|
),
|
||||||
interactive=bool(item.get("interactive", False)),
|
interactive=bool(item.get("interactive", False)),
|
||||||
thresholds=dict(item.get("thresholds", {})),
|
thresholds=dict(item.get("thresholds", {})),
|
||||||
|
change=str(item.get("change", "")),
|
||||||
)
|
)
|
||||||
for item in json.loads(path.read_text(encoding="utf-8"))
|
for item in json.loads(path.read_text(encoding="utf-8"))
|
||||||
]
|
]
|
||||||
@@ -435,3 +439,107 @@ async def _quality(judge: Judge, recommendations: str) -> Check:
|
|||||||
mean = sum(scores.values()) / len(QUALITY_RUBRIC)
|
mean = sum(scores.values()) / len(QUALITY_RUBRIC)
|
||||||
detail = ", ".join(f"{key} {scores.get(key, 0):.2f}" for key in QUALITY_RUBRIC)
|
detail = ", ".join(f"{key} {scores.get(key, 0):.2f}" for key in QUALITY_RUBRIC)
|
||||||
return Check(name, mean >= QUALITY_BAR, f"mean {mean:.2f}: {detail}")
|
return Check(name, mean >= QUALITY_BAR, f"mean {mean:.2f}: {detail}")
|
||||||
|
|
||||||
|
|
||||||
|
# --- a Change request stays contained (iteration 7, step 7) ------------------------------
|
||||||
|
|
||||||
|
UNREQUESTED_SHARE = 0.1 # of the Baseline graph's items, changed without being reached
|
||||||
|
|
||||||
|
|
||||||
|
def score_change(
|
||||||
|
interaction: Interaction, baseline_tokens: float | None, tokens: float | None
|
||||||
|
) -> list[Check]:
|
||||||
|
"""Checks on a Change request against its Baseline.
|
||||||
|
|
||||||
|
They read what the Execution recorded: the requirements' own account of what
|
||||||
|
changed, and a comparison by id of the two versions' graphs.
|
||||||
|
"""
|
||||||
|
from app.workflows.changes import ADDED, CHANGED, REMOVED, change_summary
|
||||||
|
|
||||||
|
state = interaction.state
|
||||||
|
baseline = (state.get("baseline") or {}).get("deliverables") or {}
|
||||||
|
deliverables = state.get("deliverables", {})
|
||||||
|
summary = change_summary(baseline, deliverables)
|
||||||
|
requirements = summary["requirements"]
|
||||||
|
checks = [
|
||||||
|
Check(
|
||||||
|
"change_is_recorded",
|
||||||
|
bool(baseline) and bool(requirements),
|
||||||
|
", ".join(f"{item['id']} {item['change']}" for item in requirements)
|
||||||
|
or "no requirement is marked added, changed or removed",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
graph = _graph_data(deliverables)
|
||||||
|
touched = set(summary["checks"][ADDED]) | set(summary["checks"][CHANGED])
|
||||||
|
removed_checks = set(summary["checks"][REMOVED])
|
||||||
|
old_sources = {
|
||||||
|
item.get("id"): item.get("source") for item in _graph_data(baseline)["checks"]
|
||||||
|
}
|
||||||
|
unreached = []
|
||||||
|
for item in requirements:
|
||||||
|
identifier = item["id"]
|
||||||
|
if item["change"] == REMOVED:
|
||||||
|
# What it asked for should go: a Check that came from it is removed.
|
||||||
|
reaches = any(old_sources.get(c) == identifier for c in removed_checks)
|
||||||
|
reaches = reaches or identifier not in old_sources.values()
|
||||||
|
else:
|
||||||
|
reaches = any(
|
||||||
|
check.get("id") in touched and check.get("source") == identifier
|
||||||
|
for check in graph["checks"]
|
||||||
|
)
|
||||||
|
if not reaches:
|
||||||
|
unreached.append(identifier)
|
||||||
|
checks.append(
|
||||||
|
Check(
|
||||||
|
"changed_requirements_reach_checks",
|
||||||
|
bool(requirements) and not unreached,
|
||||||
|
"no Check changed for: " + ", ".join(unreached) if unreached else "",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
unrequested = [
|
||||||
|
f"{part} {identifier}"
|
||||||
|
for part, ids in summary["unrequested"].items()
|
||||||
|
for identifier in ids
|
||||||
|
]
|
||||||
|
size = sum(len(items) for items in _graph_data(baseline).values()) or 1
|
||||||
|
checks.append(
|
||||||
|
Check(
|
||||||
|
"change_is_contained",
|
||||||
|
len(unrequested) <= max(2, UNREQUESTED_SHARE * size),
|
||||||
|
f"{len(unrequested)} of {size} items changed without being reached"
|
||||||
|
+ (": " + ", ".join(unrequested[:12]) if unrequested else ""),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
cheaper = (
|
||||||
|
baseline_tokens is not None and tokens is not None and tokens < baseline_tokens
|
||||||
|
)
|
||||||
|
checks.append(
|
||||||
|
Check(
|
||||||
|
"change_costs_less_than_baseline",
|
||||||
|
cheaper,
|
||||||
|
f"{tokens or 0:,.0f} tokens against {baseline_tokens or 0:,.0f}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return checks
|
||||||
|
|
||||||
|
|
||||||
|
def _graph_data(
|
||||||
|
deliverables: Mapping[str, Mapping[str, Any]],
|
||||||
|
) -> dict[str, list[dict[str, Any]]]:
|
||||||
|
text = (
|
||||||
|
(deliverables.get(DeliverableKind.TARGET_ARCHITECTURE.value) or {})
|
||||||
|
.get("attachments", {})
|
||||||
|
.get("target-architecture.json", "")
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
data = json.loads(text)
|
||||||
|
except ValueError:
|
||||||
|
data = {}
|
||||||
|
if not isinstance(data, dict):
|
||||||
|
data = {}
|
||||||
|
return {
|
||||||
|
part: [item for item in data.get(part, []) if isinstance(item, dict)]
|
||||||
|
for part in ("elements", "relationships", "checks")
|
||||||
|
}
|
||||||
|
|||||||
@@ -41,9 +41,11 @@ from eval.judge import DeciderJudge, GeminiJudge, Judge
|
|||||||
from eval.scoring import (
|
from eval.scoring import (
|
||||||
CaseResult,
|
CaseResult,
|
||||||
CaseSpec,
|
CaseSpec,
|
||||||
|
Check,
|
||||||
checklist_items,
|
checklist_items,
|
||||||
load_cases,
|
load_cases,
|
||||||
regressions,
|
regressions,
|
||||||
|
score_change,
|
||||||
score_execution,
|
score_execution,
|
||||||
template_headings,
|
template_headings,
|
||||||
)
|
)
|
||||||
@@ -179,7 +181,33 @@ async def run_case(
|
|||||||
settings, runtime=runtime, recorder=recorder, output_root=run_dir
|
settings, runtime=runtime, recorder=recorder, output_root=run_dir
|
||||||
) as assembly:
|
) as assembly:
|
||||||
execution_id = f"{case.id}-{datetime.now(UTC):%H%M%S}"
|
execution_id = f"{case.id}-{datetime.now(UTC):%H%M%S}"
|
||||||
if case.interactive:
|
baseline_tokens: float | None = None
|
||||||
|
baseline_seconds: float | None = None
|
||||||
|
if case.change:
|
||||||
|
# A change case: deliver the request, then run the change against it.
|
||||||
|
base = await assembly.engine.start(
|
||||||
|
case.workflow_request,
|
||||||
|
customer_case=case.id,
|
||||||
|
pre_approved=True,
|
||||||
|
allow_cloud_read=False,
|
||||||
|
execution_id=f"{execution_id}-base",
|
||||||
|
scripted_answers=case.scripted_answers,
|
||||||
|
)
|
||||||
|
baseline_tokens = token_report(adk_runtime.usage)["total"].get("total")
|
||||||
|
baseline_seconds = round(time.monotonic() - started, 1)
|
||||||
|
if base.status != "completed":
|
||||||
|
interaction = base # nothing to change: score what there is
|
||||||
|
else:
|
||||||
|
interaction = await assembly.engine.start(
|
||||||
|
case.change,
|
||||||
|
customer_case=case.id,
|
||||||
|
pre_approved=True,
|
||||||
|
allow_cloud_read=False,
|
||||||
|
execution_id=execution_id,
|
||||||
|
scripted_answers=case.scripted_answers,
|
||||||
|
baseline=base.execution_id,
|
||||||
|
)
|
||||||
|
elif case.interactive:
|
||||||
interaction = await run_with_scripted_user(
|
interaction = await run_with_scripted_user(
|
||||||
assembly.engine, case, execution_id
|
assembly.engine, case, execution_id
|
||||||
)
|
)
|
||||||
@@ -220,6 +248,15 @@ async def run_case(
|
|||||||
)
|
)
|
||||||
ledger = interaction.state.get("ledger", {})
|
ledger = interaction.state.get("ledger", {})
|
||||||
tokens = token_report(adk_runtime.usage)
|
tokens = token_report(adk_runtime.usage)
|
||||||
|
change_checks: tuple[Check, ...] = ()
|
||||||
|
if case.change:
|
||||||
|
total = tokens["total"].get("total")
|
||||||
|
change_tokens = (
|
||||||
|
total - baseline_tokens
|
||||||
|
if total is not None and baseline_tokens is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
change_checks = tuple(score_change(interaction, baseline_tokens, change_tokens))
|
||||||
revisions = {
|
revisions = {
|
||||||
kind: item["revisions"]
|
kind: item["revisions"]
|
||||||
for kind, item in ledger.get("deliverables", {}).items()
|
for kind, item in ledger.get("deliverables", {}).items()
|
||||||
@@ -250,7 +287,17 @@ async def run_case(
|
|||||||
"conversation": interaction.state.get("conversation", []),
|
"conversation": interaction.state.get("conversation", []),
|
||||||
"output_dir": str(execution_dir(run_dir, interaction.execution_id, case.id)),
|
"output_dir": str(execution_dir(run_dir, interaction.execution_id, case.id)),
|
||||||
}
|
}
|
||||||
return CaseResult(result.case_id, result.status, result.checks, details)
|
if case.change:
|
||||||
|
details["baseline"] = {
|
||||||
|
"execution_id": (interaction.state.get("baseline") or {}).get(
|
||||||
|
"execution_id"
|
||||||
|
),
|
||||||
|
"seconds": baseline_seconds,
|
||||||
|
"tokens": baseline_tokens,
|
||||||
|
}
|
||||||
|
return CaseResult(
|
||||||
|
result.case_id, result.status, (*result.checks, *change_checks), details
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def make_judge(settings: Settings) -> Judge:
|
def make_judge(settings: Settings) -> Judge:
|
||||||
|
|||||||
@@ -387,3 +387,66 @@ def test_a_pre_approved_task_is_not_read_as_a_question(tmp_path):
|
|||||||
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||||
assert decider.asked == []
|
assert decider.asked == []
|
||||||
assert runtime.runs
|
assert runtime.runs
|
||||||
|
|
||||||
|
|
||||||
|
def delivered_case(tmp_path: Path, case: str = "acme") -> None:
|
||||||
|
folder = tmp_path / "deliverables" / "cases" / case / "intent"
|
||||||
|
folder.mkdir(parents=True)
|
||||||
|
(folder / "history.json").write_text(
|
||||||
|
'{"customer_case": "acme", "intents": [{"version": 1, "workflow_request": '
|
||||||
|
'"Design ingestion.", "execution_id": "v1", "status": "completed"}]}'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_to_delivered_work_names_the_baseline_and_starts_nothing(tmp_path):
|
||||||
|
delivered_case(tmp_path)
|
||||||
|
decider = FakeDecider({"intake": Answer("change_request", 0.98)})
|
||||||
|
client, runtime = client_for(tmp_path, {}, decider)
|
||||||
|
with client:
|
||||||
|
task = send(
|
||||||
|
client,
|
||||||
|
[{"text": "Availability must now be 99.99%."}],
|
||||||
|
metadata={"customer_case": "acme"},
|
||||||
|
)
|
||||||
|
|
||||||
|
text, data = status_parts(task)
|
||||||
|
assert task["status"]["state"] == "TASK_STATE_COMPLETED"
|
||||||
|
assert 'send this again with "baseline" set to "v1"' in text
|
||||||
|
assert data == {"status": "answered", "intake": "change_request", "baseline": "v1"}
|
||||||
|
assert runtime.runs == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_caller_who_says_it_is_new_work_is_not_second_guessed(tmp_path):
|
||||||
|
delivered_case(tmp_path)
|
||||||
|
decider = FakeDecider({"intake": Answer("change_request", 0.98)})
|
||||||
|
turns = {"orchestrator": [[Call("ask_user", {"questions": ["Which region?"]})]]}
|
||||||
|
client, runtime = client_for(tmp_path, turns, decider)
|
||||||
|
with client:
|
||||||
|
task = send(
|
||||||
|
client,
|
||||||
|
[{"text": "Availability must now be 99.99%."}],
|
||||||
|
metadata={"customer_case": "acme", "new_request": True},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert task["status"]["state"] == "TASK_STATE_INPUT_REQUIRED"
|
||||||
|
assert decider.asked == [] and len(runtime.runs) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_baseline_that_cannot_be_one_fails_the_task_and_generate(tmp_path):
|
||||||
|
client, runtime = client_for(tmp_path, {})
|
||||||
|
with client:
|
||||||
|
task = send(
|
||||||
|
client,
|
||||||
|
[{"text": "A change."}],
|
||||||
|
metadata={"customer_case": "acme", "baseline": "nope"},
|
||||||
|
)
|
||||||
|
response = client.post(
|
||||||
|
"/generate",
|
||||||
|
json={"request": "A change.", "customer_case": "acme", "baseline": "nope"},
|
||||||
|
)
|
||||||
|
|
||||||
|
assert task["status"]["state"] == "TASK_STATE_FAILED"
|
||||||
|
assert "is not an Execution of Customer case acme" in status_parts(task)[0]
|
||||||
|
assert response.status_code == 400
|
||||||
|
assert "is not an Execution of Customer case acme" in response.json()["error"]
|
||||||
|
assert runtime.runs == []
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
"""Scoring a finished Execution against a benchmark case, without models or network."""
|
"""Scoring a finished Execution against a benchmark case, without models or network."""
|
||||||
|
|
||||||
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from app.adk.fake_runtime import SAMPLE_DATA
|
from app.adk.fake_runtime import SAMPLE_DATA
|
||||||
@@ -614,3 +615,133 @@ def test_calibration_without_labels_still_measures_stability():
|
|||||||
assert result["labelled"] == 0
|
assert result["labelled"] == 0
|
||||||
assert result["judges"]["jev"]["agreement_with_labels"] is None
|
assert result["judges"]["jev"]["agreement_with_labels"] is None
|
||||||
assert result["judges"]["jev"]["stability"] == 1.0
|
assert result["judges"]["jev"]["stability"] == 1.0
|
||||||
|
|
||||||
|
|
||||||
|
# --- a Change request stays contained ---------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def change_interaction(graph_change=None, requirement=None) -> Interaction:
|
||||||
|
from app.adk.fake_runtime import SAMPLE_DATA
|
||||||
|
|
||||||
|
names = {
|
||||||
|
"requirements": "requirements.json",
|
||||||
|
"technical_decomposition": "components.json",
|
||||||
|
"design_recommendations": "recommendations.json",
|
||||||
|
"target_architecture": "target-architecture.json",
|
||||||
|
}
|
||||||
|
|
||||||
|
def version() -> dict:
|
||||||
|
return {
|
||||||
|
kind: {"attachments": {name: SAMPLE_DATA[name]}}
|
||||||
|
for kind, name in names.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
baseline, current = version(), version()
|
||||||
|
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||||
|
for item in requirements:
|
||||||
|
item["change"] = "unchanged"
|
||||||
|
if requirement is not False:
|
||||||
|
requirements[1].update(
|
||||||
|
requirement
|
||||||
|
or {"change": "changed", "text": "99.99%.", "reason": "Customer-facing."}
|
||||||
|
)
|
||||||
|
current["requirements"]["attachments"] = {
|
||||||
|
"requirements.json": json.dumps(requirements)
|
||||||
|
}
|
||||||
|
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||||
|
if graph_change:
|
||||||
|
graph_change(graph)
|
||||||
|
current["target_architecture"]["attachments"] = {
|
||||||
|
"target-architecture.json": json.dumps(graph)
|
||||||
|
}
|
||||||
|
state = {"baseline": {"deliverables": baseline}, "deliverables": current}
|
||||||
|
return Interaction("v2", "completed", None, state)
|
||||||
|
|
||||||
|
|
||||||
|
def results(checks) -> dict[str, bool]:
|
||||||
|
return {check.name: check.passed for check in checks}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_contained_change_passes_every_change_check():
|
||||||
|
from eval.scoring import score_change
|
||||||
|
|
||||||
|
def tighten(graph):
|
||||||
|
graph["checks"][0]["statement"] = "The service runs at least three instances."
|
||||||
|
|
||||||
|
checks = score_change(change_interaction(tighten), 4_000_000, 1_500_000)
|
||||||
|
|
||||||
|
assert results(checks) == {
|
||||||
|
"change_is_recorded": True,
|
||||||
|
"changed_requirements_reach_checks": True,
|
||||||
|
"change_is_contained": True,
|
||||||
|
"change_costs_less_than_baseline": True,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_changed_requirement_that_changes_no_check_fails():
|
||||||
|
from eval.scoring import score_change
|
||||||
|
|
||||||
|
checks = score_change(change_interaction(), 4_000_000, 1_500_000)
|
||||||
|
|
||||||
|
by_name = {check.name: check for check in checks}
|
||||||
|
assert not by_name["changed_requirements_reach_checks"].passed
|
||||||
|
assert by_name["changed_requirements_reach_checks"].detail == (
|
||||||
|
"no Check changed for: NFR-1"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_that_cost_more_than_the_baseline_or_changed_nothing_fails():
|
||||||
|
from eval.scoring import score_change
|
||||||
|
|
||||||
|
dearer = results(score_change(change_interaction(), 2_000_000, 3_000_000))
|
||||||
|
nothing = results(
|
||||||
|
score_change(change_interaction(requirement=False), 2_000_000, 1_000_000)
|
||||||
|
)
|
||||||
|
|
||||||
|
assert not dearer["change_costs_less_than_baseline"]
|
||||||
|
assert not nothing["change_is_recorded"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_many_changes_nobody_asked_for_fail_containment():
|
||||||
|
from eval.scoring import score_change
|
||||||
|
|
||||||
|
def rewrite(graph):
|
||||||
|
# Everything about the identity changes; NFR-1 reaches none of it.
|
||||||
|
graph["elements"][1]["name"] = "A different identity"
|
||||||
|
graph["checks"][1]["statement"] = "Something else."
|
||||||
|
graph["elements"].append(
|
||||||
|
{
|
||||||
|
"id": "E-9",
|
||||||
|
"name": "An extra key",
|
||||||
|
"product": "Example KMS",
|
||||||
|
"resource_type": "example_key",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
graph["checks"].append(
|
||||||
|
{"id": "K-9", "applies_to": "E-9", "statement": "x", "source": "DR-1"}
|
||||||
|
)
|
||||||
|
|
||||||
|
components = [{"id": "C-1", "name": "Ingress", "requirements": ["FR-1"]}]
|
||||||
|
interaction = change_interaction(rewrite)
|
||||||
|
for version in (
|
||||||
|
interaction.state["baseline"]["deliverables"],
|
||||||
|
interaction.state["deliverables"],
|
||||||
|
):
|
||||||
|
version["technical_decomposition"]["attachments"] = {
|
||||||
|
"components.json": json.dumps(components)
|
||||||
|
}
|
||||||
|
|
||||||
|
by_name = {check.name: check for check in score_change(interaction, 4, 1)}
|
||||||
|
|
||||||
|
assert not by_name["change_is_contained"].passed
|
||||||
|
assert by_name["change_is_contained"].detail.startswith("4 of 5 items changed")
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_change_case_loads_with_its_statement_of_change():
|
||||||
|
from eval.scoring import load_cases
|
||||||
|
|
||||||
|
cases = load_cases(Path(__file__).parents[2] / "eval/datasets/solution_cases.json")
|
||||||
|
case = next(c for c in cases if c.id == "case-05-change-retention")
|
||||||
|
|
||||||
|
assert "7 years" in case.change
|
||||||
|
assert all(not c.change for c in cases if c.id != case.id)
|
||||||
|
|||||||
633
tests/execution/test_change_requests.py
Normal file
633
tests/execution/test_change_requests.py
Normal file
@@ -0,0 +1,633 @@
|
|||||||
|
"""Change requests (ADR-0006): an Execution that starts from a Baseline.
|
||||||
|
|
||||||
|
A Baseline is a completed Execution of the same Customer case. A Change request is a
|
||||||
|
new Execution that begins with the Baseline's Deliverables; the Baseline itself is
|
||||||
|
never modified.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from langgraph.checkpoint.memory import InMemorySaver
|
||||||
|
|
||||||
|
from app.adk.fake_runtime import (
|
||||||
|
SAMPLE_DATA,
|
||||||
|
Call,
|
||||||
|
FakeAgentRuntime,
|
||||||
|
sample_submission,
|
||||||
|
)
|
||||||
|
from app.adk.runtime import NoDecider, RoleSpec
|
||||||
|
from app.agent import ExecutionEngine
|
||||||
|
from app.baseline import BaselineError, load_baseline
|
||||||
|
from app.nodes.router import Router, Thresholds
|
||||||
|
from app.recorder import InMemoryRecorder
|
||||||
|
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||||
|
|
||||||
|
K = DeliverableKind
|
||||||
|
CASE = "acme-webhooks"
|
||||||
|
ROLES = {
|
||||||
|
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||||
|
**{
|
||||||
|
spec.specialist: RoleSpec(spec.specialist, "Produce.", budget=15)
|
||||||
|
for spec in DEPENDENCY_GRAPH.values()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def once() -> dict:
|
||||||
|
return {
|
||||||
|
spec.specialist: [[sample_submission(kind)]]
|
||||||
|
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def revised(**changes) -> Call:
|
||||||
|
"""The requirements of a Change request: the Baseline's, each marked, plus what
|
||||||
|
`changes` adds or alters by id."""
|
||||||
|
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||||
|
for item in requirements:
|
||||||
|
item["change"] = "unchanged"
|
||||||
|
if item["id"] in changes:
|
||||||
|
item.update(changes.pop(item["id"]))
|
||||||
|
requirements += [{"id": key, **value} for key, value in changes.items()]
|
||||||
|
call = sample_submission(K.REQUIREMENTS)
|
||||||
|
return Call(
|
||||||
|
call.tool,
|
||||||
|
{
|
||||||
|
**call.args,
|
||||||
|
"attachments": {"requirements.json": json.dumps(requirements)},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
RETENTION = {
|
||||||
|
"NFR-1": {
|
||||||
|
"change": "changed",
|
||||||
|
"text": "99.99% availability.",
|
||||||
|
"reason": "The service is now customer-facing.",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def changed(**overrides) -> dict:
|
||||||
|
"""Scripted turns for a Change request that changes NFR-1 and nothing else."""
|
||||||
|
turns = once()
|
||||||
|
turns["requirements-analyst"] = [[revised(**json.loads(json.dumps(RETENTION)))]]
|
||||||
|
turns.update(overrides)
|
||||||
|
return turns
|
||||||
|
|
||||||
|
|
||||||
|
def engine_for(tmp_path: Path, turns: dict, **options) -> ExecutionEngine:
|
||||||
|
return ExecutionEngine(
|
||||||
|
runtime=FakeAgentRuntime(turns),
|
||||||
|
roles=ROLES,
|
||||||
|
recorder=InMemoryRecorder(),
|
||||||
|
checkpointer=InMemorySaver(),
|
||||||
|
output_root=tmp_path,
|
||||||
|
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||||
|
**options,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def delivered(tmp_path: Path, case: str = CASE, execution_id: str = "v1"):
|
||||||
|
"""A completed Execution on disk, to be a Baseline."""
|
||||||
|
result = await engine_for(tmp_path, once()).start(
|
||||||
|
"Ingest webhooks",
|
||||||
|
customer_case=case,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id=execution_id,
|
||||||
|
)
|
||||||
|
assert result.status == "completed"
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def digest(folder: Path) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
str(path.relative_to(folder)): hashlib.sha256(path.read_bytes()).hexdigest()
|
||||||
|
for path in sorted(folder.rglob("*"))
|
||||||
|
if path.is_file()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def history(tmp_path: Path, case: str = CASE) -> list[dict]:
|
||||||
|
path = tmp_path / "deliverables" / "cases" / case / "intent" / "history.json"
|
||||||
|
return json.loads(path.read_text())["intents"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- step 2: start against a Baseline --------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_baseline_is_read_from_its_files(tmp_path):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
|
||||||
|
baseline = load_baseline(tmp_path, CASE, "v1")
|
||||||
|
|
||||||
|
assert baseline["execution_id"] == "v1"
|
||||||
|
assert baseline["workflow_request"] == "Ingest webhooks"
|
||||||
|
assert set(baseline["deliverables"]) == {kind.value for kind in DEPENDENCY_GRAPH}
|
||||||
|
graph = baseline["deliverables"]["target_architecture"]["attachments"]
|
||||||
|
assert "target-architecture.json" in graph
|
||||||
|
assert baseline["deliverables"]["requirements"]["content"].startswith("#")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_change_request_starts_with_its_baseline_and_is_recorded(tmp_path):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
|
||||||
|
result = await engine_for(tmp_path, changed()).start(
|
||||||
|
"Retention goes from 30 days to 7 years.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result.status == "completed"
|
||||||
|
assert result.state["baseline"]["execution_id"] == "v1"
|
||||||
|
first, second = history(tmp_path)
|
||||||
|
assert first["baseline"] is None
|
||||||
|
assert second["baseline"] == "v1" and second["execution_id"] == "v2"
|
||||||
|
case = tmp_path / "deliverables" / "cases" / CASE
|
||||||
|
manifest = json.loads((case / "executions" / "v2" / "execution.json").read_text())
|
||||||
|
assert manifest["baseline"] == "v1"
|
||||||
|
assert "- Changes: `v1`" in (case / "intent" / "002.md").read_text()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_baseline_is_never_modified(tmp_path):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
folder = tmp_path / "deliverables" / "cases" / CASE / "executions" / "v1"
|
||||||
|
before = digest(folder)
|
||||||
|
|
||||||
|
await engine_for(tmp_path, changed()).start(
|
||||||
|
"Add a second region.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert digest(folder) == before
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_change_request_may_itself_be_a_baseline(tmp_path):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
for version, previous in (("v2", "v1"), ("v3", "v2")):
|
||||||
|
await engine_for(tmp_path, changed()).start(
|
||||||
|
"Another change.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id=version,
|
||||||
|
baseline=previous,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [item["baseline"] for item in history(tmp_path)] == [None, "v1", "v2"]
|
||||||
|
assert load_baseline(tmp_path, CASE, "v3")["baseline"] == "v2"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_without_a_baseline_an_execution_starts_clean(tmp_path):
|
||||||
|
result = await delivered(tmp_path)
|
||||||
|
|
||||||
|
assert result.state["baseline"] is None
|
||||||
|
manifest = json.loads(
|
||||||
|
(
|
||||||
|
tmp_path / "deliverables/cases" / CASE / "executions/v1/execution.json"
|
||||||
|
).read_text()
|
||||||
|
)
|
||||||
|
assert manifest["baseline"] is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"case, baseline, message",
|
||||||
|
[
|
||||||
|
(CASE, "nope", "is not an Execution of Customer case acme-webhooks"),
|
||||||
|
("other-client", "v1", "belongs to another Customer case"),
|
||||||
|
(None, "v1", "needs a customer_case"),
|
||||||
|
(CASE, "../v1", "is not an Execution id"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
async def test_a_baseline_that_cannot_be_one_refuses_the_start(
|
||||||
|
tmp_path, case, baseline, message
|
||||||
|
):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
engine = engine_for(tmp_path, once())
|
||||||
|
|
||||||
|
with pytest.raises(BaselineError, match=message):
|
||||||
|
await engine.start(
|
||||||
|
"A change.", customer_case=case, execution_id="v2", baseline=baseline
|
||||||
|
)
|
||||||
|
|
||||||
|
# Refused before anything ran or was recorded.
|
||||||
|
assert [item["execution_id"] for item in history(tmp_path)] == ["v1"]
|
||||||
|
assert not (tmp_path / "deliverables/cases/other-client").exists()
|
||||||
|
|
||||||
|
|
||||||
|
async def test_an_execution_that_did_not_complete_cannot_be_a_baseline(tmp_path):
|
||||||
|
from tests.execution.test_router import failing
|
||||||
|
|
||||||
|
turns = once()
|
||||||
|
turns["validator"] = [[failing("K-1")]]
|
||||||
|
engine = engine_for(tmp_path, turns, max_revisions=0)
|
||||||
|
unfinished = await engine.start(
|
||||||
|
"Ingest webhooks", customer_case=CASE, pre_approved=True, execution_id="v1"
|
||||||
|
)
|
||||||
|
assert unfinished.status == "completed_with_warnings"
|
||||||
|
|
||||||
|
with pytest.raises(BaselineError, match="did not complete"):
|
||||||
|
load_baseline(tmp_path, CASE, "v1")
|
||||||
|
|
||||||
|
|
||||||
|
# --- step 3: the requirements say what changed -------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def requirements_check(revised_call: Call, baseline=None) -> tuple[str, ...]:
|
||||||
|
from app.workflows.deliverables import check_submission
|
||||||
|
|
||||||
|
baseline = baseline or {
|
||||||
|
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]}
|
||||||
|
}
|
||||||
|
return check_submission(
|
||||||
|
K.REQUIREMENTS,
|
||||||
|
revised_call.args["content"],
|
||||||
|
revised_call.args["attachments"],
|
||||||
|
baseline=baseline,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_revised_requirement_list_that_accounts_for_every_change_is_accepted():
|
||||||
|
call = revised(
|
||||||
|
**{
|
||||||
|
"NFR-1": RETENTION["NFR-1"],
|
||||||
|
"NFR-2": {
|
||||||
|
"kind": "non_functional",
|
||||||
|
"text": "A second region.",
|
||||||
|
"conditions": ["The service runs in two regions."],
|
||||||
|
"change": "added",
|
||||||
|
"reason": "Customers are now in two continents.",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert requirements_check(call) == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_requirement_in_a_change_request_says_whether_it_changed():
|
||||||
|
call = sample_submission(K.REQUIREMENTS) # no `change` on anything
|
||||||
|
|
||||||
|
missing = requirements_check(call)
|
||||||
|
|
||||||
|
assert "requirements.json: FR-1 needs a `change`" in missing[0]
|
||||||
|
assert len(missing) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_needs_its_reason_and_an_unchanged_requirement_its_text():
|
||||||
|
no_reason = revised(**{"NFR-1": {"change": "changed", "text": "99.99%."}})
|
||||||
|
quietly = revised(**{"NFR-1": {"text": "99.99% availability."}})
|
||||||
|
quietly_other = revised(
|
||||||
|
**{"NFR-1": RETENTION["NFR-1"], "FR-1": {"text": "Accept events and files."}}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert requirements_check(no_reason) == (
|
||||||
|
"requirements.json: NFR-1 is changed and needs a `reason`",
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
"NFR-1 is marked unchanged, but its text differs"
|
||||||
|
in (requirements_check(quietly)[0])
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
"FR-1 is marked unchanged, but its text differs"
|
||||||
|
in (requirements_check(quietly_other)[0])
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_baseline_requirement_cannot_silently_disappear_or_be_reused():
|
||||||
|
call = revised(**{"NFR-1": RETENTION["NFR-1"]})
|
||||||
|
requirements = json.loads(call.args["attachments"]["requirements.json"])
|
||||||
|
dropped = Call(
|
||||||
|
call.tool,
|
||||||
|
{
|
||||||
|
**call.args,
|
||||||
|
"attachments": {"requirements.json": json.dumps(requirements[1:])},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
reused = revised(
|
||||||
|
**{
|
||||||
|
"NFR-1": {
|
||||||
|
"change": "added",
|
||||||
|
"text": "Something else entirely.",
|
||||||
|
"reason": "New.",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
)
|
||||||
|
invented = revised(
|
||||||
|
**{
|
||||||
|
"NFR-1": RETENTION["NFR-1"],
|
||||||
|
"NFR-9": {
|
||||||
|
"kind": "non_functional",
|
||||||
|
"text": "x",
|
||||||
|
"conditions": ["x"],
|
||||||
|
"change": "changed",
|
||||||
|
"reason": "y",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "Baseline requirement FR-1 is missing" in requirements_check(dropped)[0]
|
||||||
|
assert (
|
||||||
|
"NFR-1 is marked added, but the Baseline has that id"
|
||||||
|
in (requirements_check(reused)[0])
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
"NFR-9 is marked changed, but the Baseline has no such requirement"
|
||||||
|
in (requirements_check(invented)[0])
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_request_must_change_something():
|
||||||
|
assert (
|
||||||
|
"nothing is marked added, changed or removed"
|
||||||
|
in (requirements_check(revised())[0])
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_removed_requirement_stays_on_record_and_asks_nothing_downstream():
|
||||||
|
from app.workflows.chain import read_list
|
||||||
|
|
||||||
|
call = revised(
|
||||||
|
**{"NFR-1": {"change": "removed", "reason": "No longer a requirement."}}
|
||||||
|
)
|
||||||
|
upstream = {"requirements": {"attachments": call.args["attachments"]}}
|
||||||
|
|
||||||
|
assert requirements_check(call) == ()
|
||||||
|
live = read_list(upstream, "requirements", "requirements.json")
|
||||||
|
assert [item["id"] for item in live] == ["FR-1"]
|
||||||
|
assert '"NFR-1"' in call.args["attachments"]["requirements.json"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- step 4: later Specialists revise from the Baseline ---------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_each_specialist_is_given_its_delivered_version_and_what_changed(
|
||||||
|
tmp_path,
|
||||||
|
):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
runtime = FakeAgentRuntime(changed())
|
||||||
|
engine = ExecutionEngine(
|
||||||
|
runtime=runtime,
|
||||||
|
roles=ROLES,
|
||||||
|
recorder=InMemoryRecorder(),
|
||||||
|
checkpointer=InMemorySaver(),
|
||||||
|
output_root=tmp_path,
|
||||||
|
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||||
|
)
|
||||||
|
|
||||||
|
await engine.start(
|
||||||
|
"Availability goes up.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
by_role = {run.role: run for run in runtime.runs}
|
||||||
|
analyst = by_role["requirements-analyst"].task.context["baseline"]
|
||||||
|
assert analyst["execution_id"] == "v1"
|
||||||
|
assert analyst["original_request"] == "Ingest webhooks"
|
||||||
|
assert "requirements.json" in analyst["deliverable"]["attachments"]
|
||||||
|
assert analyst["changes"] is None # nothing has changed yet
|
||||||
|
|
||||||
|
architect = by_role["target-architect"].task.context
|
||||||
|
assert architect["baseline"]["changes"]["requirements"] == [
|
||||||
|
{
|
||||||
|
"id": "NFR-1",
|
||||||
|
"change": "changed",
|
||||||
|
"reason": "The service is now customer-facing.",
|
||||||
|
"text": "99.99% availability.",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
# Its working copy already holds the delivered graph.
|
||||||
|
assert architect["baseline"]["deliverable"]["attachments"][
|
||||||
|
"target-architecture.json"
|
||||||
|
]
|
||||||
|
assert "graph_read" in by_role["target-architect"].tools
|
||||||
|
# The guide is assembled afresh: the Baseline's is not handed back.
|
||||||
|
assert by_role["packager"].task.context["baseline"]["deliverable"] is None
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_description_whose_design_did_not_change_shape_is_carried_over(
|
||||||
|
tmp_path,
|
||||||
|
):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
runtime = FakeAgentRuntime(changed())
|
||||||
|
engine = ExecutionEngine(
|
||||||
|
runtime=runtime,
|
||||||
|
roles=ROLES,
|
||||||
|
recorder=InMemoryRecorder(),
|
||||||
|
checkpointer=InMemorySaver(),
|
||||||
|
output_root=tmp_path,
|
||||||
|
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await engine.start(
|
||||||
|
"Availability goes up.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
# The same Elements and Relationships as the Baseline: the describer is not run.
|
||||||
|
assert "describer" not in {run.role for run in runtime.runs}
|
||||||
|
assert result.state["deliverables"]["architecture_description"]["content"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_code_authors_working_copy_starts_as_the_delivered_code(tmp_path):
|
||||||
|
await delivered(tmp_path)
|
||||||
|
seen: dict = {}
|
||||||
|
|
||||||
|
def author(task):
|
||||||
|
seen["working_copy"] = task.context["working_copy"]
|
||||||
|
return [sample_submission(K.DEPLOYMENT_GUIDANCE)]
|
||||||
|
|
||||||
|
engine = engine_for(tmp_path, changed(**{"iac-author": [author]}))
|
||||||
|
|
||||||
|
await engine.start(
|
||||||
|
"Availability goes up.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
# It is told nothing of a working copy on a first attempt; the tools hold it.
|
||||||
|
assert seen["working_copy"] is None
|
||||||
|
|
||||||
|
|
||||||
|
# --- what changed, worked out from the ids ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def versions(graph_change=None, requirement_change=None) -> tuple[dict, dict]:
|
||||||
|
"""A Baseline and a revised set of Deliverables, as data files."""
|
||||||
|
|
||||||
|
def deliverables() -> dict:
|
||||||
|
return {
|
||||||
|
kind: {"attachments": {name: SAMPLE_DATA[name]}}
|
||||||
|
for kind, name in (
|
||||||
|
("requirements", "requirements.json"),
|
||||||
|
("technical_decomposition", "components.json"),
|
||||||
|
("design_recommendations", "recommendations.json"),
|
||||||
|
("target_architecture", "target-architecture.json"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
before, after = deliverables(), deliverables()
|
||||||
|
change = requirement_change or RETENTION
|
||||||
|
after["requirements"]["attachments"] = revised(
|
||||||
|
**json.loads(json.dumps(change))
|
||||||
|
).args["attachments"]
|
||||||
|
if graph_change:
|
||||||
|
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||||
|
graph_change(graph)
|
||||||
|
after["target_architecture"]["attachments"] = {
|
||||||
|
"target-architecture.json": json.dumps(graph)
|
||||||
|
}
|
||||||
|
return before, after
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_inside_the_reach_of_a_changed_requirement_is_requested():
|
||||||
|
from app.workflows.changes import change_summary
|
||||||
|
|
||||||
|
def tighten(graph):
|
||||||
|
graph["checks"][0]["statement"] = "The service runs at least three instances."
|
||||||
|
|
||||||
|
summary = change_summary(*versions(tighten))
|
||||||
|
|
||||||
|
assert [item["id"] for item in summary["requirements"]] == ["NFR-1"]
|
||||||
|
assert summary["checks"] == {"added": [], "changed": ["K-1"], "removed": []}
|
||||||
|
assert summary["elements"] == {"added": [], "changed": [], "removed": []}
|
||||||
|
assert summary["unrequested"] == {}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_change_no_changed_requirement_reaches_is_listed_as_unrequested():
|
||||||
|
from app.workflows.changes import change_summary
|
||||||
|
|
||||||
|
def wander(graph):
|
||||||
|
# K-2 comes from DR-1, which did not change, on E-2, which realises nothing
|
||||||
|
# the changed requirement touches. NFR-1 reaches K-1 and E-1 only.
|
||||||
|
graph["checks"][1]["statement"] = "The identity has no keys."
|
||||||
|
|
||||||
|
change = {
|
||||||
|
"NFR-1": {"change": "changed", "text": "99.99%.", "reason": "Customer-facing."}
|
||||||
|
}
|
||||||
|
before, after = versions(wander, change)
|
||||||
|
# NFR-1 is served by C-1, which E-1 realises; cut that link to isolate E-2.
|
||||||
|
components = [{"id": "C-1", "name": "Ingress", "requirements": ["FR-1"]}]
|
||||||
|
for version in (before, after):
|
||||||
|
version["technical_decomposition"]["attachments"] = {
|
||||||
|
"components.json": json.dumps(components)
|
||||||
|
}
|
||||||
|
|
||||||
|
summary = change_summary(before, after)
|
||||||
|
|
||||||
|
assert summary["checks"]["changed"] == ["K-2"]
|
||||||
|
assert summary["unrequested"] == {"checks": ["K-2"]}
|
||||||
|
|
||||||
|
|
||||||
|
def test_added_and_removed_items_are_found_by_id():
|
||||||
|
from app.workflows.changes import change_summary
|
||||||
|
|
||||||
|
def grow(graph):
|
||||||
|
graph["elements"].append(
|
||||||
|
{
|
||||||
|
"id": "E-3",
|
||||||
|
"name": "Second region service",
|
||||||
|
"component": "C-1",
|
||||||
|
"product": "Example Run",
|
||||||
|
"resource_type": "example_run_service",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
graph["checks"] = [graph["checks"][0]] + [
|
||||||
|
{"id": "K-3", "applies_to": "E-3", "statement": "x", "source": "NFR-1"}
|
||||||
|
]
|
||||||
|
|
||||||
|
summary = change_summary(*versions(grow))
|
||||||
|
|
||||||
|
assert summary["elements"]["added"] == ["E-3"]
|
||||||
|
assert summary["checks"] == {"added": ["K-3"], "changed": [], "removed": ["K-2"]}
|
||||||
|
# K-2 came from DR-1 on E-2: nothing that changed reaches it.
|
||||||
|
assert summary["unrequested"] == {"checks": ["K-2"]}
|
||||||
|
|
||||||
|
|
||||||
|
# --- step 5: Approvals show the change --------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_an_approval_in_a_change_request_shows_what_that_phase_changes(
|
||||||
|
tmp_path,
|
||||||
|
):
|
||||||
|
from app.presentation import describe_prompt
|
||||||
|
|
||||||
|
await delivered(tmp_path)
|
||||||
|
engine = engine_for(tmp_path, changed())
|
||||||
|
|
||||||
|
first = await engine.start(
|
||||||
|
"Availability goes up.", customer_case=CASE, execution_id="v2", baseline="v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
approval = first.prompt["approval"]
|
||||||
|
assert approval["phase"] == "requirements_discovery"
|
||||||
|
assert approval["baseline"] == "v1"
|
||||||
|
assert [item["id"] for item in approval["changes"]["requirements"]] == ["NFR-1"]
|
||||||
|
text = describe_prompt(first.prompt)
|
||||||
|
assert "This changes the architecture delivered in v1." in text
|
||||||
|
assert (
|
||||||
|
"- Requirement NFR-1 changed: 99.99% availability. (The service is now "
|
||||||
|
"customer-facing.)"
|
||||||
|
) in text
|
||||||
|
|
||||||
|
second = await engine.resume(
|
||||||
|
"v2", {"approval": {"phase": "requirements_discovery", "decision": "approve"}}
|
||||||
|
)
|
||||||
|
|
||||||
|
# The design did not change, and the Approval says so.
|
||||||
|
design = second.prompt["approval"]
|
||||||
|
assert design["phase"] == "solution_architecture" and design["changes"] == {}
|
||||||
|
assert "Nothing in this Phase differs from it." in describe_prompt(second.prompt)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_an_ordinary_approval_says_nothing_of_changes(tmp_path):
|
||||||
|
first = await engine_for(tmp_path, once()).start("Ingest webhooks")
|
||||||
|
|
||||||
|
assert "changes" not in first.prompt["approval"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- step 6: the change summary and the line of versions ----------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_change_request_writes_changes_json_and_a_full_set_of_files(tmp_path):
|
||||||
|
from app.workflows.changes import change_summary
|
||||||
|
|
||||||
|
await delivered(tmp_path)
|
||||||
|
|
||||||
|
result = await engine_for(tmp_path, changed()).start(
|
||||||
|
"Availability goes up.",
|
||||||
|
customer_case=CASE,
|
||||||
|
pre_approved=True,
|
||||||
|
execution_id="v2",
|
||||||
|
baseline="v1",
|
||||||
|
)
|
||||||
|
|
||||||
|
case = tmp_path / "deliverables" / "cases" / CASE / "executions"
|
||||||
|
written = json.loads((case / "v2" / "changes.json").read_text())
|
||||||
|
assert [item["id"] for item in written["requirements"]] == ["NFR-1"]
|
||||||
|
assert written["unrequested"] == {}
|
||||||
|
# It matches a comparison of the two Executions' data files.
|
||||||
|
assert written == change_summary(
|
||||||
|
load_baseline(tmp_path, CASE, "v1")["deliverables"],
|
||||||
|
result.state["deliverables"],
|
||||||
|
)
|
||||||
|
# The Change request stands on its own: every Deliverable is written in full.
|
||||||
|
manifest = json.loads((case / "v2" / "execution.json").read_text())
|
||||||
|
assert manifest["changes"] == "changes.json"
|
||||||
|
assert set(manifest["deliverables"]) == {kind.value for kind in DEPENDENCY_GRAPH}
|
||||||
|
assert (case / "v2" / "solution-architecture-guide.md").is_file()
|
||||||
|
assert not (case / "v1" / "changes.json").exists()
|
||||||
@@ -11,7 +11,12 @@ def delivered(tmp_path, case: str, statuses: list[str]) -> None:
|
|||||||
folder = tmp_path / "deliverables" / "cases" / case / "intent"
|
folder = tmp_path / "deliverables" / "cases" / case / "intent"
|
||||||
folder.mkdir(parents=True)
|
folder.mkdir(parents=True)
|
||||||
intents = [
|
intents = [
|
||||||
{"version": n, "workflow_request": f"Design thing {n}.", "status": status}
|
{
|
||||||
|
"version": n,
|
||||||
|
"workflow_request": f"Design thing {n}.",
|
||||||
|
"status": status,
|
||||||
|
"execution_id": f"v{n}",
|
||||||
|
}
|
||||||
for n, status in enumerate(statuses, start=1)
|
for n, status in enumerate(statuses, start=1)
|
||||||
]
|
]
|
||||||
(folder / "history.json").write_text(
|
(folder / "history.json").write_text(
|
||||||
@@ -53,6 +58,7 @@ async def test_a_change_needs_something_delivered_to_change(tmp_path):
|
|||||||
no_case = await intake.read("Raise availability to 99.99%.")
|
no_case = await intake.read("Raise availability to 99.99%.")
|
||||||
|
|
||||||
assert with_history.kind == CHANGE_REQUEST and with_history.earlier == 1
|
assert with_history.kind == CHANGE_REQUEST and with_history.earlier == 1
|
||||||
|
assert with_history.latest == "v1" # what the change would change
|
||||||
assert "- Design thing 1." in decider.asked[0][0]
|
assert "- Design thing 1." in decider.asked[0][0]
|
||||||
assert "Design thing 2." not in decider.asked[0][0] # still in progress
|
assert "Design thing 2." not in decider.asked[0][0] # still in progress
|
||||||
assert without.kind == NEW_REQUEST and no_case.kind == NEW_REQUEST
|
assert without.kind == NEW_REQUEST and no_case.kind == NEW_REQUEST
|
||||||
|
|||||||
Reference in New Issue
Block a user