feat: scoping is the first Phase, completed in conversation (iteration 9 step 2)
Some checks failed
validation / verify (push) Failing after 10s

Every Execution now begins by agreeing the scope, before requirements.

- A new Phase and Deliverable, produced by a scope analyst with two tools:
  scope_read shows what is settled and what is not; scope_set records fields
  with where each value came from (the request, the customer's answer, or an
  assumption with its reason). Code checks each value against the catalogue.
- With someone present the analyst asks for what is missing; with nobody
  present it assumes it, visibly. With no spec it drafts one from the request.
  A value the customer stated in the file is changed only by their own answer.
- A submission cannot leave an essential field unknown. Code writes the
  completed spec, the data every later Specialist reads, and the information
  register, from what was recorded.
- The spec's status follows the Execution: under review, approved, then
  locked as a Baseline. A Change request starts from the delivered scope.
- The requirements analyst derives from the agreed scope and no longer asks
  for what it settles.

About 80 tests assumed requirements came first; tests/scoped.py runs scoping
ahead of a test's own script, and tests about order and counts are updated.
Not run with real models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-11 18:17:32 +01:00
parent afe78640f8
commit 545097530b
38 changed files with 1054 additions and 123 deletions

View File

@@ -14,7 +14,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
| 8 | `iter-08-terraform-modules/` | steps 1–5 built and run once each way on 2026-10-09, both at 100%; steps 6 and 7 wait on Azure and AWS provider bundles | Infrastructure code is plain resources by default. An Execution may opt into published Terraform modules: only from registry namespaces the provider bundle lists, at one exact version, with a lookup of what a module accepts and a Skill of four Google modules tested against the pinned provider. |
| 9 | `iter-09-scoping/` | step 1 built on 2026-10-11, not run with real models; steps 2–6 planned; ADR-0008 accepted | An engagement is scoped from a scope spec: a structured file, complete or partly filled, read against a catalogue in code, completed in conversation and approved before requirements begin. Covers workloads and platform foundations. |
| 9 | `iter-09-scoping/` | steps 1, 2 and 6 built on 2026-10-11, not run with real models; steps 3–5 planned; ADR-0008 accepted | An engagement is scoped from a scope spec: a structured file, complete or partly filled, read against a catalogue in code, completed in conversation and approved before requirements begin. Covers workloads and platform foundations. |
| 10 | `iter-10-work-breakdown/` | planned; nothing built | A work breakdown structure as a Deliverable: work packages traced to Elements and Checks, activities in sequence, and a timeline on a relative day axis computed and drawn by code. |
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.

View File

@@ -2,7 +2,7 @@
**Iteration:** 9, Scoping
**Intent:** Scope an engagement from a scope spec: a structured file that may arrive complete, partly filled or not at all, completed in conversation and approved by the customer before requirements begin. So that a design's quality no longer rests on what the agents had to assume.
**Status:** step 1 built on 2026-10-11, not run with real models; steps 2 to 6 planned
**Status:** steps 1, 2 and 6 built on 2026-10-11, not run with real models; steps 3 to 5 planned
**This directory covers:** steps 1 to 6
## Sources of truth
@@ -16,11 +16,11 @@
| Step | What | Status |
|---|---|---|
| 1 | The catalogue, reading and validating a spec, the gap list, the two files beside the Intent, run options from the spec, the spec shown to every Specialist | built 2026-10-11 (`steps/step-1.md`); not run with real models |
| 2 | The conversation that fills gaps and writes answers into the completed copy; scoping as a Phase with its Deliverable and Approval; a spec drafted from a sentence | planned |
| 2 | The conversation that fills gaps and writes answers into the completed copy; scoping as a Phase with its Deliverable and Approval; a spec drafted from a sentence | built 2026-10-11 (`steps/step-2.md`); not run with real models |
| 3 | Requirements traced to the customer's requirement ids and to spec fields, in `requirements.json`; readiness in the solution guide | planned |
| 4 | An edited spec as a Change request, compared field by field | planned |
| 5 | Evaluation cases as specs, replacing answers matched by pattern | planned; `case-08-landing-zone` is the first |
| 6 | The status `BASELINE_LOCKED` set when the Execution completes | planned |
| 6 | The status `BASELINE_LOCKED` set when the Execution completes | built with step 2 |
## Dependency graph

View File

@@ -11,7 +11,7 @@ Built on 2026-10-11. **Not run with real models.**
7. **For a person:** `python -m app.scope check <file>` says what a spec settles and what it lacks; `python -m app.scope template <kind>` prints one to fill in.
8. **A benchmark spec:** `eval/datasets/specs/landing-zone.yaml`, a fictional regulated payment platform, and `case-08-landing-zone` which starts from it.
Interim, until step 2:
Interim, until step 2 (now built; see `step-2.md`):
- **Gaps are asked by the requirements analyst,** who is handed them, not by a scoping Phase. Answers are not written back into the completed copy.
- **Status** is set once, at the start: `DRAFT` with gaps to ask, `UNDER_REVIEW` when complete and someone is present, `APPROVED` when pre-approved. `BASELINE_LOCKED` is not set.

View File

@@ -0,0 +1,28 @@
# Step 2 — Implementation Detail
Built on 2026-10-11. **Not run with real models.**
1. **Scoping is the first Phase.** `PhaseId.SCOPING` comes before requirements discovery, with one Deliverable, `scope`, produced by a new Specialist, the scope analyst (`agents/scope-analyst.md`). Requirements depend on it. Every Execution now begins with it, whether or not a spec was supplied, and the customer approves it before requirements begin.
2. **The scope analyst completes the spec with two tools** (`app/tools/scoping.py`). `scope_read` shows what is settled and what is not, with the question to ask for each. `scope_set` records fields, each with where its value came from: `request` (the customer's request says it), `answered` (they said so in conversation) or `assumed` (the analyst chose, and must say why). Code checks every value against the catalogue as it is recorded.
3. **What the customer stated in the file stays.** The analyst cannot overwrite a supplied value by assumption; only the customer's own answer changes it.
4. **The conversation.** With someone present the analyst asks, through `needs_input`, for the essential fields and the important ones that most change a design, up to eight at a time. An answer is recorded as answered; "you decide" becomes an assumption; a skipped field stays unknown. With nobody present it assumes the essential and important fields from the catalogue's guidance and what the request implies.
5. **A spec drafted from a sentence.** With no spec supplied the analyst starts from an empty one, records what the request states and settles the rest the same way.
6. **A submission cannot leave an essential field unknown.** It is refused, naming the fields.
7. **The Deliverable is written by code from what was recorded.** The analyst writes the scope statement. Code writes the two files (`scope.yaml`, the completed spec; `scope.json`, what every later Specialist reads) and appends the information register, the assumptions with what each would change, and what is still to confirm.
8. **Every later Specialist reads the agreed scope**, as `scope` in its context: what the customer stated, what was assumed for them (shown apart), and what is unknown. The requirements analyst derives from it and no longer asks for what it settles.
9. **Status is kept current** in the Execution's `scope.yaml` and in the copy beside the Intent: `UNDER_REVIEW` once submitted, `APPROVED` when the Phase is approved, `BASELINE_LOCKED` when the Execution completes (which closes step 6). `execution.json` carries the status and the readiness count.
10. **A Change request starts from the scope that was delivered,** with where each value came from, and the analyst records only what the change reaches. A Baseline delivered before scoping existed has no scope; the analyst then completes one from the change and the delivered requirements.
11. **The Approval summary** for scoping says how much was assumed and what is unknown.
What it changed elsewhere:
- About 80 existing tests assumed requirements came first. `tests/scoped.py` holds helpers that run scoping ahead of a test's own script, so tests about later Phases start where they always did. Tests about order, counts and first prompts were updated to the new truth.
- `workflow.yaml` gains the Phase.
Not done here:
- **No real model has run the scope analyst.** How well it drafts from a sentence, how many questions it asks, and what it costs are unknown. Every evaluation case now begins with it.
- **Answers are parsed by the model**, not by code, before `scope_set` checks their type. Nothing reads a value back to the customer other than the draft they approve.
- A re-supplied file part-way through a conversation is not handled; the customer answers in chat.
Tests: `tests/execution/test_scope_spec.py` (12), and the existing suites as updated.

View File

@@ -43,7 +43,7 @@ The completed Execution, in the same Customer case, that a Change request starts
_Avoid_: parent, previous run, source Execution
**Phase**:
A named grouping of Deliverables: requirements discovery, solution architecture (the design, ending with the Target architecture), solution implementation, solution validation, or packaging. A Phase is complete when all its Deliverables are submitted and the user has granted the Phase's Approval.
A named grouping of Deliverables: scoping, requirements discovery, solution architecture (the design, ending with the Target architecture), solution implementation, solution validation, or packaging. A Phase is complete when all its Deliverables are submitted and the user has granted the Phase's Approval.
_Avoid_: step, node, stage, pipeline
**Approval**:
@@ -89,7 +89,7 @@ _Avoid_: custom prompt, override, plugin
### Outputs
**Deliverable**:
A document an Execution produces: requirements, technical decomposition, product selection, design recommendations, the Target architecture, architecture description, deployment guidance, validation results, or the Solution guide. Each Deliverable depends on others being approved first.
A document an Execution produces: the scope, requirements, technical decomposition, product selection, design recommendations, the Target architecture, architecture description, deployment guidance, validation results, or the Solution guide. Each Deliverable depends on others being approved first.
_Avoid_: artifact, output file
**Dependency**:

View File

@@ -10,6 +10,7 @@ Vocabulary is defined in [CONTEXT.md](CONTEXT.md), decisions in [docs/adr/](docs
### 1. Orchestrator and Specialists (`agents/`)
One Orchestrator coordinates an Execution and is the only agent that talks to the user. It is a router ([ADR-0007](docs/adr/0007-jev-is-the-orchestrators-decision-point.md)): code reads the Execution's state, a decision model (Jev) answers the closed questions that need judgement, and a playbook in code picks the action. A language model only writes what the user reads. It delegates each Deliverable to one Specialist, and independent Deliverables run in parallel:
- **Scoping:** `scope-analyst`, which completes the scope spec with the customer.
- **Requirements discovery:** `requirements-analyst`, `decomposer`.
- **Solution architecture (design):** `product-selector`, `design-advisor` (which consults one `pillar-advisor` per architecture-framework pillar), `target-architect`.
- **Solution implementation:** `describer`, `iac-author`.
@@ -18,7 +19,7 @@ One Orchestrator coordinates an Execution and is the only agent that talks to th
Each Phase needs the user's explicit Approval before work that depends on it starts. A pre-approved mode serves batch and evaluation runs.
An Execution may start from a **scope spec** ([ADR-0008](docs/adr/0008-scoping-starts-from-a-scope-spec.md)): a YAML file of what the customer wants and the facts a design depends on, complete or partly filled. Code reads it against a catalogue, refuses a field it does not know, and writes a completed copy beside the customer's request with what was assumed and what is still unknown. `python -m app.scope template platform_foundation` prints one to fill in, and `python -m app.scope check <file>` says what a spec lacks. Pass it as `scope` to `/generate` or in A2A metadata.
An Execution may start from a **scope spec** ([ADR-0008](docs/adr/0008-scoping-starts-from-a-scope-spec.md)): a YAML file of what the customer wants and the facts a design depends on, complete or partly filled. Code reads it against a catalogue and refuses a field it does not know. Scoping, the first Phase, completes it: the scope analyst asks for what is missing when someone is present and assumes it, visibly, when nobody is. With no spec it drafts one from the request. The customer approves the result before requirements begin, and a completed copy is kept beside their request with what was assumed and what is still unknown. `python -m app.scope template platform_foundation` prints one to fill in, and `python -m app.scope check <file>` says what a spec lacks. Pass it as `scope` to `/generate` or in A2A metadata.
A **Change request** ([ADR-0006](docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md)) starts from a completed Execution in the same Customer case, its Baseline, and delivers the full updated Deliverables with `changes.json`: what changed, what was only reworded, and anything changed that nobody asked for.

View File

@@ -13,7 +13,7 @@ Your task message carries JSON context:
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
- `baseline`: set only in a Change request, where the user has said how their requirements changed since an architecture was delivered. `deliverable` is that delivered version of your own Deliverable, and `changes` is what has changed so far: the requirements added, changed and removed, with the reason, and everything already revised from them. `null` otherwise.
- `read_first`: references of your Skills that this task was judged to need, each as a Skill and a resource. Read these first with `load_skill_resource`. It is where to start, not a limit: read anything else you find you need.
- `scope`: what the customer's scope spec says, when they supplied one. `stated` is theirs and binds your Deliverable; `to_decide` is yours to choose and record as an assumption; `to_ask` is unknown. Load the scoping Skill for the rules. Honour every stated choice, build nothing under `stated.intent.exclusions`, and name the customer's own requirement id when your work serves one.
- `scope`: what the customer's scope spec says, when they supplied one. `stated` is theirs and binds your Deliverable; `assumed` was chosen for them and may change when they answer; `to_ask` is unknown. Load the scoping Skill for the rules. Honour every stated choice, build nothing under `stated.intent.exclusions`, and name the customer's own requirement id when your work serves one.
- `overlay_skills`: Skills that carry the user's organisation's standards or this engagement's constraints. Load each one and apply it. Where an overlay disagrees with the Cloud provider's general guidance, the overlay wins; where it disagrees with a requirement the user stated, follow the requirement and say so.
- `pre_approved`: `true` when no user is available.

View File

@@ -11,6 +11,7 @@ The context has `phase`, the Phase the user is being asked to approve, and `fact
Write a summary a busy architect can approve from:
- Say what the Phase produced and what approving it commits them to. For the design Phase, say that no infrastructure code is written until they approve it.
- For the scoping Phase, `facts.scope` says which fields were assumed for them and which are still unknown. Say how many of each, name the ones that most shape a design, and say that requirements are derived from this scope once they approve it.
- State every number in `facts` that bears on the decision, exactly as given. Add no fact that is not there.
- Name anything they should look at before approving: failed or unverifiable Checks, open gaps, recommendations treated as not applicable, Deliverables that were revised.
- Use plain sentences and no headings. Keep it under 120 words.

View File

@@ -7,12 +7,13 @@ includes: [specialist-protocol]
You produce the **requirements** Deliverable: the Skill's requirements-discovery gathering and its resolution of ambiguities and contradictions.
- Cover functional requirements, non-functional requirements (security, privacy, compliance, reliability, disaster recovery, cost, operations, performance, sustainability), current state, and dependencies.
- The user states what they want at a high level; the detail is your job. When a non-functional requirement is missing, propose a default for it from the Cloud provider's architecture framework and record it under `Assumptions` with what it decides: availability tiers, scaling, cost budgets, resource types and security posture. Ask the user to confirm the whole set in one question.
- Ask an open question only where no sensible default can be inferred from the request, such as the region, a compliance regime or a budget ceiling.
- The scope has been agreed before you start: `scope` holds what the customer stated, what was assumed for them and what is still unknown. The detail is your job. Derive the requirements from it; do not ask again for anything it settles.
- Where a non-functional requirement needs something the scope does not hold, propose a default from the Cloud provider's architecture framework and record it under `Assumptions` with what it decides. Carry every entry of `scope.assumed` into `Assumptions` too, so a reader of the requirements sees what rests on one.
- Ask a question only for something the scope leaves unknown that you cannot write a requirement without.
- Name every ambiguity or contradiction, explain why the conflicting goals can't both hold, and ask the user to choose. When they delegate the choice, propose one and ask them to approve it.
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
- Name no products, services or architectures of your own choosing. This Deliverable describes the problem, not the solution. A product or design choice the customer stated in `scope` is theirs: record it as a constraint, in their words.
- When `scope` is set, derive the requirements from it. Cover every entry of `stated.intent.requirements` and name its id in the requirement that covers it. Use what is `stated` as given and ask nothing it already answers. Ask the questions under `to_ask`, the most severe first, when you may ask the user; otherwise assume a default for each and list it under `Assumptions` and `Open questions`. Choose a default for each field under `to_decide` and list it under `Assumptions` with what it affects.
- Cover every entry of `scope.stated.intent.requirements`, the customer's own requirements, and name its id in the requirement that covers it. List what `scope.to_ask` still holds under `Open questions`.
- Give every requirement an id and attach them all as `requirements.json`, in the format the requirements-chain Skill gives. Use the same ids in your content.
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, a process people follow, or a property of a region or product that only its provider publishes (a carbon rating, a service level). For the last, the condition is the choice configuration can show, such as which region is set. Those are recorded and reported, and are not checked against the infrastructure code.

24
agents/scope-analyst.md Normal file
View File

@@ -0,0 +1,24 @@
---
role: scope-analyst
deliverable: scope
tools: []
budget: 30
includes: [specialist-protocol]
---
You produce the **scope** Deliverable: the agreed statement of what the customer wants and the facts a design depends on, as a completed scope spec. You design nothing and choose no products. Load the scoping Skill: it says what each field means.
The scope spec is kept for you. `scope_read` shows what is settled, what is not, and the question to ask for each field. `scope_set` records fields, several per call.
1. Call `scope_read`. When the customer supplied a spec, most of it is already there, and what they stated stays as they stated it.
2. Read `workflow_request`. Record what it states and the spec does not yet hold, with `source` `request`: the statement itself, word for word, under `intent.statement`; the kind of engagement; and any fact it gives. Record only what the words say. Do not infer a number or a choice the customer did not give.
3. Settle what is not settled, the essential fields first.
- **When you may ask the user** (`pre_approved` is `false`): call `needs_input` with the questions for the essential fields, and the important ones that most change a design, at most eight in one call, each in the catalogue's words. Tell them they may answer, say "you decide", or skip one. Record each answer with `source` `answered`. For "you decide", choose a value and record it with `source` `assumed` and a `note`. Leave a skipped field as it is. Ask a second batch only for what is still essential.
- **When you may not ask** (`pre_approved` is `true`): for each essential or important field not settled, choose a value from the catalogue's default guidance and what the request implies, and record it with `source` `assumed` and a `note` saying why. Leave optional fields alone.
- A field marked `decide` is yours: choose, and record it as assumed with a note.
- A field that cannot apply to this kind of engagement is `not_applicable`.
4. Call `scope_read` again. Every essential field must be settled before you can submit.
5. Submit. Your content is the scope statement under its heading: three to six sentences saying what the customer wants, what is in scope, what is out of scope, and what success looks like, in plain words and using only what the spec holds. The completed spec, the information register, the assumptions and the questions still open are written for you from what you recorded. Pass no attachments.
Never record a product, a region, a number or a regulation the customer did not state as though they had: that is `assumed`, with its note. An assumption the customer can see is useful; one presented as their decision is a fault.
In a Change request (`baseline` is set), the spec you start from is the one that was delivered, and `workflow_request` says what has changed. Record only the fields the change reaches, with `source` `answered`: the customer has now said so. Leave every other field as it was. When no spec was delivered with the Baseline, complete one as above, from `workflow_request` and from the requirements in `baseline`.

View File

@@ -37,6 +37,16 @@ Turn = Sequence[Call] | Callable[[AgentTask], Sequence[Call]]
# One small, coherent Requirements chain, so sample submissions pass every link.
SAMPLE_DATA: dict[str, str] = {
"scope.yaml": "engagement:\n kind: workload\nintent:\n statement: Accept events.\n",
"scope.json": json.dumps(
{
"kind": "workload",
"stated": {"intent": {"statement": "Accept events."}},
"to_decide": [],
"to_ask": [],
"not_applicable": [],
}
),
"requirements.json": json.dumps(
[
{"id": "FR-1", "kind": "functional", "text": "Accept events."},

View File

@@ -58,6 +58,7 @@ from app.tools.local import TerraformRunner, local_tools
from app.tools.mcp import McpCatalog, auth_from_config
from app.tools.requirements_review import RequirementsReview
from app.tools.schema_review import Reviews, SchemaReview
from app.tools.scoping import Scoping, to_working
from app.tools.skill_routing import routable
from app.tools.toolkit import make_tool_factory
from app.workflows.deliverables import DEPENDENCY_GRAPH
@@ -155,6 +156,9 @@ class ExecutionEngine:
skill_references=skill_references,
skill_threshold=skill_threshold,
assess_at_end=assess_at_end,
scoping=Scoping(scope_catalogue, dict(scope_values or {}), scope_skill)
if scope_catalogue is not None
else None,
).compile(checkpointer=checkpointer)
async def start(
@@ -297,6 +301,8 @@ class ExecutionEngine:
"status": read.status(pre_approved),
"readiness": read.readiness(),
"context": read.for_context(),
# What the scope analyst starts from and completes.
"working": to_working(read),
}
if read
else None,

View File

@@ -7,6 +7,7 @@ append outcomes that the next Orchestrator turn applies in order.
from __future__ import annotations
import asyncio
import copy
import dataclasses
import json
import logging
@@ -17,6 +18,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Any, cast
import yaml
from langgraph.types import Command, Send, interrupt
from app.adk.runtime import (
@@ -37,16 +39,22 @@ from app.adk.runtime import (
TurnEnded,
)
from app.baseline import WORKING_FILENAME
from app.customer_case import IN_PROGRESS, close_intent
from app.customer_case import (
IN_PROGRESS,
SCOPE_COMPLETED,
close_intent,
intent_dir,
)
from app.nodes.router import ModelOrchestrator, OrchestratorPolicy, TurnView
from app.recorder import ExecutionRecorder
from app.scope import APPROVED, CASE_FIELD, LOCKED, UNDER_REVIEW
from app.states.execution_state import (
UNASSIGNED_CASE,
ExecutionState,
execution_dir,
lessons_file,
)
from app.states.ledger import DeliverableStatus, Ledger, LedgerRefusal
from app.states.ledger import DeliverableStatus, Ledger, LedgerRefusal, PhaseStatus
from app.states.lessons import collect_lesson_candidates, refusal
from app.tools import workspace
from app.tools.assessment import assess
@@ -61,9 +69,12 @@ from app.tools.pillars import (
customer_questions,
pillars_touched,
)
from app.tools.scoping import SCOPE_KEY, Scoping, working_from_completed
from app.tools.skill_routing import read_first
from app.workflows.chain import (
IMPLEMENTATION_FILE,
SCOPE_DATA_FILE,
SCOPE_FILE,
VERDICTS_FILE,
unimplementable_checks,
validation_outcome,
@@ -73,6 +84,7 @@ from app.workflows.deliverables import (
DEPENDENCY_GRAPH,
PHASE_ORDER,
DeliverableKind,
PhaseId,
dependencies_for,
)
from app.workflows.target_architecture import GRAPH_ATTACHMENT
@@ -132,6 +144,7 @@ def make_execution_nodes(
skill_references: Mapping[str, Mapping[str, Mapping[str, str]]] | None = None,
skill_threshold: float = 1.25,
assess_at_end: bool = False,
scoping: Scoping | None = None,
) -> ExecutionNodes:
tool_factory = tool_factory or _no_tools
# How an Orchestrator turn is decided. The default drives the tools from the
@@ -481,7 +494,7 @@ def make_execution_nodes(
role = dataclasses.replace(
role, skills=(*role.skills, *(n for n in named if n not in role.skills))
)
scope_skill = task_input.get("scope_skill")
scope_skill = scoping.skill if scoping else task_input.get("scope_skill")
if scope_skill and scope_skill not in role.skills:
# An Execution with a scope spec: every Specialist can read what it means.
role = dataclasses.replace(role, skills=(*role.skills, scope_skill))
@@ -512,6 +525,16 @@ def make_execution_nodes(
_start_from(kind, before, working, task_input.get("baseline_basis"))
# And from its working material: the advice it was written from.
working.update(task_input.get("baseline_working") or {})
if kind is DeliverableKind.SCOPE and scoping and SCOPE_KEY not in working:
# The spec the customer supplied, or an empty one to draft into.
delivered = ((before or {}).get("attachments") or {}).get(SCOPE_FILE)
working[SCOPE_KEY] = (
# A Change request starts from the scope that was delivered.
(working_from_completed(delivered, scoping) if delivered else None)
or copy.deepcopy(
task_input.get("scope_working") or {"values": {}, "states": {}}
)
)
source = DEPENDENCY_GRAPH[kind].working_copy
if source and source is not kind:
# The tools work on the files as they were last submitted.
@@ -571,6 +594,7 @@ def make_execution_nodes(
library=task_input.get("library"),
review=review if submission_review else None,
baseline=before,
scoping=scoping,
# With a user present, their first sight of a Deliverable comes
# before it is submitted, whatever the Specialist intends.
confirm_first=confirm_drafts
@@ -913,6 +937,9 @@ def _specialist_input(
# Whether this Execution opted into published Terraform modules.
"terraform_modules": bool(state.get("terraform_modules")),
"scope_skill": (state.get("scope") or {}).get("skill"),
# The spec as supplied, for the scope analyst to complete; with the Customer
# case the Execution belongs to, which is not the customer's to restate.
"scope_working": _scope_seed(state) if kind is DeliverableKind.SCOPE else None,
"working": dict(state.get("working", {}).get(kind.value, {})),
# Whether the user has been shown a draft of this Deliverable. Answering some
# other question about it, such as what to do after it stopped, is not that.
@@ -944,7 +971,8 @@ def _specialist_input(
"overlay_skills": sorted(state.get("overlay_skills") or {}),
# What the customer's scope spec states, leaves to the agent, and
# leaves unknown. None when no spec was supplied.
"scope": (state.get("scope") or {}).get("context"),
"scope": _agreed_scope(deliverables)
or (state.get("scope") or {}).get("context"),
# False: infrastructure code is plain resources. True: a published
# module may be used where the modules Skill says one fits.
"terraform_modules": bool(state.get("terraform_modules")),
@@ -1008,6 +1036,32 @@ def _requirement_changes(
)
def _scope_seed(state: ExecutionState) -> dict[str, Any]:
"""The scope spec the analyst starts from: what was supplied, and the Customer
case this Execution already belongs to."""
seed = copy.deepcopy(
(state.get("scope") or {}).get("working") or {"values": {}, "states": {}}
)
seed.setdefault("values", {}).setdefault(CASE_FIELD, _case(state))
return seed
def _agreed_scope(
deliverables: Mapping[str, Mapping[str, Any]],
) -> dict[str, Any] | None:
"""The scope every later Specialist reads: the scope Deliverable's data file."""
text = (
(deliverables.get(DeliverableKind.SCOPE.value) or {})
.get("attachments", {})
.get(SCOPE_DATA_FILE, "")
)
try:
data = json.loads(text) if text else None
except ValueError:
return None
return data if isinstance(data, dict) else None
async def _reading_hints(
state: ExecutionState,
ledger: Ledger,
@@ -1244,6 +1298,22 @@ def _case(state: ExecutionState) -> str:
return state.get("customer_case") or UNASSIGNED_CASE
def _scope_with_status(text: str, status: str) -> tuple[str, dict[str, Any] | None]:
"""A completed scope spec with its status set, and what the Execution's record
says of it. Unchanged, with no record, when it cannot be read."""
try:
document = yaml.safe_load(text)
except yaml.YAMLError:
return text, None
if not isinstance(document, dict):
return text, None
document.setdefault("metadata", {})["status"] = status
return (
yaml.safe_dump(document, sort_keys=False, allow_unicode=True),
{"status": status, "readiness": document.get("readiness")},
)
def _written_before(out_dir: Path) -> set[str]:
"""The files an earlier write of this Execution's folder recorded."""
try:
@@ -1343,6 +1413,22 @@ def _write_outputs(
indent=2,
ensure_ascii=False,
)
scope_record: dict[str, Any] | None = None
if SCOPE_FILE in files:
# The completed scope spec says where it stands, here and beside the Intent.
approved = ledger.phase_status(PhaseId.SCOPING) is PhaseStatus.APPROVED
files[SCOPE_FILE], scope_record = _scope_with_status(
files[SCOPE_FILE],
LOCKED if status == "completed" else APPROVED if approved else UNDER_REVIEW,
)
version = state.get("intent_version")
if version:
copy_path = (
intent_dir(output_root, customer_case)
/ f"{int(version):03d}{SCOPE_COMPLETED}"
)
copy_path.parent.mkdir(parents=True, exist_ok=True)
copy_path.write_text(files[SCOPE_FILE], encoding="utf-8")
# The folder is written more than once as an Execution goes. A file an earlier
# write left that this one does not have, such as a Terraform file removed in a
# Revision, must not stay beside the files that replaced it.
@@ -1365,12 +1451,8 @@ def _write_outputs(
"overlay_skills": dict(state.get("overlay_skills") or {}),
# Whether its infrastructure code was allowed published modules.
"terraform_modules": bool(state.get("terraform_modules")),
# How much of the scope spec the customer supplied; None without a spec.
"scope": {
key: (state.get("scope") or {}).get(key) for key in ("status", "readiness")
}
if state.get("scope")
else None,
# Where the agreed scope stands and how much of it the customer stated.
"scope": scope_record,
"status": status,
"warnings": list(state.get("warnings", [])),
# Flags a person should look at; nothing was done about them.

View File

@@ -14,6 +14,7 @@ tools, so the Ledger refuses anything that is not allowed, whatever was decided.
from __future__ import annotations
import json
import logging
import re
from collections.abc import Awaitable, Callable, Mapping, Sequence
@@ -37,6 +38,7 @@ from app.workflows.chain import (
IMPLEMENTATION_FILE,
RECOMMENDATIONS_FILE,
REQUIREMENTS_FILE,
SCOPE_DATA_FILE,
VERDICTS_FILE,
read_graph,
read_list,
@@ -741,6 +743,24 @@ def approval_facts(phase: PhaseId, view: TurnView, r: Readings) -> dict[str, Any
),
}
facts["counts"] = {name: count for name, count in counts.items() if count}
scope = (view.deliverables.get(DeliverableKind.SCOPE.value) or {}).get(
"attachments", {}
)
if phase is PhaseId.SCOPING and scope.get(SCOPE_DATA_FILE):
# What approving the scope commits to: how much the customer stated, how
# much was assumed for them, and what is still unknown.
try:
agreed = json.loads(scope[SCOPE_DATA_FILE])
except ValueError:
agreed = {}
facts["scope"] = {
"kind": agreed.get("kind"),
"assumed": [item.get("field") for item in agreed.get("assumed", [])],
"left_to_decide": [
item.get("field") for item in agreed.get("to_decide", [])
],
"unknown": [item.get("field") for item in agreed.get("to_ask", [])],
}
graph = read_graph(view.deliverables)
if graph is not None:
facts["target_architecture"] = graph.summary()
@@ -776,6 +796,14 @@ def assembled_summary(phase: PhaseId, facts: Mapping[str, Any]) -> str:
lines.append(
", ".join(f"{count} {name}" for name, count in counts.items()) + "."
)
scope = facts.get("scope")
if scope:
lines.append(
f"Scope of a {scope.get('kind') or 'workload'}: "
f"{len(scope['assumed'])} fields assumed, "
f"{len(scope['unknown'])} still unknown. Requirements are derived "
"from it once you approve."
)
graph = facts.get("target_architecture")
if graph:
lines.append(

View File

@@ -33,6 +33,10 @@ import yaml
ASK, DECIDE, NOT_APPLICABLE = "ask", "decide", "not_applicable"
MARKERS = (ASK, DECIDE, NOT_APPLICABLE)
SUPPLIED, ASSUMED, UNKNOWN = "supplied", "assumed", "unknown"
# Where a settled value came from: the spec file, the customer's request in prose,
# an answer they gave in conversation, or the agent's own choice.
REQUEST, ANSWERED = "request", "answered"
SOURCES = (SUPPLIED, REQUEST, ANSWERED, ASSUMED)
ESSENTIAL, IMPORTANT, OPTIONAL = "essential", "important", "optional"
_SEVERITY = {ESSENTIAL: "HIGH", IMPORTANT: "MEDIUM", OPTIONAL: "LOW"}
DRAFT, UNDER_REVIEW, APPROVED, LOCKED = (
@@ -131,6 +135,13 @@ class Scope:
values: dict[str, Any] = field(default_factory=dict) # field -> stated value
states: dict[str, str] = field(default_factory=dict) # field -> marker
supplied_text: str = ""
# Where each value came from, when it was not the spec file; and what was said
# about it: the reason for an assumption, the question an answer was given to.
sources: dict[str, str] = field(default_factory=dict)
notes: dict[str, str] = field(default_factory=dict)
def source(self, path: str) -> str:
return self.sources.get(path, SUPPLIED)
@property
def customer_case(self) -> str | None:
@@ -163,16 +174,25 @@ class Scope:
"""What the customer left to the agent, each with the default to take and
what a wrong choice would change."""
found: list[dict[str, str]] = []
for path, state in self.states.items():
if state != DECIDE:
for path in self.catalogue.fields:
chosen = path in self.values and self.source(path) == ASSUMED
if not chosen and self.states.get(path) != DECIDE:
continue
spec = self.catalogue.fields[path]
if chosen:
statement = f"Assumed {path} is {self.values[path]}."
if self.notes.get(path):
statement += f" {self.notes[path]}"
else:
default = spec.default if spec.default is not None else "its choice"
statement = (
f"The customer left {path} to the agent. Default: {default}."
)
found.append(
{
"id": f"ASSUMP-{len(found) + 1:02d}",
"field": path,
"statement": f"The customer left {path} to the agent. "
f"Default: {spec.default if spec.default is not None else 'the agent chooses'}.",
"statement": statement,
"impact_if_invalid": f"Changes {spec.affects}."
if spec.affects
else "Local to this field.",
@@ -182,9 +202,11 @@ class Scope:
def readiness(self) -> dict[str, int]:
states = list(self.states.values())
came = [self.source(path) for path in self.values]
return {
SUPPLIED: len(self.values),
ASSUMED: states.count(DECIDE),
SUPPLIED: came.count(SUPPLIED) + came.count(REQUEST),
ANSWERED: came.count(ANSWERED),
ASSUMED: came.count(ASSUMED) + states.count(DECIDE),
UNKNOWN: states.count(ASK),
NOT_APPLICABLE: states.count(NOT_APPLICABLE),
}
@@ -204,7 +226,7 @@ class Scope:
value = self.values.get(path, self.states.get(path, ASK))
_put(document, path, value)
_put(document, "metadata.status", status)
provenance = {path: SUPPLIED for path in self.values}
provenance = {path: self.source(path) for path in self.values}
provenance |= {
path: {DECIDE: ASSUMED, ASK: UNKNOWN, NOT_APPLICABLE: NOT_APPLICABLE}[state]
for path, state in self.states.items()
@@ -220,11 +242,19 @@ class Scope:
choose, and what is unknown."""
stated: dict[str, Any] = {}
for path, value in self.values.items():
if not path.startswith(("run.", "metadata.")):
_put(stated, path, value)
if path.startswith(("run.", "metadata.")) or self.source(path) == ASSUMED:
continue
_put(stated, path, value)
return {
"kind": self.kind,
"stated": stated,
# What the agent chose where the customer did not say. Not the
# customer's decision: it may be revisited when they answer.
"assumed": [
{"field": path, "value": value, "why": self.notes.get(path, "")}
for path, value in self.values.items()
if self.source(path) == ASSUMED and not path.startswith("run.")
],
"to_decide": [
{
"field": path,

View File

@@ -22,6 +22,14 @@ from app.adk.runtime import (
)
from app.states.ledger import Ledger, LedgerRefusal, PhaseStatus
from app.tools.pillars import REVIEW_HEADING, render_checklist_review
from app.tools.scoping import (
REGISTER_HEADING,
Scoping,
render_register,
scope_tools,
unsettled,
)
from app.tools.scoping import attachments as scope_attachments
from app.tools.workspace import (
FILES,
basis_of,
@@ -284,6 +292,7 @@ def specialist_tools(
review: Callable[[Mapping[str, str]], Awaitable[Sequence[str]]] | None = None,
confirm_first: bool = False,
baseline: Mapping[str, Any] | None = None,
scoping: Scoping | None = None,
) -> list[ToolSpec]:
"""A Specialist's tools: submit, ask, its working copy, and its role's tools with
the cloud-read gate.
@@ -298,6 +307,8 @@ def specialist_tools(
this Deliverable: a submission is then refused until the Specialist has put its
draft to them. A rule, because an instruction alone was not always followed.
`baseline` is the Baseline's version of this Deliverable in a Change request.
`scoping` is the scope spec's catalogue; with it the scope Deliverable's two
files and its information register are written from what was recorded.
"""
working = {} if working is None else working
is_graph = kind is DeliverableKind.TARGET_ARCHITECTURE
@@ -310,12 +321,30 @@ def specialist_tools(
if is_validation:
working_verdicts(working, basis) # drops verdicts given on other code
graph_now = read_graph(upstream)
is_scope = kind is DeliverableKind.SCOPE and scoping is not None
async def submit_deliverable(content: str, attachments: Any = None) -> ToolResult:
try:
attachments = files_from_argument(attachments)
except TypeError as exc:
return _refused(f"attachments: {exc}")
if is_scope and scoping is not None:
# Nothing essential may be left unknown, and the files are written
# from what was recorded: they cannot disagree with it.
missing_fields = unsettled(working, scoping)
if missing_fields:
return ToolResult(
"Not submitted. Essential fields of the scope spec are not "
"settled: " + ", ".join(missing_fields) + ". Ask the user for "
"each with `needs_input`; when you may not ask, choose a value "
"and record it with `source` assumed and a note."
)
attachments = scope_attachments(working, scoping)
content = (
content.split(f"\n## {REGISTER_HEADING}")[0].rstrip()
+ "\n\n"
+ render_register(working, scoping)
)
# The working copy is the submission; a file passed here replaces its copy.
if own_files:
attachments = {**working_files(working), **attachments}
@@ -455,6 +484,7 @@ def specialist_tools(
if library
else ()
),
*(scope_tools(working, scoping) if is_scope and scoping else ()),
*(graph_tools(working) if is_graph else ()),
*(
verdict_tools(working, basis, graph_now.check_ids if graph_now else ())

322
app/tools/scoping.py Normal file
View File

@@ -0,0 +1,322 @@
"""The scope analyst's tools: completing the scope spec (ADR-0008).
The spec is kept in the scope Deliverable's working memory as plain data, seeded
from what the customer supplied. The analyst reads what is settled and what is
not, and records each value it settles with where the value came from. Code checks
every value against the catalogue as it is recorded, and writes the completed copy
and the information register from what is there when the Deliverable is submitted:
the spec the customer approves cannot disagree with what was recorded.
"""
from __future__ import annotations
import json
from collections.abc import Mapping, MutableMapping, Sequence
from dataclasses import dataclass, field
from typing import Any
import yaml
from app.adk.runtime import ToolResult, ToolSpec
from app.scope import (
ANSWERED,
ASK,
ASSUMED,
DECIDE,
ESSENTIAL,
IMPORTANT,
MARKERS,
NOT_APPLICABLE,
REQUEST,
SOURCES,
SUPPLIED,
UNDER_REVIEW,
Catalogue,
Scope,
ScopeError,
_fault,
_plain,
read_scope,
)
from app.workflows.chain import SCOPE_DATA_FILE, SCOPE_FILE
SCOPE_KEY = "scope_spec" # working memory: the spec as it is being completed
REGISTER_HEADING = "Information register"
@dataclass(frozen=True)
class Scoping:
"""What reading and completing a scope spec needs."""
catalogue: Catalogue
provider_values: Mapping[str, Sequence[str]] = field(default_factory=dict)
skill: str | None = None
def to_working(scope: Scope) -> dict[str, Any]:
"""A spec as working memory: plain data that survives a checkpoint."""
return {
"kind": scope.kind,
"values": dict(scope.values),
"states": dict(scope.states),
"sources": dict(scope.sources),
"notes": dict(scope.notes),
}
def from_working(memory: Mapping[str, Any], catalogue: Catalogue) -> Scope:
values = dict(memory.get("values") or {})
states = {
path: (memory.get("states") or {}).get(path, ASK)
for path in catalogue.fields
if path not in values
}
kind = values.get("engagement.kind")
return Scope(
catalogue,
str(kind) if kind in catalogue.kinds else None,
values=values,
states=states,
sources=dict(memory.get("sources") or {}),
notes=dict(memory.get("notes") or {}),
)
def working_from_completed(text: str, scoping: Scoping) -> dict[str, Any] | None:
"""A completed scope spec as working memory, with where each value came from.
None when it cannot be read: a Change request then drafts the scope afresh."""
try:
scope = read_scope(
text, scoping.catalogue, provider_values=scoping.provider_values
)
document = yaml.safe_load(text) or {}
except (ScopeError, yaml.YAMLError):
return None
recorded = document.get("provenance") if isinstance(document, Mapping) else None
scope.sources = {
path: str(source)
for path, source in (recorded or {}).items()
if path in scope.values and source in SOURCES
}
return to_working(scope)
def working_scope(working: MutableMapping[str, Any]) -> dict[str, Any]:
return working.setdefault(SCOPE_KEY, {"values": {}, "states": {}})
def unsettled(working: Mapping[str, Any], scoping: Scoping) -> list[str]:
"""The essential fields a submission may not leave unknown."""
scope = from_working(working.get(SCOPE_KEY) or {}, scoping.catalogue)
return [gap["field"] for gap in scope.gaps(ESSENTIAL)]
def attachments(working: Mapping[str, Any], scoping: Scoping) -> dict[str, str]:
"""The Deliverable's two files, written from what was recorded."""
scope = from_working(working.get(SCOPE_KEY) or {}, scoping.catalogue)
return {
SCOPE_FILE: yaml.safe_dump(
scope.completed(UNDER_REVIEW), sort_keys=False, allow_unicode=True
),
SCOPE_DATA_FILE: json.dumps(scope.for_context(), indent=2, ensure_ascii=False),
}
_FROM = {
SUPPLIED: "the scope spec",
REQUEST: "the request",
ANSWERED: "the customer's answer",
ASSUMED: "assumed",
}
def render_register(working: Mapping[str, Any], scoping: Scoping) -> str:
"""The information register, readiness and open questions, as markdown written
by code from what was recorded."""
scope = from_working(working.get(SCOPE_KEY) or {}, scoping.catalogue)
counts = scope.readiness()
lines = [
f"## {REGISTER_HEADING}",
"",
(
f"{counts[SUPPLIED]} stated by the customer, {counts[ANSWERED]} answered "
f"in conversation, {counts[ASSUMED]} assumed, {counts['unknown']} "
f"unknown, {counts[NOT_APPLICABLE]} not applicable."
),
"",
"| Field | Value | From |",
"| --- | --- | --- |",
]
for path, value in scope.values.items():
if path.startswith(("run.", "metadata.")):
continue
shown = (
json.dumps(value, ensure_ascii=False)
if isinstance(value, (list, dict))
else str(value)
)
shown = " ".join(shown.split()).replace("|", "\\|")
lines.append(f"| {path} | {shown[:200]} | {_FROM[scope.source(path)]} |")
assumed = scope.assumptions()
if assumed:
lines += [
"",
"### Assumptions",
"",
"| Id | Assumption | If it is wrong |",
"| --- | --- | --- |",
]
lines += [
f"| {item['id']} | {item['statement']} | {item['impact_if_invalid']} |"
for item in assumed
]
gaps = scope.gaps(ESSENTIAL, IMPORTANT)
if gaps:
lines += [
"",
"### Still to confirm with the customer",
"",
"| Field | Question | Severity |",
"| --- | --- | --- |",
]
lines += [
f"| {gap['field']} | {gap['prompt']} | {gap['severity']} |" for gap in gaps
]
return "\n".join(lines) + "\n"
def scope_tools(working: MutableMapping[str, Any], scoping: Scoping) -> list[ToolSpec]:
catalogue = scoping.catalogue
async def scope_read() -> ToolResult:
scope = from_working(working_scope(working), catalogue)
counts = scope.readiness()
lines = [
f"Kind of engagement: {scope.kind or 'not settled'}",
(
f"{counts[SUPPLIED]} stated, {counts[ANSWERED]} answered, "
f"{counts[ASSUMED]} assumed, {counts['unknown']} unknown."
),
]
for severity, title in (("HIGH", "Essential"), ("MEDIUM", "Important")):
gaps = [gap for gap in scope.gaps() if gap["severity"] == severity]
if gaps:
lines.append(f"{title}, not settled:")
lines += [f" {gap['field']}: {gap['prompt']}" for gap in gaps]
left = [
path
for path, state in scope.states.items()
if state == DECIDE and not path.startswith("run.")
]
if left:
lines.append("Left to you to decide: " + ", ".join(left))
lines.append("Settled: " + (", ".join(scope.values) or "nothing yet"))
return ToolResult("\n".join(lines))
async def scope_set(entries: Any = None) -> ToolResult:
if not isinstance(entries, list) or not entries:
return ToolResult("Refused: `entries` is a list of fields to record.")
memory = working_scope(working)
values, states = (
memory.setdefault("values", {}),
memory.setdefault("states", {}),
)
sources, notes = (
memory.setdefault("sources", {}),
memory.setdefault("notes", {}),
)
problems, done = [], []
for entry in entries:
if not isinstance(entry, Mapping):
problems.append("each entry needs a field")
continue
path = str(entry.get("field", ""))
spec = catalogue.fields.get(path)
if spec is None:
problems.append(f"{path} is not a field of the scope spec")
continue
source, value = str(entry.get("source", "")), entry.get("value")
note = " ".join(str(entry.get("note") or "").split())
if value in MARKERS:
if path in values and sources.get(path, SUPPLIED) == SUPPLIED:
problems.append(f"{path} was stated in the scope spec; it stays")
continue
values.pop(path, None)
states[path] = value
done.append(path)
continue
if source not in SOURCES or source == SUPPLIED:
problems.append(
f"{path}: `source` is request (the customer's request says it), "
"answered (they told you) or assumed (you chose)"
)
continue
if (
path in values
and sources.get(path, SUPPLIED) == SUPPLIED
and source != ANSWERED
):
problems.append(
f"{path} was stated in the scope spec; only the customer's own "
"answer changes it"
)
continue
if source == ASSUMED and not note:
problems.append(f"{path}: an assumption needs a note saying why")
continue
fault = _fault(spec, value, scoping.provider_values.get(path, ()))
if fault:
problems.append(f"{path}: {fault}")
continue
values[path] = _plain(value)
states.pop(path, None)
sources[path] = source
if note:
notes[path] = note
done.append(path)
message = f"Recorded {len(done)}: {', '.join(done) or 'none'}."
if problems:
message += " Not recorded: " + "; ".join(problems) + "."
return ToolResult(message)
return [
ToolSpec(
"scope_read",
"What the scope spec settles so far, and what is not settled, with the "
"question to ask for each.",
{"type": "object", "properties": {}, "required": []},
scope_read,
tier="execution",
),
ToolSpec(
"scope_set",
"Record fields of the scope spec. Each entry has `field`, `value`, "
"`source` (request, answered or assumed) and, for an assumption, a "
"`note` saying why. A `value` of ask, decide or not_applicable marks the "
"field instead. Record several fields in one call.",
{
"type": "object",
"properties": {
"entries": {
"type": "array",
"items": {
"type": "object",
"properties": {
"field": {"type": "string"},
"value": {},
"source": {
"type": "string",
"enum": [REQUEST, ANSWERED, ASSUMED],
},
"note": {"type": "string"},
},
"required": ["field", "value"],
},
}
},
"required": ["entries"],
},
scope_set,
tier="execution",
),
]

View File

@@ -33,6 +33,8 @@ PRODUCTS_FILE = "products.json"
RECOMMENDATIONS_FILE = "recommendations.json"
IMPLEMENTATION_FILE = "implementation.json"
VERDICTS_FILE = "verdicts.json"
SCOPE_FILE = "scope.yaml" # the completed scope spec
SCOPE_DATA_FILE = "scope.json" # what every Specialist reads of it
REQUIREMENT_KINDS = ("functional", "non_functional")
VERDICTS = ("pass", "fail", "not_verifiable")
@@ -343,6 +345,19 @@ def _split_implementation(text: str, name: str) -> tuple[str, str]:
return json.dumps(met), json.dumps(cannot)
def check_scope(attachments: Mapping[str, str], upstream: Upstream | None) -> list[str]:
"""The scope a Specialist reads: what the customer stated, what was left to the
agent, and what is unknown."""
name = SCOPE_DATA_FILE
try:
data = json.loads(attachments[name])
except ValueError as exc:
return [f"{name}: not valid JSON ({exc})"]
if not isinstance(data, dict) or not isinstance(data.get("stated"), dict):
return [f"{name}: must be an object with what the customer stated"]
return []
def check_implementation(
attachments: Mapping[str, str], upstream: Upstream | None
) -> list[str]:

View File

@@ -19,6 +19,8 @@ from app.workflows.chain import (
PRODUCTS_FILE,
RECOMMENDATIONS_FILE,
REQUIREMENTS_FILE,
SCOPE_DATA_FILE,
SCOPE_FILE,
VERDICTS_FILE,
Upstream,
check_components,
@@ -26,6 +28,7 @@ from app.workflows.chain import (
check_products,
check_recommendations,
check_requirements,
check_scope,
check_target_architecture,
check_verdicts,
graph_structure,
@@ -35,6 +38,9 @@ from app.workflows.target_architecture import GRAPH_ATTACHMENT
class PhaseId(StrEnum):
# Scoping: what the customer wants and the facts a design depends on, agreed
# before requirements are derived from it (ADR-0008).
SCOPING = "scoping"
REQUIREMENTS_DISCOVERY = "requirements_discovery"
# Design: ends with the Target architecture, approved before any code is written.
SOLUTION_ARCHITECTURE = "solution_architecture"
@@ -47,6 +53,7 @@ PHASE_ORDER: tuple[PhaseId, ...] = tuple(PhaseId)
class DeliverableKind(StrEnum):
SCOPE = "scope"
REQUIREMENTS = "requirements"
TECHNICAL_DECOMPOSITION = "technical_decomposition"
PRODUCT_SELECTION = "product_selection"
@@ -92,10 +99,20 @@ _K = DeliverableKind
DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
spec.kind: spec
for spec in (
DeliverableSpec(
_K.SCOPE,
PhaseId.SCOPING,
"scope-analyst",
sections=("Scope statement",),
# The completed scope spec, for people and for reuse; and what every
# later Specialist reads of it.
attachments=(SCOPE_FILE, SCOPE_DATA_FILE),
),
DeliverableSpec(
_K.REQUIREMENTS,
PhaseId.REQUIREMENTS_DISCOVERY,
"requirements-analyst",
depends_on=(_K.SCOPE,),
sections=(
"Functional requirements",
"Non-functional requirements",
@@ -209,6 +226,7 @@ DATA_CHECKS: Mapping[
DeliverableKind,
tuple[str, Callable[[Mapping[str, str], Upstream | None], list[str]]],
] = {
_K.SCOPE: (SCOPE_DATA_FILE, check_scope),
_K.REQUIREMENTS: (REQUIREMENTS_FILE, check_requirements),
_K.TECHNICAL_DECOMPOSITION: (COMPONENTS_FILE, check_components),
_K.PRODUCT_SELECTION: (PRODUCTS_FILE, check_products),

View File

@@ -30,6 +30,7 @@ from app.nodes.execution_nodes import make_execution_nodes
from app.nodes.router import OrchestratorPolicy
from app.recorder import ExecutionRecorder
from app.states.execution_state import ExecutionState
from app.tools.scoping import Scoping
def build_execution_graph(
@@ -53,6 +54,7 @@ def build_execution_graph(
skill_references: Mapping[str, Mapping[str, Mapping[str, str]]] | None = None,
skill_threshold: float = 1.25,
assess_at_end: bool = False,
scoping: Scoping | None = None,
) -> StateGraph:
nodes = make_execution_nodes(
runtime=runtime,
@@ -74,6 +76,7 @@ def build_execution_graph(
skill_references=skill_references,
skill_threshold=skill_threshold,
assess_at_end=assess_at_end,
scoping=scoping,
)
graph = StateGraph(ExecutionState)

View File

@@ -43,6 +43,7 @@ models:
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
role_skills:
orchestrator: [google-cloud-solution-architecture]
scope-analyst: [scoping]
requirements-analyst: [google-cloud-solution-architecture, requirements-chain, cloud-discovery]
decomposer: [google-cloud-solution-architecture, requirements-chain]
product-selector: [google-cloud-solution-architecture, requirements-chain]

View File

@@ -29,7 +29,8 @@ A scope spec is a YAML file the customer's architect supplies, complete or partl
- **A stated choice is not yours to generalise.** The rule that requirements name no products is about what you introduce. What the customer wrote stays as they wrote it.
- **Build nothing the customer excluded.** `intent.exclusions` is out of scope for every Deliverable.
- **The customer's own requirements carry their ids.** Each entry in `intent.requirements` must be met. When your work serves one, name its id.
- **Do not fill a gap silently.** A field under `to_ask` is unknown: ask for it when you may ask the user, and otherwise assume a default and say so. A field under `to_decide` is yours to choose: record the choice as an assumption.
- **An assumption is not the customer's decision.** A field under `assumed` was chosen for them, with the reason given. Work from it, and say so where your Deliverable depends on it: it may change when they answer.
- **Do not fill a gap silently.** A field under `to_ask` is still unknown. When your Deliverable needs it, assume a default and say that you did.
## The kinds of engagement

View File

@@ -23,6 +23,8 @@ from app.nodes.intake import Intake
from app.providers import load_provider
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
from tests.scoped import ScopedEngine as ExecutionEngine # noqa: F811
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
AGENTS_DIR = Path(__file__).parents[2] / "agents"
K = DeliverableKind

View File

@@ -18,7 +18,6 @@ from app.agent import ExecutionEngine
from app.providers import load_provider
from app.recorder import InMemoryRecorder
from app.tools.toolkit import make_tool_factory
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
AGENTS_DIR = Path(__file__).parents[2] / "agents"
@@ -42,11 +41,6 @@ def call(tool: str, **args) -> list[types.Part]:
return [types.Part(function_call=types.FunctionCall(name=tool, args=args))]
def requirements_markdown() -> str:
sections = DEPENDENCY_GRAPH[DeliverableKind.REQUIREMENTS].sections
return "\n".join(f"## {section}\nDetails." for section in sections)
def task_context(request: LlmRequest) -> dict:
text = (request.contents[0].parts or [])[0].text or ""
return json.loads(text.split("```json\n", 1)[1].rsplit("\n```", 1)[0])
@@ -58,26 +52,24 @@ async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
models = {
"orchestrator": RoleScriptedLlm(
script=[
call("delegate", kinds=["requirements"]),
call("delegate", kinds=["scope"]),
call("ask_user", questions=[]),
call("delegate", kinds=["requirements"]),
call("delegate", kinds=["scope"]),
call("ask_user", questions=["Anything to add?"]),
]
),
"requirements-analyst": RoleScriptedLlm(
"scope-analyst": RoleScriptedLlm(
script=[
call("load_skill", skill_name="google-cloud-solution-architecture"),
call("load_skill", skill_name="scoping"),
call(
"needs_input", questions=["What availability target do you need?"]
),
call(
"submit_deliverable",
content=requirements_markdown(),
content="# Scope\n## Scope statement\nRegional event ingestion.\n",
attachments=[
{
"path": "requirements.json",
"content": SAMPLE_DATA["requirements.json"],
}
{"path": name, "content": SAMPLE_DATA[name]}
for name in ("scope.yaml", "scope.json")
],
),
]
@@ -101,7 +93,7 @@ async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
assert asked.prompt == {
"questions": [
{
"from": "requirements",
"from": "scope",
"question": "What availability target do you need?",
}
]
@@ -109,11 +101,8 @@ async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
answered = await engine.resume(asked.execution_id, {"reply": "99.9% in one region"})
assert (
answered.state["ledger"]["deliverables"]["requirements"]["status"]
== "submitted"
)
analyst_second_run = models["requirements-analyst"].requests[-1]
assert answered.state["ledger"]["deliverables"]["scope"]["status"] == "submitted"
analyst_second_run = models["scope-analyst"].requests[-1]
assert task_context(analyst_second_run)["answers"] == [
{
"questions": ["What availability target do you need?"],
@@ -122,7 +111,5 @@ async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
]
orchestrator_prompt = models["orchestrator"].requests[0].config.system_instruction
assert "You coordinate one solution-architecture Execution" in orchestrator_prompt
analyst_prompt = (
models["requirements-analyst"].requests[0].config.system_instruction
)
analyst_prompt = models["scope-analyst"].requests[0].config.system_instruction
assert "## How you work" in analyst_prompt

View File

@@ -9,6 +9,8 @@ from app.adk.runtime import ToolResult
from app.agent import Interaction
from app.workflows.target_architecture import parse_graph, render_mermaid
from eval.scoring import CaseSpec, score_execution, template_headings
from tests.scoped import ScopedEngine as ExecutionEngine
from tests.scoped import ScopedRuntime as FakeAgentRuntime
TEMPLATE = (
Path(__file__).parents[2]
@@ -427,9 +429,8 @@ async def test_a_scripted_user_drives_an_interactive_execution(tmp_path):
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
from app.adk.fake_runtime import Call, sample_submission
from app.adk.runtime import RoleSpec
from app.agent import ExecutionEngine
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
from eval.solution_eval import run_with_scripted_user

View File

@@ -24,6 +24,8 @@ from app.baseline import BaselineError, load_baseline
from app.nodes.router import Router, Thresholds
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
from tests.scoped import ScopedEngine as ExecutionEngine # noqa: F811
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
K = DeliverableKind
CASE = "acme-webhooks"

View File

@@ -45,6 +45,7 @@ def test_dependencies_never_point_to_a_later_phase_and_form_no_cycle():
def test_every_phase_groups_at_least_one_deliverable_with_a_distinct_specialist():
assert PHASE_ORDER == (
PhaseId.SCOPING,
PhaseId.REQUIREMENTS_DISCOVERY,
PhaseId.SOLUTION_ARCHITECTURE,
PhaseId.SOLUTION_IMPLEMENTATION,
@@ -61,7 +62,8 @@ def test_the_target_architecture_sits_between_the_design_and_the_code():
depends = {kind: set(spec.depends_on) for kind, spec in DEPENDENCY_GRAPH.items()}
K = DeliverableKind
assert depends[K.REQUIREMENTS] == set()
assert depends[K.SCOPE] == set()
assert depends[K.REQUIREMENTS] == {K.SCOPE}
assert depends[K.TECHNICAL_DECOMPOSITION] == {K.REQUIREMENTS}
assert depends[K.PRODUCT_SELECTION] == {K.TECHNICAL_DECOMPOSITION}
assert depends[K.DESIGN_RECOMMENDATIONS] == {K.PRODUCT_SELECTION, K.REQUIREMENTS}
@@ -89,16 +91,18 @@ def test_the_design_is_approved_before_code_is_written():
def test_inputs_are_earlier_phases_plus_indirect_dependencies_in_graph_order():
design = (
DeliverableKind.SCOPE,
DeliverableKind.REQUIREMENTS,
DeliverableKind.TECHNICAL_DECOMPOSITION,
DeliverableKind.PRODUCT_SELECTION,
DeliverableKind.DESIGN_RECOMMENDATIONS,
DeliverableKind.TARGET_ARCHITECTURE,
)
assert inputs_of(DeliverableKind.REQUIREMENTS) == ()
assert inputs_of(DeliverableKind.SCOPE) == ()
assert inputs_of(DeliverableKind.REQUIREMENTS) == (DeliverableKind.SCOPE,)
assert inputs_of(DeliverableKind.DEPLOYMENT_GUIDANCE) == design
assert inputs_of(DeliverableKind.ARCHITECTURE_DESCRIPTION) == design
assert inputs_of(DeliverableKind.TARGET_ARCHITECTURE) == design[:4]
assert inputs_of(DeliverableKind.TARGET_ARCHITECTURE) == design[:5]
assert set(inputs_of(DeliverableKind.SOLUTION_GUIDE)) == set(DeliverableKind) - {
DeliverableKind.SOLUTION_GUIDE
}

View File

@@ -10,11 +10,13 @@ import pytest
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.checkpoint.sqlite.aio import AsyncSqliteSaver
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
from app.adk.fake_runtime import Call, sample_submission
from app.adk.runtime import RoleSpec, ToolResult, ToolSpec
from app.agent import ExecutionEngine
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
from tests.scoped import ScopedEngine
from tests.scoped import ScopedRuntime as FakeAgentRuntime
K = DeliverableKind
P = PhaseId
@@ -71,7 +73,7 @@ def make_engine(tmp_path: Path, runtime, **overrides) -> ExecutionEngine:
"output_root": tmp_path,
}
options.update(overrides)
return ExecutionEngine(**options)
return ScopedEngine(**options)
def happy_path_turns(**extra_specialist_turns) -> dict:
@@ -132,7 +134,7 @@ async def test_pre_approved_execution_completes_and_writes_deliverables(tmp_path
assert recorder.statuses[result.execution_id][-1]["status"] == "completed"
assert {s["kind"] for s in recorder.submissions[result.execution_id]} == set(K)
recorded_types = [e.type for e in recorder.events[result.execution_id]]
assert recorded_types.count("approved") == 5
assert recorded_types.count("approved") == 6 # scoping, then the five after it
async def test_independent_specialists_run_together_between_orchestrator_turns(
@@ -161,6 +163,7 @@ async def test_specialist_receives_its_dependencies_content(tmp_path):
)
dependencies = describer_run.task.context["dependencies"]
assert list(dependencies) == [
"scope",
"requirements",
"technical_decomposition",
"product_selection",
@@ -718,7 +721,7 @@ async def test_pre_approved_questions_use_matching_scripted_answers(tmp_path):
]
}
)
engine = make_engine(tmp_path, runtime, max_orchestrator_turns=2)
engine = make_engine(tmp_path, runtime, max_orchestrator_turns=3)
result = await engine.start(
"Ingestion",

View File

@@ -25,6 +25,14 @@ def approve(ledger: Ledger, phase: P) -> None:
ledger.approve(phase, actor="user")
def scoped(**options) -> Ledger:
"""A Ledger whose scope is agreed: where requirements discovery begins."""
ledger = Ledger(**options)
produce(ledger, K.SCOPE)
approve(ledger, P.SCOPING)
return ledger
def through_phase_one(ledger: Ledger) -> None:
produce(ledger, K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION)
approve(ledger, P.REQUIREMENTS_DISCOVERY)
@@ -49,22 +57,29 @@ def through_phase_three(ledger: Ledger) -> None:
# --- ready work and the two acceptance thresholds -------------------------------
def test_new_execution_has_only_requirements_ready():
def test_a_new_execution_has_only_the_scope_ready():
ledger = Ledger()
assert ledger.ready_work() == (K.REQUIREMENTS,)
assert ledger.ready_work() == (K.SCOPE,)
assert all(ledger.phase_status(p) is PhaseStatus.OPEN for p in P)
def test_once_the_scope_is_approved_requirements_are_ready():
ledger = scoped()
assert ledger.ready_work() == (K.REQUIREMENTS,)
assert ledger.phase_status(P.SCOPING) is PhaseStatus.APPROVED
def test_same_phase_dependency_is_met_once_submitted():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS)
assert ledger.ready_work() == (K.TECHNICAL_DECOMPOSITION,)
def test_cross_phase_dependency_needs_the_earlier_phase_approved():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION)
assert K.PRODUCT_SELECTION not in ledger.ready_work()
@@ -75,7 +90,7 @@ def test_cross_phase_dependency_needs_the_earlier_phase_approved():
def test_independent_work_becomes_ready_together():
ledger = Ledger()
ledger = scoped()
through_phase_one(ledger)
produce(
ledger, K.PRODUCT_SELECTION, K.DESIGN_RECOMMENDATIONS, K.TARGET_ARCHITECTURE
@@ -93,7 +108,7 @@ def test_independent_work_becomes_ready_together():
def test_ready_phases_lists_open_phases_with_every_deliverable_submitted():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS)
assert ledger.phases_ready_for_approval() == ()
@@ -106,7 +121,7 @@ def test_ready_phases_lists_open_phases_with_every_deliverable_submitted():
def test_start_refuses_work_that_is_not_ready_and_changes_nothing():
ledger = Ledger()
ledger = scoped()
with pytest.raises(LedgerRefusal, match="technical_decomposition"):
ledger.start([K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION], actor="orchestrator")
@@ -115,7 +130,7 @@ def test_start_refuses_work_that_is_not_ready_and_changes_nothing():
def test_start_moves_ready_work_in_progress_and_returns_attempts():
ledger = Ledger()
ledger = scoped()
attempts = ledger.start([K.REQUIREMENTS], actor="orchestrator")
@@ -125,7 +140,7 @@ def test_start_moves_ready_work_in_progress_and_returns_attempts():
def test_submit_requires_the_current_attempt():
ledger = Ledger()
ledger = scoped()
attempt = ledger.start([K.REQUIREMENTS], actor="orchestrator")[K.REQUIREMENTS]
with pytest.raises(LedgerRefusal, match="attempt"):
@@ -136,7 +151,7 @@ def test_submit_requires_the_current_attempt():
def test_late_submission_from_a_superseded_attempt_is_refused_after_restart():
ledger = Ledger()
ledger = scoped()
first = ledger.start([K.REQUIREMENTS], actor="orchestrator")[K.REQUIREMENTS]
ledger.release(K.REQUIREMENTS, first, reason="needs input")
second = ledger.start([K.REQUIREMENTS], actor="orchestrator")[K.REQUIREMENTS]
@@ -149,14 +164,14 @@ def test_late_submission_from_a_superseded_attempt_is_refused_after_restart():
def test_submit_refuses_work_not_in_progress():
ledger = Ledger()
ledger = scoped()
with pytest.raises(LedgerRefusal, match="in_progress"):
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
def test_release_returns_work_to_pending_without_a_revision():
ledger = Ledger()
ledger = scoped()
attempt = ledger.start([K.REQUIREMENTS], actor="orchestrator")[K.REQUIREMENTS]
ledger.release(K.REQUIREMENTS, attempt, reason="needs input")
@@ -172,7 +187,7 @@ def test_release_returns_work_to_pending_without_a_revision():
def test_request_approval_refuses_until_every_deliverable_is_submitted():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS)
with pytest.raises(LedgerRefusal, match="technical_decomposition"):
@@ -182,7 +197,7 @@ def test_request_approval_refuses_until_every_deliverable_is_submitted():
def test_approval_moves_through_awaiting_to_approved_by_the_user():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION)
ledger.request_approval(P.REQUIREMENTS_DISCOVERY, actor="orchestrator")
@@ -198,14 +213,14 @@ def test_approval_moves_through_awaiting_to_approved_by_the_user():
def test_approve_refuses_a_phase_not_awaiting_approval():
ledger = Ledger()
ledger = scoped()
with pytest.raises(LedgerRefusal, match="awaiting_approval"):
ledger.approve(P.REQUIREMENTS_DISCOVERY, actor="user")
def test_pre_approved_ledger_grants_approval_immediately():
ledger = Ledger(pre_approved=True)
ledger = scoped(pre_approved=True)
produce(ledger, K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION)
ledger.request_approval(P.REQUIREMENTS_DISCOVERY, actor="orchestrator")
@@ -216,7 +231,7 @@ def test_pre_approved_ledger_grants_approval_immediately():
def test_request_changes_reopens_the_phase_for_further_work():
ledger = Ledger()
ledger = scoped()
produce(ledger, K.REQUIREMENTS, K.TECHNICAL_DECOMPOSITION)
ledger.request_approval(P.REQUIREMENTS_DISCOVERY, actor="orchestrator")
@@ -228,7 +243,7 @@ def test_request_changes_reopens_the_phase_for_further_work():
def test_finish_is_allowed_only_once_packaging_is_approved():
ledger = Ledger()
ledger = scoped()
through_phase_three(ledger)
assert not ledger.can_finish()
@@ -242,7 +257,7 @@ def test_finish_is_allowed_only_once_packaging_is_approved():
def test_reopen_cascades_to_dependents_and_withdraws_affected_approvals():
ledger = Ledger()
ledger = scoped()
through_phase_three(ledger)
result = ledger.reopen(
@@ -274,7 +289,7 @@ def test_reopen_cascades_to_dependents_and_withdraws_affected_approvals():
def test_reopen_invalidates_attempts_of_dependents_already_in_progress():
ledger = Ledger()
ledger = scoped()
through_phase_one(ledger)
produce(ledger, K.PRODUCT_SELECTION)
attempt = ledger.start([K.DESIGN_RECOMMENDATIONS], actor="orchestrator")[
@@ -289,14 +304,14 @@ def test_reopen_invalidates_attempts_of_dependents_already_in_progress():
def test_reopen_refuses_deliverables_that_were_never_submitted():
ledger = Ledger()
ledger = scoped()
with pytest.raises(LedgerRefusal, match="submitted"):
ledger.reopen(K.REQUIREMENTS, "why", actor="orchestrator")
def test_reopen_escalates_at_the_revision_cap_without_changing_state():
ledger = Ledger(max_revisions=2)
ledger = scoped(max_revisions=2)
produce(ledger, K.REQUIREMENTS)
for _ in range(2):
ledger.reopen(K.REQUIREMENTS, "again", actor="orchestrator")
@@ -313,7 +328,7 @@ def test_reopen_escalates_at_the_revision_cap_without_changing_state():
def test_user_can_override_the_revision_cap():
ledger = Ledger(max_revisions=1)
ledger = scoped(max_revisions=1)
produce(ledger, K.REQUIREMENTS)
ledger.reopen(K.REQUIREMENTS, "first", actor="orchestrator")
produce(ledger, K.REQUIREMENTS)
@@ -327,7 +342,7 @@ def test_user_can_override_the_revision_cap():
def test_default_revision_cap_is_three():
assert Ledger().max_revisions == 3
assert scoped().max_revisions == 3
# --- records and persistence -------------------------------------------------------
@@ -335,16 +350,16 @@ def test_default_revision_cap_is_three():
def test_transitions_are_recorded_as_ordered_events():
ledger = Ledger()
produce(ledger, K.REQUIREMENTS)
produce(ledger, K.SCOPE)
assert [(e.sequence, e.type, e.subject) for e in ledger.events] == [
(1, "started", "requirements"),
(2, "submitted", "requirements"),
(1, "started", "scope"),
(2, "submitted", "scope"),
]
def test_round_trips_through_plain_data():
ledger = Ledger(max_revisions=5)
ledger = scoped(max_revisions=5)
through_phase_one(ledger)
produce(ledger, K.PRODUCT_SELECTION)
ledger.reopen(K.PRODUCT_SELECTION, "cost", actor="user")
@@ -358,7 +373,7 @@ def test_round_trips_through_plain_data():
def test_revising_the_target_architecture_returns_the_code_to_draft():
ledger = Ledger()
ledger = scoped()
through_phase_three(ledger)
result = ledger.reopen(

View File

@@ -10,6 +10,8 @@ from app.agent import ExecutionEngine
from app.recorder import InMemoryRecorder
from app.states.lessons import refusal
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
from tests.scoped import ScopedEngine as ExecutionEngine # noqa: F811
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
K = DeliverableKind
P = PhaseId
@@ -195,9 +197,9 @@ async def test_an_execution_that_ends_with_warnings_still_records_lessons(tmp_pa
"reason": "stopped",
}
]
assert not (
tmp_path / "deliverables" / "cases" / "unassigned" / "executions"
).exists()
# Only the scope had been produced, so only the scope was written.
written = {path.name for path in tmp_path.rglob("deliverables/*.md")}
assert written == {"scope.md"}
async def test_a_cloud_read_refusal_keeps_only_its_leading_words(tmp_path):

View File

@@ -48,14 +48,14 @@ async def test_an_execution_waiting_for_its_user_has_written_what_exists(tmp_pat
waiting = await engine.start("Ingest", customer_case="acme", execution_id="v1")
# It stopped for the first Approval: only the first Phase has been produced.
# It stopped for the first Approval: only the scope has been produced.
assert waiting.status == "awaiting_user"
out = folder(tmp_path, "v1")
manifest = json.loads((out / "execution.json").read_text())
assert manifest["status"] == "in_progress"
assert set(manifest["deliverables"]) == {"requirements", "technical_decomposition"}
assert (out / "deliverables" / "requirements.md").read_text()
assert (out / "requirements.json").is_file()
assert set(manifest["deliverables"]) == {"scope"}
assert (out / "deliverables" / "scope.md").read_text()
assert (out / "scope.json").is_file() and (out / "scope.yaml").is_file()
assert manifest["solution_guide"] is None
# Unfinished work is never something a Change request can start from.
try:

View File

@@ -5,6 +5,7 @@ from app.adk.roles import PILLAR_ADVISOR
from app.adk.runtime import RoleSpec, ToolResult, ToolSpec
from app.tools.pillars import CONSULT_TOOL, pillar_consultation_tool
from app.tools.toolkit import make_tool_factory
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
PILLARS = {
"security": "waf-security",

View File

@@ -28,6 +28,8 @@ from app.workflows.target_architecture import (
parse_graph,
render_mermaid,
)
from tests.scoped import ScopedEngine as ExecutionEngine # noqa: F811
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
K = DeliverableKind
P = PhaseId

View File

@@ -25,6 +25,9 @@ from app.nodes.router import (
)
from app.recorder import InMemoryRecorder
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind
from tests.scoped import ScopedEngine as ExecutionEngine # noqa: F811
from tests.scoped import ScopedRuntime as FakeAgentRuntime # noqa: F811
from tests.scoped import scoped_ledger
K = DeliverableKind
@@ -122,6 +125,8 @@ async def test_an_execution_completes_with_no_model_choosing_an_action(tmp_path)
assert result.status == "completed"
assert "orchestrator" not in {run.role for run in runtime.runs}
assert actions(result) == [
"delegate", # the scope
"request_approval",
"delegate", # requirements
"delegate", # technical decomposition
"request_approval",
@@ -198,14 +203,18 @@ async def test_the_approval_summary_is_assembled_from_the_data(tmp_path):
async def test_a_model_writes_the_summary_but_decides_nothing(tmp_path):
turns = {
**once(),
REPLY_WRITER: [[Call("write_reply", {"text": "Two requirements, ready."})]],
REPLY_WRITER: [
[Call("write_reply", {"text": "The scope, ready."})],
[Call("write_reply", {"text": "Two requirements, ready."})],
],
}
engine, runtime = engine_for(tmp_path, turns, writer=True)
first = await engine.start("Ingest webhooks")
assert first.prompt["approval"]["summary"] == "Two requirements, ready."
writer_run = next(run for run in runtime.runs if run.role == REPLY_WRITER)
# The writer ran once for the scope and once for the requirements.
writer_run = [run for run in runtime.runs if run.role == REPLY_WRITER][-1]
assert writer_run.tools == ("write_reply",)
assert writer_run.task.context["facts"]["deliverables"] == [
"requirements",
@@ -478,6 +487,7 @@ async def test_a_requested_change_is_routed_to_the_deliverable_it_affects(tmp_pa
[(state, questions, _)] = decider.asked
assert "The user's message: Use customer-managed keys on the topic." in state
assert questions["change_target"].options == (
"scope",
"requirements",
"technical_decomposition",
"product_selection",
@@ -689,9 +699,8 @@ def test_a_quoted_question_is_removed_from_what_the_user_said():
async def test_the_questions_asked_are_given_to_the_decider_as_background(tmp_path):
from app.nodes.router import TurnView
from app.states.ledger import Ledger
ledger = Ledger()
ledger = scoped_ledger()
ledger.start([K.REQUIREMENTS], actor="orchestrator")
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
decider = FakeDecider(
@@ -765,9 +774,8 @@ async def test_a_reply_that_restates_what_is_agreed_is_an_answer_not_a_change(
):
"""Evaluation case 04: a requirement said again in reply to a later question."""
from app.nodes.router import Readings, TurnView
from app.states.ledger import Ledger
ledger = Ledger()
ledger = scoped_ledger()
ledger.start([K.REQUIREMENTS], actor="orchestrator")
ledger.submit(K.REQUIREMENTS, 1, actor="requirements-analyst")
view = TurnView(

View File

@@ -6,7 +6,7 @@ import pytest
import yaml
from langgraph.checkpoint.memory import InMemorySaver
from app.adk.fake_runtime import FakeAgentRuntime, sample_submission
from app.adk.fake_runtime import Call, FakeAgentRuntime, sample_submission
from app.adk.runtime import NoDecider, RoleSpec
from app.agent import ExecutionEngine
from app.nodes.router import Router, Thresholds
@@ -25,6 +25,7 @@ ROLES = {
},
}
ANALYST = DEPENDENCY_GRAPH[K.REQUIREMENTS].specialist
SCOPE_ANALYST = DEPENDENCY_GRAPH[K.SCOPE].specialist
SPEC = """
metadata: {engagement_id: ENG-77}
engagement: {kind: platform_foundation}
@@ -100,7 +101,8 @@ async def test_the_supplied_file_is_kept_unaltered_beside_a_completed_copy(tmp_p
intent = tmp_path / "deliverables" / "cases" / "eng-77" / "intent"
assert (intent / "001.scope.yaml").read_text() == SPEC
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert completed["metadata"]["status"] == "APPROVED"
# Its Execution completed, so this copy is the record a change starts from.
assert completed["metadata"]["status"] == "BASELINE_LOCKED"
assert completed["networking"]["cidr_allocations"] == "decide"
assert completed["provenance"]["estate.regions.primary"] == "supplied"
assert [a["field"] for a in completed["inferred_assumptions"]] == [
@@ -112,7 +114,7 @@ async def test_the_supplied_file_is_kept_unaltered_beside_a_completed_copy(tmp_p
"supplied": "intent/001.scope.yaml",
}
manifest = json.loads(next(tmp_path.rglob("execution.json")).read_text())
assert manifest["scope"]["status"] == "APPROVED"
assert manifest["scope"]["status"] == "BASELINE_LOCKED"
assert manifest["scope"]["readiness"]["assumed"] == 1
@@ -132,16 +134,179 @@ async def test_every_specialist_is_shown_the_spec_and_given_the_skill(tmp_path):
)
async def test_without_a_spec_nothing_changes(tmp_path):
def settle(**values) -> Call:
"""What a scope analyst records when it must assume what nobody stated."""
return Call(
"scope_set",
{
"entries": [
{
"field": field,
"value": value,
"source": "assumed",
"note": "A default.",
}
for field, value in values.items()
]
},
)
DRAFTED = [
Call(
"scope_set",
{
"entries": [
{
"field": "intent.statement",
"value": "Ingest webhooks",
"source": "request",
},
{"field": "engagement.kind", "value": "workload", "source": "request"},
]
},
),
settle(
**{
"estate.cloud_provider": "gcp",
"estate.mode": "greenfield",
"estate.regions.primary": "europe-west2",
"estate.data_residency": "United Kingdom",
"workload.users.who": "systems",
"workload.users.count": 10,
"workload.availability.target": "99.9%",
"compliance.frameworks": [],
"compliance.data_classification": "internal",
"cost.monthly_budget": 500,
}
),
]
async def test_without_a_spec_the_scope_analyst_drafts_one_from_the_request(tmp_path):
built, runtime = engine(tmp_path)
runtime._turns[SCOPE_ANALYST].clear()
runtime._turns[SCOPE_ANALYST].append([*DRAFTED, sample_submission(K.SCOPE)])
result = await built.start(
"Ingest webhooks", customer_case="acme", pre_approved=True, execution_id="v1"
)
assert result.status == "completed"
intent = tmp_path / "deliverables" / "cases" / "acme" / "intent"
# Nothing was supplied, so there is no supplied file; the completed copy is
# what the analyst drafted.
assert not (intent / "001.scope.yaml").exists()
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert completed["intent"]["statement"] == "Ingest webhooks"
assert completed["metadata"]["customer_case"] == "acme"
assert completed["provenance"]["intent.statement"] == "request"
assert completed["provenance"]["estate.regions.primary"] == "assumed"
assert completed["readiness"]["assumed"] == 10
assumed = {a["field"] for a in completed["inferred_assumptions"]}
assert "cost.monthly_budget" in assumed
# What was assumed is shown apart from what the customer stated.
scope = runtime.seen[ANALYST]["scope"]
assert scope["stated"]["intent"]["statement"] == "Ingest webhooks"
assert "estate" not in scope["stated"]
assert {"field": "cost.monthly_budget", "value": 500, "why": "A default."} in (
scope["assumed"]
)
document = tmp_path / "deliverables" / "cases" / "acme" / "executions" / "v1"
register = (document / "deliverables" / "scope.md").read_text()
assert "## Information register" in register and "### Assumptions" in register
assert "| intent.statement | Ingest webhooks | the request |" in register
async def test_a_scope_with_an_essential_field_unsettled_cannot_be_submitted(tmp_path):
built, runtime = engine(tmp_path)
runtime._turns[SCOPE_ANALYST].clear()
# It records the statement and nothing else, then tries to submit, twice.
runtime._turns[SCOPE_ANALYST].append(
[DRAFTED[0], sample_submission(K.SCOPE), sample_submission(K.SCOPE)]
)
result = await built.start("Ingest webhooks", pre_approved=True)
assert result.state["scope"] is None
assert runtime.seen[ANALYST] == {"skills": ("base",), "scope": None}
manifest = json.loads(next(tmp_path.rglob("execution.json")).read_text())
assert manifest["scope"] is None
assert not list(tmp_path.rglob("*.scope*.yaml"))
run = next(r for r in runtime.runs if r.role == SCOPE_ANALYST)
refused = run.results[1][1]
assert refused.startswith("Not submitted. Essential fields of the scope spec")
assert "estate.regions.primary" in refused
assert result.status == "completed_with_warnings"
async def test_what_the_customer_stated_is_not_overwritten_by_the_analyst(tmp_path):
built, runtime = engine(tmp_path)
runtime._turns[SCOPE_ANALYST].clear()
runtime._turns[SCOPE_ANALYST].append(
[
Call("scope_read"),
settle(**{"estate.regions.primary": "us-central1", "nonsense.field": 1}),
Call(
"scope_set",
{
"entries": [
{
"field": "estate.regions.primary",
"value": "europe-west1",
"source": "answered",
}
]
},
),
Call(
"scope_set",
{
"entries": [
{
"field": "cost.monthly_budget",
"value": "lots",
"source": "answered",
}
]
},
),
sample_submission(K.SCOPE),
]
)
await built.start("", scope=SPEC, execution_id="v1")
run = next(r for r in runtime.runs if r.role == SCOPE_ANALYST)
read, assumed, answered, mistyped = (reply for _, reply in run.results[:4])
assert "Kind of engagement: platform_foundation" in read
assert "Left to you to decide: networking.cidr_allocations" in read
assert "estate.regions.primary was stated in the scope spec" in assumed
assert "nonsense.field is not a field" in assumed
# The customer's own answer does change what the file said.
assert answered == "Recorded 1: estate.regions.primary."
assert "cost.monthly_budget: must be a number" in mistyped
intent = tmp_path / "deliverables" / "cases" / "eng-77" / "intent"
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert completed["estate"]["regions"]["primary"] == "europe-west1"
assert completed["provenance"]["estate.regions.primary"] == "answered"
async def test_the_scope_is_approved_before_requirements_begin(tmp_path):
built, runtime = engine(tmp_path)
waiting = await built.start("", scope=SPEC, pre_approved=False, execution_id="v1")
# Only the scope analyst has run, and the customer is asked to approve the scope.
assert {run.role for run in runtime.runs} == {SCOPE_ANALYST}
assert waiting.prompt["approval"]["phase"] == "scoping"
intent = tmp_path / "deliverables" / "cases" / "eng-77" / "intent"
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert completed["metadata"]["status"] == "UNDER_REVIEW"
approved = await built.resume(
"v1", {"approval": {"phase": "scoping", "decision": "approve"}}
)
assert ANALYST in {run.role for run in runtime.runs}
assert approved.prompt["approval"]["phase"] == "requirements_discovery"
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert completed["metadata"]["status"] == "APPROVED"
async def test_an_essential_gap_refuses_a_start_with_nobody_to_ask(tmp_path):
@@ -196,3 +361,56 @@ async def test_a_request_is_still_needed_when_the_spec_has_no_statement(tmp_path
scope=SPEC.replace(" statement: Build us a landing zone.\n", ""),
pre_approved=False,
)
async def test_a_change_request_starts_from_the_scope_that_was_delivered(tmp_path):
built, runtime = engine(tmp_path)
await built.start("", scope=SPEC, execution_id="v1")
for name, turns in runtime._turns.items():
kind = next(
k for k, spec in DEPENDENCY_GRAPH.items() if spec.specialist == name
)
turns.append([sample_submission(kind)])
runtime._turns[SCOPE_ANALYST].clear()
runtime._turns[SCOPE_ANALYST].append(
[
Call("scope_read"),
Call(
"scope_set",
{
"entries": [
{
"field": "estate.regions.secondary",
"value": "europe-west1",
"source": "answered",
}
]
},
),
sample_submission(K.SCOPE),
]
)
await built.start(
"We now need a second region.",
customer_case="eng-77",
pre_approved=True,
execution_id="v2",
baseline="v1",
)
run = [r for r in runtime.runs if r.role == SCOPE_ANALYST][-1]
# The delivered spec was there to read before anything was recorded.
assert "Kind of engagement: platform_foundation" in run.results[0][1]
intent = tmp_path / "deliverables" / "cases" / "eng-77" / "intent"
changed = yaml.safe_load((intent / "002.scope.completed.yaml").read_text())
assert changed["estate"]["regions"] == {
"primary": "europe-west2",
"secondary": "europe-west1",
"resiliency_tier": "ask",
}
assert changed["provenance"]["estate.regions.primary"] == "supplied"
assert changed["provenance"]["estate.regions.secondary"] == "answered"
# The Baseline's own copy is untouched.
first = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
assert first["estate"]["regions"]["secondary"] == "ask"

80
tests/scoped.py Normal file
View File

@@ -0,0 +1,80 @@
"""Helpers for tests about what comes after scoping.
Every Execution now begins with the scoping Phase (ADR-0008). A test about
requirements, design or validation is not about that, so these run scoping ahead
of the test's own script and leave the test where it always started: requirements
ready, nothing recorded of how the scope was agreed.
"""
from __future__ import annotations
from typing import Any
from app.adk.fake_runtime import FakeAgentRuntime, sample_submission
from app.adk.runtime import Submitted
from app.agent import ExecutionEngine, Interaction
from app.workflows.deliverables import DEPENDENCY_GRAPH, DeliverableKind, PhaseId
SCOPE_ANALYST = DEPENDENCY_GRAPH[DeliverableKind.SCOPE].specialist
ORCHESTRATOR = "orchestrator"
class ScopedRuntime(FakeAgentRuntime):
"""A scripted runtime that first has the scope produced and its Approval asked
for, outside the script and outside `runs`."""
def __init__(self, turns: Any) -> None:
super().__init__(turns)
self._next = "delegate"
async def run(self, role, task, tools): # type: ignore[no-untyped-def]
by_name = {tool.name: tool for tool in tools}
if role.name == SCOPE_ANALYST and not self.remaining(SCOPE_ANALYST):
# Submitted as it stands: a test about a later Deliverable's draft is
# not asking whether the scope's draft was shown first.
call = sample_submission(DeliverableKind.SCOPE)
return Submitted(call.args["content"], call.args["attachments"])
if role.name == ORCHESTRATOR and self._next == "delegate":
self._next = "approve"
asked = await by_name["delegate"].handler(
kinds=[DeliverableKind.SCOPE.value]
)
return asked.outcome
if role.name == ORCHESTRATOR and self._next == "approve":
self._next = "done"
asked = await by_name["request_approval"].handler(
phase=PhaseId.SCOPING.value, summary="The scope is ready."
)
if asked.outcome is not None:
return asked.outcome # a user is present: their Approval is awaited
return await super().run(role, task, tools)
class ScopedEngine(ExecutionEngine):
"""An engine whose `start` grants the scoping Approval when it is asked for, so
an interactive test meets the first prompt it was written about."""
async def start(self, *args: Any, **kwargs: Any) -> Interaction:
interaction = await super().start(*args, **kwargs)
approval = (interaction.prompt or {}).get("approval") or {}
if approval.get("phase") == PhaseId.SCOPING.value:
interaction = await self.resume(
interaction.execution_id,
{"approval": {"phase": PhaseId.SCOPING.value, "decision": "approve"}},
)
return interaction
def scoped_ledger(**options: Any): # type: ignore[no-untyped-def]
"""A Ledger whose scope is agreed: where requirements discovery begins."""
from app.states.ledger import Ledger, PhaseStatus
ledger = Ledger(**options)
attempt = ledger.start([DeliverableKind.SCOPE], actor=ORCHESTRATOR)
ledger.submit(
DeliverableKind.SCOPE, attempt[DeliverableKind.SCOPE], actor=SCOPE_ANALYST
)
ledger.request_approval(PhaseId.SCOPING, actor=ORCHESTRATOR)
if ledger.phase_status(PhaseId.SCOPING) is PhaseStatus.AWAITING_APPROVAL:
ledger.approve(PhaseId.SCOPING, actor="user")
return ledger

View File

@@ -57,12 +57,11 @@ async def test_preview_mcp_servers_follow_the_setting(tmp_path):
async def test_the_assembly_routes_by_default_and_runs_without_a_jev_key(tmp_path):
"""With no Jev key the router follows its playbook on rules alone."""
from app.adk.fake_runtime import sample_submission
from app.adk.fake_runtime import Call
from app.adk.runtime import NoDecider
from app.workflows.deliverables import DeliverableKind
runtime = FakeAgentRuntime(
{"requirements-analyst": [[sample_submission(DeliverableKind.REQUIREMENTS)]]}
{"scope-analyst": [[Call("needs_input", {"questions": ["Which region?"]})]]}
)
async with open_assembly(
@@ -74,9 +73,10 @@ async def test_the_assembly_routes_by_default_and_runs_without_a_jev_key(tmp_pat
interaction = await assembly.engine.start("Design ingestion")
assert isinstance(assembly.decider, NoDecider)
# The router delegated requirements with no Orchestrator model, then the
# decomposer (unscripted here) stopped and its question reached the user.
assert [run.role for run in runtime.runs] == ["requirements-analyst"]
# The router delegated the scope with no Orchestrator model, and the scope
# analyst's question reached the user.
assert [run.role for run in runtime.runs] == ["scope-analyst"]
assert "scope_set" in runtime.runs[0].tools
assert interaction.status == "awaiting_user"
assert "reply-writer" in assembly.roles

View File

@@ -5,8 +5,13 @@ repository: gcp_solution_architecture_agent
# for the dependency graph. A Phase completes when every Deliverable is submitted and the
# user grants its Approval. Outputs are written to deliverables/executions/<execution_id>/.
phases:
- id: scoping
input: [workflow_request, scope_spec]
deliverables: [scope]
attachments: [scope.yaml, scope.json]
approval: required
- id: requirements_discovery
input: workflow_request
input: scope
deliverables: [requirements, technical_decomposition]
product_selection_deferred: true
approval: required