feat: start an Execution from a scope spec (ADR-0008, iteration 9 step 1)
Some checks failed
validation / verify (push) Failing after 10s
Some checks failed
validation / verify (push) Failing after 10s
A scope spec is a YAML file of what the customer wants and the facts a design depends on, complete or partly filled. It sits beside the Intent, which stays in the customer's words. - skills/scoping holds a provider-neutral catalogue of 63 fields; the Cloud provider's bundle supplies the values it allows for the fields left to it. - app/scope.py reads a spec against the catalogue with no model: it refuses an unknown field or a value of the wrong type, and yields what is stated, what is left to the agent, what is unknown and how pressing each gap is. A field may say ask, decide or not_applicable. - ExecutionEngine.start(scope=...), /generate, A2A metadata and evaluation cases accept one. Its statement is the request, its engagement reference names the Customer case, and its run options apply. A pre-approved start is refused when an essential field is not settled. - The spec as supplied is kept unaltered beside a completed copy with assumptions, gaps, provenance, readiness and a status. - Every Specialist is shown the spec and given the Skill: a stated choice binds, exclusions are out of scope. - Covers workloads and platform foundations; case-08-landing-zone starts from a benchmark spec. Plans for scoping (iteration 9) and the work breakdown structure (iteration 10, nothing built), and the design spec they come from. Not run with real models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,8 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
| 8 | `iter-08-terraform-modules/` | steps 1–5 built and run once each way on 2026-10-09, both at 100%; steps 6 and 7 wait on Azure and AWS provider bundles | Infrastructure code is plain resources by default. An Execution may opt into published Terraform modules: only from registry namespaces the provider bundle lists, at one exact version, with a lookup of what a module accepts and a Skill of four Google modules tested against the pinned provider. |
|
||||
| 9 | `iter-09-scoping/` | step 1 built on 2026-10-11, not run with real models; steps 2–6 planned; ADR-0008 accepted | An engagement is scoped from a scope spec: a structured file, complete or partly filled, read against a catalogue in code, completed in conversation and approved before requirements begin. Covers workloads and platform foundations. |
|
||||
| 10 | `iter-10-work-breakdown/` | planned; nothing built | A work breakdown structure as a Deliverable: work packages traced to Elements and Checks, activities in sequence, and a timeline on a relative day axis computed and drawn by code. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
|
||||
34
.agents/plans/iter-09-scoping/MAP.md
Normal file
34
.agents/plans/iter-09-scoping/MAP.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 9, Scoping
|
||||
**Intent:** Scope an engagement from a scope spec: a structured file that may arrive complete, partly filled or not at all, completed in conversation and approved by the customer before requirements begin. So that a design's quality no longer rests on what the agents had to assume.
|
||||
**Status:** step 1 built on 2026-10-11, not run with real models; steps 2 to 6 planned
|
||||
**This directory covers:** steps 1 to 6
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `docs/adr/0008-scoping-starts-from-a-scope-spec.md`: the decisions.
|
||||
- `.scratch/scoping-and-work-breakdown/spec.md`: the design, including the draft spec, the owner's own example and how the two were combined.
|
||||
- `skills/scoping/assets/catalogue.yaml`: every field a scope spec may hold.
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Status |
|
||||
|---|---|---|
|
||||
| 1 | The catalogue, reading and validating a spec, the gap list, the two files beside the Intent, run options from the spec, the spec shown to every Specialist | built 2026-10-11 (`steps/step-1.md`); not run with real models |
|
||||
| 2 | The conversation that fills gaps and writes answers into the completed copy; scoping as a Phase with its Deliverable and Approval; a spec drafted from a sentence | planned |
|
||||
| 3 | Requirements traced to the customer's requirement ids and to spec fields, in `requirements.json`; readiness in the solution guide | planned |
|
||||
| 4 | An edited spec as a Change request, compared field by field | planned |
|
||||
| 5 | Evaluation cases as specs, replacing answers matched by pattern | planned; `case-08-landing-zone` is the first |
|
||||
| 6 | The status `BASELINE_LOCKED` set when the Execution completes | planned |
|
||||
|
||||
## Dependency graph
|
||||
|
||||
- Step 1: no dependencies
|
||||
- Step 2: depends on step 1
|
||||
- Step 3: depends on step 1
|
||||
- Step 4: depends on step 1, iteration 7
|
||||
- Step 5: depends on step 1
|
||||
- Step 6: depends on step 2
|
||||
|
||||
The work breakdown structure is iteration 10 and uses the spec's `delivery` section from step 1.
|
||||
20
.agents/plans/iter-09-scoping/steps/step-1.md
Normal file
20
.agents/plans/iter-09-scoping/steps/step-1.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# Step 1 — Implementation Detail
|
||||
|
||||
Built on 2026-10-11. **Not run with real models.**
|
||||
|
||||
1. **The catalogue** is `skills/scoping/assets/catalogue.yaml`: 63 fields as dotted paths, each with a type, a tier (which may differ by kind of engagement), the question to ask and guidance for a default. It names no Cloud provider. `providers/gcp.yaml` adds, under `scope.values`, what Google Cloud allows for the four fields left to a provider: the provider itself, encryption type, network topology and the kind of hybrid connection.
|
||||
2. **Reading a spec** is `read_scope` in `app/scope.py`, with no model. It refuses, naming every problem: YAML that does not parse, a field the catalogue does not have, a value of the wrong type, and an engagement reference and a customer case that disagree. A whole section may be marked `ask`, `decide` or `not_applicable` at once.
|
||||
3. **What it yields:** the stated values; the gaps, most pressing first, each with its question and a severity (HIGH essential, MEDIUM important, LOW optional); the assumptions, one per `decide`, with the default to take and what a wrong one would change; a count of each; and a status.
|
||||
4. **Starting from a spec:** `ExecutionEngine.start(scope=...)`, `scope` in the `/generate` body and in A2A metadata, and on an evaluation case. The spec's statement is the request when none is given, its engagement reference names the Customer case, and its `run` section applies where the caller gave nothing. A pre-approved start is refused when an essential field is not settled.
|
||||
5. **The two files** are written beside the Intent: `NNN.scope.yaml`, the spec as supplied, byte for byte; and `NNN.scope.completed.yaml`, every field of the catalogue with the agent's sections (`inferred_assumptions`, `discovery_gaps`, `provenance`, `readiness`) and `metadata.status`. `history.json` names both. A completed copy can be supplied again as a spec; the agent's sections are replaced.
|
||||
6. **Every Specialist is shown the spec** as `scope` in its context (what is stated, what is theirs to decide, what is unknown) and is given the scoping Skill. The protocol says a stated choice binds, an exclusion is out of scope, and the customer's requirement ids are to be named. The requirements analyst is told to derive from it and ask only for what it leaves unknown.
|
||||
7. **For a person:** `python -m app.scope check <file>` says what a spec settles and what it lacks; `python -m app.scope template <kind>` prints one to fill in.
|
||||
8. **A benchmark spec:** `eval/datasets/specs/landing-zone.yaml`, a fictional regulated payment platform, and `case-08-landing-zone` which starts from it.
|
||||
|
||||
Interim, until step 2:
|
||||
|
||||
- **Gaps are asked by the requirements analyst,** who is handed them, not by a scoping Phase. Answers are not written back into the completed copy.
|
||||
- **Status** is set once, at the start: `DRAFT` with gaps to ask, `UNDER_REVIEW` when complete and someone is present, `APPROVED` when pre-approved. `BASELINE_LOCKED` is not set.
|
||||
- **A caller's `pre_approved` wins over the spec's** when both are given. `/generate` is always pre-approved.
|
||||
|
||||
Tests: `tests/test_scope.py` (18), `tests/execution/test_scope_spec.py` (8).
|
||||
23
.agents/plans/iter-10-work-breakdown/MAP.md
Normal file
23
.agents/plans/iter-10-work-breakdown/MAP.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# Plan Map
|
||||
|
||||
**Iteration:** 10, Work breakdown structure
|
||||
**Intent:** Deliver a plan for building the Target architecture: work packages traced to its Elements and Checks, activities in sequence, and a timeline on a relative day axis that code computes and draws.
|
||||
**Status:** planned; nothing built
|
||||
**This directory covers:** steps 1 to 4
|
||||
|
||||
## Sources of truth
|
||||
|
||||
- `.scratch/scoping-and-work-breakdown/spec.md`, section "The work breakdown structure".
|
||||
- `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md`: the ids it traces to.
|
||||
- Iteration 9 step 1: the scope spec's `delivery` section (team size, window, commercial model).
|
||||
|
||||
## Status
|
||||
|
||||
| Step | What | Status |
|
||||
|---|---|---|
|
||||
| 1 | The Deliverable, its Specialist and its data file; every Element in a work package, checked in code | planned |
|
||||
| 2 | The schedule computed in code: start and finish day, critical path, team size as a limit | planned |
|
||||
| 3 | The three views written by code: WBS dictionary, activity sequence list, timeline on a relative axis | planned |
|
||||
| 4 | The delivery window checked; assumptions the plan depends on shown on the timeline; milestones | planned |
|
||||
|
||||
Decisions the owner accepted on 2026-10-11: after validation, in the packaging Phase; grouped by component; the technical build only.
|
||||
364
.scratch/scoping-and-work-breakdown/spec.md
Normal file
364
.scratch/scoping-and-work-breakdown/spec.md
Normal file
@@ -0,0 +1,364 @@
|
||||
# Spec: Scoping from a scope spec, and a work breakdown structure
|
||||
|
||||
Status: ready-for-agent
|
||||
|
||||
Accepted by the owner on 2026-10-11. Stage 1 is built; `.agents/plans/iter-09-scoping/` says what is done.
|
||||
|
||||
- Glossary: [`CONTEXT.md`](../../CONTEXT.md). New terms this would add are in **Language** below.
|
||||
- Builds on: [ADR-0005](../../docs/adr/0005-the-target-architecture-is-an-explicit-graph.md) (ids along the Requirements chain), [ADR-0006](../../docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md) (Change requests), [ADR-0002](../../docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md) (behaviour in Skills).
|
||||
- Would need: one new ADR, for a new first Phase and a new start to the Requirements chain.
|
||||
|
||||
## Problem
|
||||
|
||||
The Intent is the customer's request, saved word for word. For case 06 it is one sentence. Everything else a design needs (how many users, what availability, which region, what budget, which compliance regime, what exists today) is assumed by the requirements analyst or, in the evaluation, supplied by pattern-matched answers.
|
||||
|
||||
Three days of runs trace most downstream trouble to what nobody stated: Checks the validator asks for on topics the customer never raised, checklist items the request cannot settle, a "low-carbon region" no code can show. Nothing tells the customer at the start how much of their design rests on assumption, and a requirement does not record whether the customer stated it or the agent assumed it.
|
||||
|
||||
Separately, the agent delivers a design and its code but no plan for building it.
|
||||
|
||||
## Decisions taken
|
||||
|
||||
The owner accepted these defaults on 2026-10-11.
|
||||
|
||||
1. **Scoping is a Phase the customer approves,** before requirements discovery.
|
||||
2. **Its centre is a scope spec:** a YAML file that may arrive complete, partly filled or not at all.
|
||||
3. **Architects write the spec,** the owner and colleagues, not end customers. Field names may be technical.
|
||||
4. **A small set of fields is essential** and always settled before work starts. Which they are depends on the kind of engagement.
|
||||
5. **Two files.** The file supplied is never altered. A completed copy is written beside it.
|
||||
6. **The run options live in the spec,** so one file starts an Execution.
|
||||
7. **Where the spec and the prose request disagree, the spec wins** and the customer is shown the conflict.
|
||||
8. **The work breakdown structure is a new Deliverable,** after validation, in the packaging Phase, grouped by component as the technical decomposition is, covering the technical build.
|
||||
9. **Scoping is built before the work breakdown structure.**
|
||||
|
||||
## Language
|
||||
|
||||
Terms to add to `CONTEXT.md` when built:
|
||||
|
||||
- **Scope spec:** the structured statement of what the customer wants and the facts a design depends on, as a file. It sits beside the Intent, which stays in the customer's words.
|
||||
- **Scoping:** the Phase that completes the scope spec and has the customer approve it.
|
||||
- **Work breakdown structure:** the Deliverable that divides building the Target architecture into work packages and activities, with a timeline.
|
||||
- **Work package:** a unit of work that delivers named Elements and is done when their Checks pass.
|
||||
|
||||
Already in the glossary and used here: Intent, Customer question, Change request, Baseline, Requirements chain.
|
||||
|
||||
## The scope spec
|
||||
|
||||
### What a field can say
|
||||
|
||||
Every field holds a value or one of three words:
|
||||
|
||||
| Written | Meaning | What the agent does |
|
||||
|---|---|---|
|
||||
| a value | The customer states this | Uses it. Recorded as `supplied`. |
|
||||
| `ask`, or the field left out | Not known yet | Asks, when someone is present. Otherwise treats it as `decide` and lists it under `discovery_gaps`. |
|
||||
| `decide` | You choose | Picks a default, recorded as `assumed` with the default and what it affects. |
|
||||
| `not_applicable` | Does not apply | Asks nothing and assumes nothing. |
|
||||
|
||||
A field name the catalogue does not have is an error that names the field. Nothing is silently ignored.
|
||||
|
||||
### Tiers
|
||||
|
||||
| Tier | When it is settled |
|
||||
|---|---|
|
||||
| essential | Always, before any work: asked when someone is present, and refused with a list when nobody is and it is neither supplied nor `decide`. |
|
||||
| important | Asked when someone is present; defaulted otherwise. |
|
||||
| optional | Defaulted unless supplied. |
|
||||
|
||||
### The draft
|
||||
|
||||
Revised on 2026-10-11 to combine the first draft with the owner's own example, an engagement spec for a regulated landing zone. What each brought is listed under **How the two were combined**. Essential fields are marked; which are essential depends on `engagement.kind`. Values shown are from the owner's example.
|
||||
|
||||
```yaml
|
||||
schema_version: 1 # the catalogue this file is written against
|
||||
|
||||
metadata:
|
||||
customer_case: fintech-global-landing-zone # essential: groups this work's Executions
|
||||
engagement_reference: ENG-2026-UK-084 # yours; carried through, never interpreted
|
||||
client_name: FinTech Global UK
|
||||
author: Pre-Sales Engineering
|
||||
revision: "1.1.0" # yours: the version of this document
|
||||
status: DRAFT # written by the agent; see Status below
|
||||
|
||||
engagement:
|
||||
kind: platform_foundation # essential: workload | platform_foundation | migration
|
||||
|
||||
# 1. What the customer said. The source of truth; nothing here is rewritten.
|
||||
intent:
|
||||
statement: | # essential: verbatim, or consolidated from calls and documents
|
||||
We are a UK-regulated payment platform expanding into Google Cloud. We need an
|
||||
enterprise-grade landing zone ready in under 10 weeks ...
|
||||
requirements: # what the customer stated outright, with their ids
|
||||
- id: REQ-01
|
||||
category: Timeline
|
||||
statement: Platform foundation must be ready for workload ingestion within 10 weeks.
|
||||
- id: REQ-03
|
||||
category: Networking
|
||||
statement: 10Gbps on-premises hybrid connectivity; centralized Palo Alto inspection for egress.
|
||||
exclusions:
|
||||
- Application container deployment (managed by app squads).
|
||||
- Active-active secondary region in phase 1 (warm standby acceptable).
|
||||
success_criteria: decide
|
||||
|
||||
# 2. Where it runs.
|
||||
estate:
|
||||
cloud_provider: gcp # essential; must be a provider this deployment has
|
||||
mode: greenfield # essential: greenfield | brownfield_expansion | migration
|
||||
regions:
|
||||
primary: europe-west2 # essential
|
||||
secondary: europe-west1
|
||||
resiliency_tier: warm_standby
|
||||
data_residency: United Kingdom # essential
|
||||
existing_systems: ask
|
||||
|
||||
# 3. Who uses it and how hard. Essential for kind workload; not_applicable for a foundation.
|
||||
workload:
|
||||
users: {who: not_applicable, count: not_applicable, peak_concurrent: not_applicable}
|
||||
availability: {target: ask, hours: always, recovery_time: ask, recovery_point: ask}
|
||||
|
||||
# 4. Regulation and governance.
|
||||
compliance:
|
||||
industry: financial_services
|
||||
frameworks: [DORA, ISO27001, PCI_DSS_v4] # essential; an empty list means none apply
|
||||
data_classification: confidential # essential
|
||||
encryption: {type: cmek, key_rotation_days: 90}
|
||||
audit_logging: {retention_days: 365, immutable_retention_policy: true}
|
||||
|
||||
# 5. Network.
|
||||
networking:
|
||||
exposure: internal_only # internal_only | internet_facing
|
||||
topology: shared_vpc_hub_and_spoke # allowed values come from the Cloud provider's catalogue
|
||||
hybrid_connectivity: {enabled: true, type: cloud_interconnect, bandwidth_gbps: 10}
|
||||
egress_control: {model: centralized_appliance, appliance_vendor: palo_alto}
|
||||
cidr_allocations: ask
|
||||
|
||||
# 6. Identity, environments and operations.
|
||||
governance:
|
||||
identity_source_of_truth: entra_id
|
||||
pipeline_authentication: workload_identity_federation
|
||||
environment_tiers:
|
||||
- {name: development, archetype: corp_workload}
|
||||
- {name: staging, archetype: corp_workload}
|
||||
- {name: production, archetype: pci_regulated_workload}
|
||||
operations:
|
||||
operating_model: central_platform_broker
|
||||
support_hours: decide
|
||||
monitoring: decide
|
||||
integrations: []
|
||||
|
||||
# 7. How it will be delivered. The work breakdown structure is built from this.
|
||||
delivery:
|
||||
target_start_date: 2026-11-01
|
||||
window_weeks: 10 # essential when the customer has a deadline
|
||||
commercial_model: fixed_price_milestone
|
||||
iac_tooling: terraform
|
||||
ci_cd: github_actions
|
||||
team_size: ask
|
||||
working_days_per_week: 5
|
||||
|
||||
cost:
|
||||
monthly_budget: ask # essential for kind workload
|
||||
currency: GBP
|
||||
|
||||
# 8. How to run the agent.
|
||||
run:
|
||||
pre_approved: false
|
||||
allow_cloud_read: false
|
||||
overlay_skills: []
|
||||
terraform_modules: false
|
||||
```
|
||||
|
||||
### Sections the agent writes
|
||||
|
||||
Only in the completed copy. The supplied file never has to carry them, and if it does they are read as the last run's and replaced.
|
||||
|
||||
```yaml
|
||||
inferred_assumptions:
|
||||
- id: ASSUMP-01
|
||||
field: networking.hybrid_connectivity
|
||||
category: Network Telecom
|
||||
statement: Assumes the client manages the carrier circuit lead time for the 10Gbps interconnect.
|
||||
impact_if_invalid: Blocks the hybrid routing test; delays every activity after it.
|
||||
|
||||
discovery_gaps:
|
||||
- field: networking.cidr_allocations
|
||||
prompt: Specify IP subnet sizes for spoke workloads, for example /22 per environment.
|
||||
severity: HIGH # HIGH is an essential field; MEDIUM important; LOW optional
|
||||
|
||||
provenance: # where every settled value came from
|
||||
estate.regions.primary: supplied
|
||||
cost.monthly_budget: answered
|
||||
operations.monitoring: assumed
|
||||
|
||||
readiness: {supplied: 34, answered: 3, assumed: 6, unknown: 2}
|
||||
```
|
||||
|
||||
`provenance` is one of `supplied`, `answered`, `discovered` (read from the customer's cloud environment), `assumed`, `unknown`. Every `assumed` field has an entry in `inferred_assumptions`, and every `unknown` one in `discovery_gaps`.
|
||||
|
||||
### Status
|
||||
|
||||
The owner's example carries a status. The agent sets it in the completed copy from what has happened; a status written by the author is not trusted.
|
||||
|
||||
| Status | Means |
|
||||
|---|---|
|
||||
| `DRAFT` | Gaps remain, or the conversation is still filling them. |
|
||||
| `UNDER_REVIEW` | Complete, and waiting for the scoping Approval. |
|
||||
| `APPROVED` | The customer approved it; requirements and the later Phases run from it. |
|
||||
| `BASELINE_LOCKED` | Its Execution completed. This copy is the record a Change request starts from, and is never altered. |
|
||||
|
||||
### How the two were combined
|
||||
|
||||
| From the owner's example | How it is kept |
|
||||
|---|---|
|
||||
| The request, verbatim, inside the spec | `intent.statement`. It is also still saved as the Intent, as today. |
|
||||
| The customer's stated requirements, with ids | `intent.requirements`. The agent's requirements trace to these ids, so the Requirements chain starts at what the customer wrote. |
|
||||
| Stated exclusions | `intent.exclusions`. The design is checked for nothing built that is excluded. |
|
||||
| Engagement metadata and status | `metadata`, with the status set by the agent. |
|
||||
| Estate, compliance, networking and governance sections | Kept as written. |
|
||||
| Start date, delivery window, commercial model, tooling | `delivery`. The window becomes a limit the plan is checked against. |
|
||||
| Assumptions with their impact, and gaps with a prompt and a severity | The two agent-written sections, in place of the first draft's flat provenance table. Severity is the tier. |
|
||||
|
||||
| From the first draft | How it is kept |
|
||||
|---|---|
|
||||
| `ask`, `decide` and `not_applicable` | Unchanged. The example had no way to say "not known" other than omission. |
|
||||
| Tiers | Kept, as the severity of a gap. |
|
||||
| Two files | Kept. The author's file is untouched; the agent's sections go in the completed copy. |
|
||||
| Where each value came from | Kept, as `provenance`. |
|
||||
| Users, scale, availability target, budget | Kept under `workload` and `cost`, and essential only for a workload. |
|
||||
| Run options | Kept under `run`. |
|
||||
|
||||
### Where the two pull apart
|
||||
|
||||
Four tensions, each with the resolution proposed. They are the owner's to confirm.
|
||||
|
||||
1. **The example states solution choices; the agent keeps requirements free of them.** A hub-and-spoke topology, an interconnect, customer-managed keys and a named inspection appliance are design decisions. The rule that requirements name no products exists so the agents do not pick a design before the design Phase. It should not stop a customer who has already decided. Proposed: a choice the spec states is a constraint the design must honour and may not change, recorded as supplied. The agents still add none of their own.
|
||||
2. **The example is a landing zone; the agent has only designed workloads.** Every benchmark case is an application. A platform foundation has no users or response times, and has an organisation hierarchy, shared networking and environment archetypes the cases have never exercised. `engagement.kind` lets one spec serve both, but a platform foundation is a new kind of work for the agent and needs its own evaluation case before it is relied on.
|
||||
3. **The example uses one Cloud provider's vocabulary in general fields.** `shared_vpc_hub_and_spoke`, `cloud_interconnect` and `cmek` are Google Cloud's terms. Proposed: the general catalogue says a field such as `networking.topology` exists; the Cloud provider's bundle supplies its allowed values. A second provider then brings its own list and the spec's shape does not change.
|
||||
4. **"Engagement" is a word the glossary avoids.** The agent's term is Customer case. Proposed: `customer_case` stays the key the agent groups by, and `engagement_reference` carries the owner's own number untouched. If the owner would rather the agent's term became Engagement, that is a glossary change to make before this is built.
|
||||
|
||||
### Where the catalogue lives
|
||||
|
||||
A provider-neutral Skill holds the catalogue: each field's type, tier by engagement kind, the question to ask, and guidance for the default. A Cloud provider's bundle adds its allowed values and any fields of its own. An overlay Skill may raise a field's tier, so an organisation can make `compliance.encryption` essential. Code validates a spec against the catalogue; no model decides what is missing.
|
||||
|
||||
## How scoping runs
|
||||
|
||||
| What arrives | What happens |
|
||||
|---|---|
|
||||
| A complete spec | Validated. No gaps, no questions. With `pre_approved` the Phase passes without stopping. |
|
||||
| A partly filled spec, someone present | Gaps are asked for, essential first, a few related fields at a time, with a count of how many are settled. |
|
||||
| A partly filled spec, nobody present | Essential gaps refuse the start, with the list. Others are defaulted and recorded. |
|
||||
| Only a sentence | A model drafts a spec from it, marking everything it did not find as `ask`. It is then a partly filled spec. |
|
||||
|
||||
In conversation an answer is a value, "you decide", or "skip". A skipped field becomes `unknown` and a Customer question in the output. A free-text answer is turned into the field's value and read back before it is written. The customer may supply an edited file at any point in place of answering.
|
||||
|
||||
The Phase ends when the customer approves the completed spec.
|
||||
|
||||
### What it writes
|
||||
|
||||
Under the Customer case, beside the Intent:
|
||||
|
||||
```
|
||||
intent/
|
||||
001.md # the request, in the customer's words (as today)
|
||||
001.scope.yaml # the spec as supplied, unaltered; absent when none was
|
||||
001.scope.completed.yaml # the completed copy: status, assumptions, gaps, provenance, readiness
|
||||
```
|
||||
|
||||
The scoping Deliverable is a short document written from the completed copy: the scope statement, the information register as a table, and the readiness summary. The solution guide opens with that summary.
|
||||
|
||||
### What it changes downstream
|
||||
|
||||
- **The requirements analyst derives; it no longer asks or assumes.** Each requirement names the stated requirement (`REQ-03`) or the spec field it comes from, so the Requirements chain starts at what the customer wrote. A requirement from an `assumed` field says so.
|
||||
- **A stated choice binds the design.** A product, topology or setting the spec states is carried to the design as a constraint, and a design that departs from it is refused.
|
||||
- **Exclusions are checked.** Nothing in `intent.exclusions` may appear in the Target architecture.
|
||||
- **Customer questions** from the pillar checklists (built 2026-10-10) are joined by the `unknown` fields.
|
||||
- **A Change request may be an edited spec.** Code compares the two specs field by field, and that list is what changed. A sentence still works, as now.
|
||||
- **An evaluation case becomes a spec,** replacing the pattern-matched answers.
|
||||
|
||||
## The work breakdown structure
|
||||
|
||||
A new Deliverable in the packaging Phase, produced by a new Specialist after validation, summarised in the solution guide.
|
||||
|
||||
### Its data file
|
||||
|
||||
```json
|
||||
{
|
||||
"work_packages": [
|
||||
{"id": "WP-1", "name": "Network foundation", "parent": null,
|
||||
"components": ["C-2"], "elements": ["E-1", "E-2"],
|
||||
"scope": "The VPC, its subnet and outbound access.",
|
||||
"produces": ["a private network the services attach to"],
|
||||
"skills": ["network engineering"]}
|
||||
],
|
||||
"activities": [
|
||||
{"id": "A-1", "work_package": "WP-1", "name": "Create the VPC and subnet",
|
||||
"days": 1, "after": []},
|
||||
{"id": "A-2", "work_package": "WP-1", "name": "Configure outbound access",
|
||||
"days": 1, "after": ["A-1"]}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### What code checks and computes
|
||||
|
||||
- Every Element belongs to a work package, and every work package to a component or a parent. A design cannot be left out of the plan.
|
||||
- A work package's acceptance criteria are the Checks on its Elements. They are read from the Target architecture, not written again.
|
||||
- Activities form no cycle, and every `after` names an activity that exists.
|
||||
- **The plan is checked against the delivery window.** When `delivery.window_weeks` is set and the computed total exceeds it, the Deliverable says so and names the critical path; it does not shorten estimates to fit.
|
||||
- **An assumption can hold up the plan.** An entry in `inferred_assumptions` that an activity depends on, such as a carrier's lead time, appears as a dependency the customer owns, so its `impact_if_invalid` is visible on the timeline.
|
||||
- **Milestones.** With a `commercial_model` of fixed price by milestone, each top-level work package ends in a milestone.
|
||||
- **The schedule is computed, not written.** From each activity's days and what it comes after, code works out its earliest start and finish, the critical path and the total. With `delivery.team_size` it limits how many activities run at once.
|
||||
|
||||
### What it renders
|
||||
|
||||
Three views, all written by code from the data file, as the architecture diagram is:
|
||||
|
||||
- **WBS dictionary:** the hierarchy, each work package with its scope, what it produces, its Elements, its acceptance Checks and the skills it needs.
|
||||
- **Activity sequence list:** each activity with its days, what it follows, its start and finish day, and whether it is on the critical path.
|
||||
- **Timeline** on a relative axis, Day 0 to Day N:
|
||||
|
||||
```
|
||||
Activity Day 0 5 10 15 18
|
||||
A-1 Create the VPC and subnet ██
|
||||
A-2 Configure outbound access ██
|
||||
A-3 Provision the database █████
|
||||
A-4 Deploy the application ████
|
||||
A-5 Load balancer and access ███
|
||||
A-6 Monitoring and alerts ██
|
||||
A-7 Validation in test ████
|
||||
```
|
||||
|
||||
The chart format is to be settled when built: a Mermaid Gantt chart if it can show a relative day axis, and otherwise a drawn table like the one above.
|
||||
|
||||
### What it says about itself
|
||||
|
||||
Durations are a model's estimates. The Deliverable states their basis (working days, the team size from the spec) and that they are indicative. The timeline stays on a relative day axis; when `delivery.target_start_date` is set, the activity list also gives calendar dates.
|
||||
|
||||
## Order of work
|
||||
|
||||
| Stage | What | Usable on its own |
|
||||
|---|---|---|
|
||||
| 1 | The catalogue, validation, the gap list, the two files saved beside the Intent, the run options read from the spec | Yes: a complete spec starts a pre-approved Execution |
|
||||
| 2 | The conversation that fills gaps; scoping as a Phase with its Deliverable and Approval; a spec drafted from a sentence | Yes |
|
||||
| 3 | Requirements traced to spec fields; readiness in the guide | Yes |
|
||||
| 4 | An edited spec as a Change request | Yes |
|
||||
| 5 | The work breakdown structure | Yes, with stage 1 for `delivery` |
|
||||
| 6 | Evaluation cases as specs | Yes |
|
||||
|
||||
## Not in this spec
|
||||
|
||||
- Attaching documents (a statement of work, a diagram) as a source for the spec.
|
||||
- Reading the customer's cloud environment to fill `current_estate`. The `discovered` source is reserved for it.
|
||||
- Delivery activities beyond the technical build, such as security review, user acceptance and cutover.
|
||||
- Cost of the work, or named people.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. The four tensions under **Where the two pull apart**: are the proposed resolutions right?
|
||||
2. Which fields are essential for each engagement kind? The draft marks them; the owner's example suggests the delivery window and compliance frameworks matter more than budget for a foundation.
|
||||
3. In `pre_approved` runs an essential gap refuses the start. Is that right, or should it default and warn?
|
||||
4. Should the scoping Approval be skippable when the spec arrived complete and someone is present?
|
||||
5. Is a platform foundation in scope now, with its own evaluation case, or should scoping be built against workloads first?
|
||||
|
||||
## Comments
|
||||
|
||||
- 2026-10-11, the owner: a platform foundation is in scope now; an engagement name and a Customer case are equivalent references; the implementation will eventually be provider-agnostic. The other tensions and open questions were taken as proposed: a stated choice binds the design; an essential gap refuses a pre-approved start; the scoping Approval is not skipped. Recorded in ADR-0008.
|
||||
12
CONTEXT.md
12
CONTEXT.md
@@ -19,13 +19,21 @@ A caller-supplied grouping of Executions. A Session never feeds earlier outputs
|
||||
_Avoid_: conversation, context
|
||||
|
||||
**Customer case**:
|
||||
A caller-supplied grouping of the Executions run for one customer workload, identified by a short slug. Every Execution belongs to one; an Execution started without one belongs to `unassigned`. A Customer case ties each request to the Deliverables generated for it. It feeds one Execution's outputs into another only when the caller starts a Change request and names the Baseline.
|
||||
_Avoid_: engagement, project, account, tenant
|
||||
A caller-supplied grouping of the Executions run for one customer engagement, identified by a short slug. An engagement and a Customer case are the same thing: a reference such as `ENG-2026-UK-084` names the case `eng-2026-uk-084`. Every Execution belongs to one; an Execution started without one belongs to `unassigned`. A Customer case ties each request to the Deliverables generated for it. It feeds one Execution's outputs into another only when the caller starts a Change request and names the Baseline.
|
||||
_Avoid_: project, account, tenant
|
||||
|
||||
**Intent**:
|
||||
What the customer wants from a Customer case, in their own words and at a high level. Each Workflow request made in the case is a statement of it, and the statements are kept in order as its history. The Intent is the customer's; requirements are what the agents derive from it.
|
||||
_Avoid_: brief, ask, requirements (which are the derived Deliverable)
|
||||
|
||||
**Scope spec**:
|
||||
The structured statement of what the customer wants and the facts a design depends on, as a file. It may arrive complete or partly filled. It sits beside the Intent, which stays in the customer's words, and a value it states is the customer's decision.
|
||||
_Avoid_: brief, questionnaire, requirements (which are derived from it)
|
||||
|
||||
**Scoping**:
|
||||
Completing the scope spec, from what was supplied, what the customer answers and what the agent is left to decide, and having the customer approve it before requirements begin.
|
||||
_Avoid_: discovery (which reads a cloud environment), intake (which reads a message)
|
||||
|
||||
**Change request**:
|
||||
An Execution started to say how the customer's Intent has changed. It begins from a Baseline and delivers the full updated Deliverables with an account of what changed and why.
|
||||
_Avoid_: amendment, update run, iteration, re-run
|
||||
|
||||
@@ -18,6 +18,8 @@ One Orchestrator coordinates an Execution and is the only agent that talks to th
|
||||
|
||||
Each Phase needs the user's explicit Approval before work that depends on it starts. A pre-approved mode serves batch and evaluation runs.
|
||||
|
||||
An Execution may start from a **scope spec** ([ADR-0008](docs/adr/0008-scoping-starts-from-a-scope-spec.md)): a YAML file of what the customer wants and the facts a design depends on, complete or partly filled. Code reads it against a catalogue, refuses a field it does not know, and writes a completed copy beside the customer's request with what was assumed and what is still unknown. `python -m app.scope template platform_foundation` prints one to fill in, and `python -m app.scope check <file>` says what a spec lacks. Pass it as `scope` to `/generate` or in A2A metadata.
|
||||
|
||||
A **Change request** ([ADR-0006](docs/adr/0006-a-change-request-starts-from-a-baseline-execution.md)) starts from a completed Execution in the same Customer case, its Baseline, and delivers the full updated Deliverables with `changes.json`: what changed, what was only reworded, and anything changed that nobody asked for.
|
||||
|
||||
The design Phase ends with the **Target architecture**: one explicit graph of Elements, Relationships and Checks ([ADR-0005](docs/adr/0005-the-target-architecture-is-an-explicit-graph.md)). The user approves it before any infrastructure code is written. The diagram is drawn from it by code, the Terraform must account for every Check, and validation gives a verdict per Check. Requirements, components, products and recommendations each carry ids in a data file, so every requirement can be traced through to its verdict; a submission with a missing link is refused.
|
||||
|
||||
@@ -13,6 +13,7 @@ Your task message carries JSON context:
|
||||
- `working_copy`: what your own tools already hold from your earlier attempts, or `null`. Start from it; do not start again.
|
||||
- `baseline`: set only in a Change request, where the user has said how their requirements changed since an architecture was delivered. `deliverable` is that delivered version of your own Deliverable, and `changes` is what has changed so far: the requirements added, changed and removed, with the reason, and everything already revised from them. `null` otherwise.
|
||||
- `read_first`: references of your Skills that this task was judged to need, each as a Skill and a resource. Read these first with `load_skill_resource`. It is where to start, not a limit: read anything else you find you need.
|
||||
- `scope`: what the customer's scope spec says, when they supplied one. `stated` is theirs and binds your Deliverable; `to_decide` is yours to choose and record as an assumption; `to_ask` is unknown. Load the scoping Skill for the rules. Honour every stated choice, build nothing under `stated.intent.exclusions`, and name the customer's own requirement id when your work serves one.
|
||||
- `overlay_skills`: Skills that carry the user's organisation's standards or this engagement's constraints. Load each one and apply it. Where an overlay disagrees with the Cloud provider's general guidance, the overlay wins; where it disagrees with a requirement the user stated, follow the requirement and say so.
|
||||
- `pre_approved`: `true` when no user is available.
|
||||
|
||||
|
||||
@@ -11,7 +11,8 @@ You produce the **requirements** Deliverable: the Skill's requirements-discovery
|
||||
- Ask an open question only where no sensible default can be inferred from the request, such as the region, a compliance regime or a budget ceiling.
|
||||
- Name every ambiguity or contradiction, explain why the conflicting goals can't both hold, and ask the user to choose. When they delegate the choice, propose one and ask them to approve it.
|
||||
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
|
||||
- Name no products, services or architectures. This Deliverable describes the problem, not the solution.
|
||||
- Name no products, services or architectures of your own choosing. This Deliverable describes the problem, not the solution. A product or design choice the customer stated in `scope` is theirs: record it as a constraint, in their words.
|
||||
- When `scope` is set, derive the requirements from it. Cover every entry of `stated.intent.requirements` and name its id in the requirement that covers it. Use what is `stated` as given and ask nothing it already answers. Ask the questions under `to_ask`, the most severe first, when you may ask the user; otherwise assume a default for each and list it under `Assumptions` and `Open questions`. Choose a default for each field under `to_decide` and list it under `Assumptions` with what it affects.
|
||||
- Give every requirement an id and attach them all as `requirements.json`, in the format the requirements-chain Skill gives. Use the same ids in your content.
|
||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, a process people follow, or a property of a region or product that only its provider publishes (a carbon rating, a service level). For the last, the condition is the choice configuration can show, such as which region is set. Those are recorded and reported, and are not checked against the infrastructure code.
|
||||
|
||||
@@ -130,8 +130,9 @@ class SolutionArchitectExecutor(AgentExecutor):
|
||||
baseline=options.get("baseline"),
|
||||
overlay_skills=options.get("overlay_skills") or (),
|
||||
terraform_modules=options.get("terraform_modules"),
|
||||
scope=options.get("scope"),
|
||||
execution_id=task_id,
|
||||
pre_approved=bool(options.get("pre_approved", False)),
|
||||
pre_approved=options.get("pre_approved"),
|
||||
allow_cloud_read=options.get("allow_cloud_read"),
|
||||
)
|
||||
else:
|
||||
|
||||
76
app/agent.py
76
app/agent.py
@@ -46,6 +46,7 @@ from app.nodes.router import (
|
||||
)
|
||||
from app.providers import ProviderBundle, load_provider
|
||||
from app.recorder import DatabaseRecorder, ExecutionRecorder
|
||||
from app.scope import ESSENTIAL, Catalogue, Scope, read_scope, slug
|
||||
from app.skills.agent_skills import discover_skills, skill_version
|
||||
from app.states.execution_state import check_customer_case
|
||||
from app.states.ledger import Ledger
|
||||
@@ -106,6 +107,9 @@ class ExecutionEngine:
|
||||
assess_at_end: bool = False,
|
||||
modules_available: bool = False,
|
||||
default_modules: bool = False,
|
||||
scope_catalogue: Catalogue | None = None,
|
||||
scope_values: Mapping[str, Sequence[str]] | None = None,
|
||||
scope_skill: str | None = None,
|
||||
) -> None:
|
||||
required = {
|
||||
ORCHESTRATOR,
|
||||
@@ -115,6 +119,11 @@ class ExecutionEngine:
|
||||
if missing:
|
||||
raise ValueError(f"Missing roles: {', '.join(missing)}")
|
||||
self._max_revisions = max_revisions
|
||||
# The catalogue a scope spec is read against (ADR-0008); None when this
|
||||
# deployment's Cloud provider has none.
|
||||
self._scope_catalogue = scope_catalogue
|
||||
self._scope_values = dict(scope_values or {})
|
||||
self._scope_skill = scope_skill
|
||||
# Whether the Cloud provider approves any module sources, and whether an
|
||||
# Execution uses them when its caller does not say.
|
||||
self._modules_available = modules_available
|
||||
@@ -154,13 +163,14 @@ class ExecutionEngine:
|
||||
*,
|
||||
session_id: str | None = None,
|
||||
customer_case: str | None = None,
|
||||
pre_approved: bool = False,
|
||||
pre_approved: bool | None = None,
|
||||
allow_cloud_read: bool | None = None,
|
||||
execution_id: str | None = None,
|
||||
scripted_answers: Sequence[Sequence[str]] = (),
|
||||
baseline: str | None = None,
|
||||
overlay_skills: Sequence[str] = (),
|
||||
terraform_modules: bool | None = None,
|
||||
scope: str | Mapping[str, Any] | None = None,
|
||||
) -> Interaction:
|
||||
"""Start an Execution. `scripted_answers` are (regex, answer[, asker]) entries a
|
||||
pre-approved Execution uses to answer questions, for evaluation runs.
|
||||
@@ -180,7 +190,50 @@ class ExecutionEngine:
|
||||
does as its Baseline did and any other Execution takes the deployment's
|
||||
default, which is plain resources. Opting in is refused when the provider
|
||||
approves no sources.
|
||||
|
||||
`scope` is a scope spec (ADR-0008), as YAML text or already parsed. It may
|
||||
be complete or partly filled. The start is refused when it names a field
|
||||
the catalogue does not have or holds a value of the wrong type, and, for a
|
||||
pre-approved Execution, when an essential field is not settled: nobody is
|
||||
there to ask. What the spec states under `run` applies where the caller
|
||||
gave no value of their own. Its statement of intent is the request when
|
||||
none is given.
|
||||
"""
|
||||
read = self._read_scope(scope) if scope is not None else None
|
||||
if read is not None:
|
||||
run = read.values
|
||||
if pre_approved is None:
|
||||
pre_approved = bool(run.get("run.pre_approved", False))
|
||||
if allow_cloud_read is None and "run.allow_cloud_read" in run:
|
||||
allow_cloud_read = bool(run["run.allow_cloud_read"])
|
||||
if terraform_modules is None and "run.terraform_modules" in run:
|
||||
terraform_modules = bool(run["run.terraform_modules"])
|
||||
overlay_skills = [*overlay_skills, *run.get("run.overlay_skills", [])]
|
||||
if (
|
||||
customer_case
|
||||
and read.customer_case
|
||||
and slug(customer_case) != (read.customer_case)
|
||||
):
|
||||
raise ValueError(
|
||||
f"The scope spec is for {read.customer_case}, and this was "
|
||||
f"started for {customer_case}."
|
||||
)
|
||||
customer_case = customer_case or read.customer_case
|
||||
# The spec's statement is the customer's; it is what gets recorded.
|
||||
workflow_request = read.statement or workflow_request
|
||||
missing = read.gaps(ESSENTIAL)
|
||||
if pre_approved and missing:
|
||||
raise ValueError(
|
||||
"The scope spec leaves essential fields unsettled, and a "
|
||||
"pre-approved Execution has nobody to ask: "
|
||||
+ ", ".join(gap["field"] for gap in missing)
|
||||
+ ". Give each a value, or `decide` to leave it to the agent."
|
||||
)
|
||||
pre_approved = bool(pre_approved)
|
||||
if not workflow_request.strip():
|
||||
raise ValueError(
|
||||
"A request is needed: text, or a scope spec with a statement."
|
||||
)
|
||||
customer_case = check_customer_case(customer_case)
|
||||
execution_id = execution_id or str(uuid.uuid4())
|
||||
if await self._exists(execution_id):
|
||||
@@ -224,6 +277,8 @@ class ExecutionEngine:
|
||||
session_id=session_id,
|
||||
workflow_request=workflow_request,
|
||||
baseline=baseline or None,
|
||||
scope_supplied=scope if isinstance(scope, str) else None,
|
||||
scope_completed=read.completed(read.status(pre_approved)) if read else None,
|
||||
)
|
||||
initial: dict[str, Any] = {
|
||||
"execution_id": execution_id,
|
||||
@@ -236,6 +291,15 @@ class ExecutionEngine:
|
||||
# Overlay Skills this Execution loads, each with the version it had.
|
||||
"overlay_skills": {name: self._overlays[name] for name in chosen},
|
||||
"terraform_modules": terraform_modules,
|
||||
# What the customer's scope spec settles and leaves open; None without one.
|
||||
"scope": {
|
||||
"skill": self._scope_skill,
|
||||
"status": read.status(pre_approved),
|
||||
"readiness": read.readiness(),
|
||||
"context": read.for_context(),
|
||||
}
|
||||
if read
|
||||
else None,
|
||||
"risks": [],
|
||||
"pre_approved": pre_approved,
|
||||
"allow_cloud_read": allow_cloud_read,
|
||||
@@ -274,6 +338,13 @@ class ExecutionEngine:
|
||||
await self._graph.ainvoke(initial, self._config(execution_id))
|
||||
return await self.interaction(execution_id)
|
||||
|
||||
def _read_scope(self, supplied: str | Mapping[str, Any]) -> Scope:
|
||||
if self._scope_catalogue is None:
|
||||
raise ValueError("This deployment has no scope spec catalogue.")
|
||||
return read_scope(
|
||||
supplied, self._scope_catalogue, provider_values=self._scope_values
|
||||
)
|
||||
|
||||
async def resume(self, execution_id: str, reply: Mapping[str, Any]) -> Interaction:
|
||||
"""Answer the Execution's current prompt: {"reply"} | {"approval"} | {"permission"}."""
|
||||
current = await self.interaction(execution_id)
|
||||
@@ -517,6 +588,9 @@ async def open_assembly(
|
||||
else None,
|
||||
skill_threshold=settings.JEV_SKILL_THRESHOLD,
|
||||
assess_at_end=settings.JEV_ASSESS_EXECUTIONS,
|
||||
scope_catalogue=bundle.scope_catalogue() if bundle.scope_skill else None,
|
||||
scope_values=bundle.scope_values,
|
||||
scope_skill=bundle.scope_skill,
|
||||
modules_available=bool(bundle.terraform_module_namespaces),
|
||||
default_modules=settings.TERRAFORM_MODULES,
|
||||
overlays={
|
||||
|
||||
@@ -14,15 +14,20 @@ nothing was delivered for it. history.json is the index of the case.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from app.states.execution_state import case_dir
|
||||
|
||||
INTENT_DIR = "intent"
|
||||
HISTORY_FILENAME = "history.json"
|
||||
IN_PROGRESS = "in_progress"
|
||||
SCOPE_SUPPLIED = ".scope.yaml"
|
||||
SCOPE_COMPLETED = ".scope.completed.yaml"
|
||||
|
||||
|
||||
def intent_dir(output_root: Path, customer_case: str) -> Path:
|
||||
@@ -50,6 +55,8 @@ def record_intent(
|
||||
session_id: str,
|
||||
workflow_request: str,
|
||||
baseline: str | None = None,
|
||||
scope_supplied: str | None = None,
|
||||
scope_completed: Mapping[str, Any] | None = None,
|
||||
) -> int:
|
||||
"""Record a new statement of the customer's intent and return its version.
|
||||
|
||||
@@ -71,6 +78,19 @@ def record_intent(
|
||||
f"## What the customer wants\n\n{workflow_request.strip()}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
scope_files = None
|
||||
if scope_completed is not None:
|
||||
# The spec as supplied, unaltered, and the completed copy beside it.
|
||||
scope_files = {"completed": f"{INTENT_DIR}/{version:03d}{SCOPE_COMPLETED}"}
|
||||
(folder / f"{version:03d}{SCOPE_COMPLETED}").write_text(
|
||||
yaml.safe_dump(dict(scope_completed), sort_keys=False, allow_unicode=True),
|
||||
encoding="utf-8",
|
||||
)
|
||||
if scope_supplied:
|
||||
scope_files["supplied"] = f"{INTENT_DIR}/{version:03d}{SCOPE_SUPPLIED}"
|
||||
(folder / f"{version:03d}{SCOPE_SUPPLIED}").write_text(
|
||||
scope_supplied, encoding="utf-8"
|
||||
)
|
||||
history["intents"].append(
|
||||
{
|
||||
"version": version,
|
||||
@@ -82,6 +102,7 @@ def record_intent(
|
||||
"status": IN_PROGRESS,
|
||||
"execution": None,
|
||||
"baseline": baseline,
|
||||
"scope": scope_files,
|
||||
}
|
||||
)
|
||||
_write_history(folder, history)
|
||||
|
||||
@@ -62,8 +62,10 @@ def create_app(
|
||||
except ValueError:
|
||||
body = {}
|
||||
workflow_request = str(body.get("request") or "").strip()
|
||||
if not workflow_request:
|
||||
return JSONResponse({"error": "'request' is required"}, status_code=400)
|
||||
if not workflow_request and not body.get("scope"):
|
||||
return JSONResponse(
|
||||
{"error": "'request' or a 'scope' spec is required"}, status_code=400
|
||||
)
|
||||
try:
|
||||
interaction = await assembly.engine.start(
|
||||
workflow_request,
|
||||
@@ -72,6 +74,7 @@ def create_app(
|
||||
baseline=body.get("baseline"),
|
||||
overlay_skills=body.get("overlay_skills") or (),
|
||||
terraform_modules=body.get("terraform_modules"),
|
||||
scope=body.get("scope"),
|
||||
pre_approved=True,
|
||||
allow_cloud_read=bool(body.get("allow_cloud_read", False)),
|
||||
)
|
||||
|
||||
@@ -481,6 +481,10 @@ def make_execution_nodes(
|
||||
role = dataclasses.replace(
|
||||
role, skills=(*role.skills, *(n for n in named if n not in role.skills))
|
||||
)
|
||||
scope_skill = task_input.get("scope_skill")
|
||||
if scope_skill and scope_skill not in role.skills:
|
||||
# An Execution with a scope spec: every Specialist can read what it means.
|
||||
role = dataclasses.replace(role, skills=(*role.skills, scope_skill))
|
||||
opted_in = bool(task_input.get("terraform_modules"))
|
||||
MODULES_OPTED_IN.set(opted_in)
|
||||
if opted_in and role.modules_skill:
|
||||
@@ -908,6 +912,7 @@ def _specialist_input(
|
||||
"allow_cloud_read": state.get("allow_cloud_read"),
|
||||
# Whether this Execution opted into published Terraform modules.
|
||||
"terraform_modules": bool(state.get("terraform_modules")),
|
||||
"scope_skill": (state.get("scope") or {}).get("skill"),
|
||||
"working": dict(state.get("working", {}).get(kind.value, {})),
|
||||
# Whether the user has been shown a draft of this Deliverable. Answering some
|
||||
# other question about it, such as what to do after it stopped, is not that.
|
||||
@@ -937,6 +942,9 @@ def _specialist_input(
|
||||
# Skills carrying the organisation's standards or this engagement's
|
||||
# constraints. Where one disagrees with general guidance, it wins.
|
||||
"overlay_skills": sorted(state.get("overlay_skills") or {}),
|
||||
# What the customer's scope spec states, leaves to the agent, and
|
||||
# leaves unknown. None when no spec was supplied.
|
||||
"scope": (state.get("scope") or {}).get("context"),
|
||||
# False: infrastructure code is plain resources. True: a published
|
||||
# module may be used where the modules Skill says one fits.
|
||||
"terraform_modules": bool(state.get("terraform_modules")),
|
||||
@@ -1357,6 +1365,12 @@ def _write_outputs(
|
||||
"overlay_skills": dict(state.get("overlay_skills") or {}),
|
||||
# Whether its infrastructure code was allowed published modules.
|
||||
"terraform_modules": bool(state.get("terraform_modules")),
|
||||
# How much of the scope spec the customer supplied; None without a spec.
|
||||
"scope": {
|
||||
key: (state.get("scope") or {}).get(key) for key in ("status", "readiness")
|
||||
}
|
||||
if state.get("scope")
|
||||
else None,
|
||||
"status": status,
|
||||
"warnings": list(state.get("warnings", [])),
|
||||
# Flags a person should look at; nothing was done about them.
|
||||
|
||||
@@ -14,6 +14,7 @@ from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from app.scope import Catalogue, load_catalogue
|
||||
from app.skills.agent_skills import SkillFormatError, SkillMeta, discover_skills
|
||||
|
||||
TIERS = ("general", "cloud_read")
|
||||
@@ -85,6 +86,17 @@ class ProviderBundle:
|
||||
# The Skill approved lessons are published in (ADR-0004); None when the provider
|
||||
# has none. A Specialist is given it once it holds lessons for its Deliverable.
|
||||
lessons_skill: str | None = None
|
||||
# The Skill that holds the scope spec's catalogue (ADR-0008), and the values
|
||||
# this provider allows for the fields the catalogue leaves to a provider.
|
||||
scope_skill: str | None = None
|
||||
scope_values: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
|
||||
|
||||
def scope_catalogue(self) -> Catalogue:
|
||||
if not self.scope_skill:
|
||||
raise ProviderConfigError(f"The {self.name} bundle names no scope Skill.")
|
||||
return load_catalogue(
|
||||
self.skills[self.scope_skill].path / "assets" / "catalogue.yaml"
|
||||
)
|
||||
|
||||
def mcp_servers_for_role(
|
||||
self, role: str, include_preview: bool = True
|
||||
@@ -233,6 +245,15 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
||||
if lessons_skill is not None and lessons_skill not in skills:
|
||||
raise fail(f"'lessons_skill' references unknown skill {lessons_skill!r}")
|
||||
|
||||
scope = config.get("scope") or {}
|
||||
scope_skill = scope.get("skill") or None
|
||||
if scope_skill is not None and scope_skill not in skills:
|
||||
raise fail(f"'scope.skill' references unknown skill {scope_skill!r}")
|
||||
scope_values = {
|
||||
str(path): tuple(str(value) for value in values or ())
|
||||
for path, values in (scope.get("values") or {}).items()
|
||||
}
|
||||
|
||||
return ProviderBundle(
|
||||
name=name,
|
||||
skills_dirs=skills_dirs,
|
||||
@@ -256,4 +277,6 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
||||
or ()
|
||||
),
|
||||
lessons_skill=lessons_skill,
|
||||
scope_skill=scope_skill,
|
||||
scope_values=scope_values,
|
||||
)
|
||||
|
||||
460
app/scope.py
Normal file
460
app/scope.py
Normal file
@@ -0,0 +1,460 @@
|
||||
"""The scope spec: what the customer wants and the facts a design depends on, as a
|
||||
file (ADR-0008).
|
||||
|
||||
A spec may arrive complete, partly filled or not at all. This reads one against the
|
||||
catalogue of fields a Skill holds, and says exactly what is settled and what is not.
|
||||
No model is involved: whether a field is filled is a fact.
|
||||
|
||||
a value the customer states it supplied
|
||||
ask, or absent not known yet a gap, with its question
|
||||
decide the customer leaves the choice to the agent an assumption
|
||||
not_applicable does not apply nothing asked or assumed
|
||||
|
||||
The file supplied is never altered. `Scope.completed()` is the copy written beside
|
||||
it, with the sections the agent writes: assumptions, gaps, where each value came
|
||||
from, and a count of each.
|
||||
|
||||
.venv/bin/python -m app.scope check <spec.yaml> [provider]
|
||||
.venv/bin/python -m app.scope template [kind]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import re
|
||||
import sys
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
ASK, DECIDE, NOT_APPLICABLE = "ask", "decide", "not_applicable"
|
||||
MARKERS = (ASK, DECIDE, NOT_APPLICABLE)
|
||||
SUPPLIED, ASSUMED, UNKNOWN = "supplied", "assumed", "unknown"
|
||||
ESSENTIAL, IMPORTANT, OPTIONAL = "essential", "important", "optional"
|
||||
_SEVERITY = {ESSENTIAL: "HIGH", IMPORTANT: "MEDIUM", OPTIONAL: "LOW"}
|
||||
DRAFT, UNDER_REVIEW, APPROVED, LOCKED = (
|
||||
"DRAFT",
|
||||
"UNDER_REVIEW",
|
||||
"APPROVED",
|
||||
"BASELINE_LOCKED",
|
||||
)
|
||||
KIND_FIELD = "engagement.kind"
|
||||
CASE_FIELD = "metadata.customer_case"
|
||||
STATEMENT_FIELD = "intent.statement"
|
||||
# What the agent writes into the completed copy. A supplied file may still carry
|
||||
# them from an earlier run; they are read as that run's and replaced.
|
||||
_AGENT_WRITTEN = (
|
||||
"schema_version",
|
||||
"inferred_assumptions",
|
||||
"discovery_gaps",
|
||||
"provenance",
|
||||
"readiness",
|
||||
)
|
||||
_AGENT_FIELDS = ("metadata.status",)
|
||||
_SLUG = re.compile(r"[^a-z0-9]+")
|
||||
|
||||
|
||||
class ScopeError(ValueError):
|
||||
"""A scope spec that cannot be used, with every problem found in it."""
|
||||
|
||||
def __init__(self, problems: Sequence[str]) -> None:
|
||||
self.problems = list(problems)
|
||||
super().__init__("scope spec: " + "; ".join(self.problems))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FieldSpec:
|
||||
path: str
|
||||
type: str = "string"
|
||||
tier: Any = OPTIONAL # a tier, or {kind: tier}
|
||||
question: str = ""
|
||||
default: Any = None
|
||||
affects: str = ""
|
||||
values: tuple[str, ...] = ()
|
||||
provider_values: bool = False
|
||||
keys: tuple[str, ...] = ()
|
||||
optional_keys: tuple[str, ...] = ()
|
||||
aliases: tuple[str, ...] = ()
|
||||
|
||||
def tier_for(self, kind: str | None) -> str:
|
||||
if isinstance(self.tier, Mapping):
|
||||
return str(self.tier.get(kind, OPTIONAL))
|
||||
return str(self.tier)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Catalogue:
|
||||
schema_version: int
|
||||
kinds: tuple[str, ...]
|
||||
fields: Mapping[str, FieldSpec]
|
||||
|
||||
def aliases(self) -> dict[str, str]:
|
||||
return {
|
||||
alias: spec.path for spec in self.fields.values() for alias in spec.aliases
|
||||
}
|
||||
|
||||
|
||||
def load_catalogue(path: Path) -> Catalogue:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
|
||||
fields = {}
|
||||
for name, raw in (data.get("fields") or {}).items():
|
||||
raw = raw or {}
|
||||
fields[name] = FieldSpec(
|
||||
path=name,
|
||||
type=str(raw.get("type", "string")),
|
||||
tier=raw.get("tier", OPTIONAL),
|
||||
question=" ".join(str(raw.get("question", "")).split()),
|
||||
default=raw.get("default"),
|
||||
affects=str(raw.get("affects", "")),
|
||||
values=tuple(str(v) for v in raw.get("values") or ()),
|
||||
provider_values=bool(raw.get("provider_values")),
|
||||
keys=tuple(raw.get("keys") or ()),
|
||||
optional_keys=tuple(raw.get("optional_keys") or ()),
|
||||
aliases=tuple(raw.get("aliases") or ()),
|
||||
)
|
||||
return Catalogue(
|
||||
schema_version=int(data.get("schema_version", 1)),
|
||||
kinds=tuple(data.get("kinds") or ()),
|
||||
fields=fields,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Scope:
|
||||
"""A scope spec read against the catalogue."""
|
||||
|
||||
catalogue: Catalogue
|
||||
kind: str | None
|
||||
values: dict[str, Any] = field(default_factory=dict) # field -> stated value
|
||||
states: dict[str, str] = field(default_factory=dict) # field -> marker
|
||||
supplied_text: str = ""
|
||||
|
||||
@property
|
||||
def customer_case(self) -> str | None:
|
||||
value = self.values.get(CASE_FIELD)
|
||||
return str(value) if value else None
|
||||
|
||||
@property
|
||||
def statement(self) -> str:
|
||||
return str(self.values.get(STATEMENT_FIELD) or "").strip()
|
||||
|
||||
def tier(self, path: str) -> str:
|
||||
return self.catalogue.fields[path].tier_for(self.kind)
|
||||
|
||||
def gaps(self, *tiers: str) -> list[dict[str, str]]:
|
||||
"""What is not known, most pressing first: each with the question to ask."""
|
||||
wanted = tiers or (ESSENTIAL, IMPORTANT, OPTIONAL)
|
||||
found = [
|
||||
{
|
||||
"field": path,
|
||||
"prompt": self.catalogue.fields[path].question or f"What is {path}?",
|
||||
"severity": _SEVERITY[self.tier(path)],
|
||||
}
|
||||
for path, state in self.states.items()
|
||||
if state == ASK and self.tier(path) in wanted
|
||||
]
|
||||
order = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
|
||||
return sorted(found, key=lambda gap: order[gap["severity"]])
|
||||
|
||||
def assumptions(self) -> list[dict[str, str]]:
|
||||
"""What the customer left to the agent, each with the default to take and
|
||||
what a wrong choice would change."""
|
||||
found: list[dict[str, str]] = []
|
||||
for path, state in self.states.items():
|
||||
if state != DECIDE:
|
||||
continue
|
||||
spec = self.catalogue.fields[path]
|
||||
found.append(
|
||||
{
|
||||
"id": f"ASSUMP-{len(found) + 1:02d}",
|
||||
"field": path,
|
||||
"statement": f"The customer left {path} to the agent. "
|
||||
f"Default: {spec.default if spec.default is not None else 'the agent chooses'}.",
|
||||
"impact_if_invalid": f"Changes {spec.affects}."
|
||||
if spec.affects
|
||||
else "Local to this field.",
|
||||
}
|
||||
)
|
||||
return found
|
||||
|
||||
def readiness(self) -> dict[str, int]:
|
||||
states = list(self.states.values())
|
||||
return {
|
||||
SUPPLIED: len(self.values),
|
||||
ASSUMED: states.count(DECIDE),
|
||||
UNKNOWN: states.count(ASK),
|
||||
NOT_APPLICABLE: states.count(NOT_APPLICABLE),
|
||||
}
|
||||
|
||||
def status(self, pre_approved: bool = False) -> str:
|
||||
"""Where the spec stands. With nobody to ask, what is important and unknown
|
||||
is defaulted and listed, so only an essential gap keeps it a draft."""
|
||||
if pre_approved:
|
||||
return DRAFT if self.gaps(ESSENTIAL) else APPROVED
|
||||
return DRAFT if self.gaps(ESSENTIAL, IMPORTANT) else UNDER_REVIEW
|
||||
|
||||
def completed(self, status: str) -> dict[str, Any]:
|
||||
"""The completed copy: every field of the catalogue, and the agent's
|
||||
sections."""
|
||||
document: dict[str, Any] = {"schema_version": self.catalogue.schema_version}
|
||||
for path in self.catalogue.fields:
|
||||
value = self.values.get(path, self.states.get(path, ASK))
|
||||
_put(document, path, value)
|
||||
_put(document, "metadata.status", status)
|
||||
provenance = {path: SUPPLIED for path in self.values}
|
||||
provenance |= {
|
||||
path: {DECIDE: ASSUMED, ASK: UNKNOWN, NOT_APPLICABLE: NOT_APPLICABLE}[state]
|
||||
for path, state in self.states.items()
|
||||
}
|
||||
document["inferred_assumptions"] = self.assumptions()
|
||||
document["discovery_gaps"] = self.gaps()
|
||||
document["provenance"] = dict(sorted(provenance.items()))
|
||||
document["readiness"] = self.readiness()
|
||||
return document
|
||||
|
||||
def for_context(self) -> dict[str, Any]:
|
||||
"""What a Specialist is shown: what the customer stated, what is theirs to
|
||||
choose, and what is unknown."""
|
||||
stated: dict[str, Any] = {}
|
||||
for path, value in self.values.items():
|
||||
if not path.startswith(("run.", "metadata.")):
|
||||
_put(stated, path, value)
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"stated": stated,
|
||||
"to_decide": [
|
||||
{
|
||||
"field": path,
|
||||
"default": self.catalogue.fields[path].default,
|
||||
"affects": self.catalogue.fields[path].affects,
|
||||
}
|
||||
for path, state in self.states.items()
|
||||
if state == DECIDE and not path.startswith("run.")
|
||||
],
|
||||
"to_ask": [
|
||||
gap for gap in self.gaps() if not gap["field"].startswith("run.")
|
||||
],
|
||||
"not_applicable": sorted(
|
||||
path for path, state in self.states.items() if state == NOT_APPLICABLE
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _put(document: dict[str, Any], path: str, value: Any) -> None:
|
||||
*parents, last = path.split(".")
|
||||
node = document
|
||||
for part in parents:
|
||||
node = node.setdefault(part, {})
|
||||
node[last] = value
|
||||
|
||||
|
||||
def slug(reference: str) -> str:
|
||||
"""A Customer case id from an engagement reference: `ENG-2026-UK-084` and
|
||||
`eng-2026-uk-084` name the same case."""
|
||||
return _SLUG.sub("-", reference.strip().lower()).strip("-")[:64]
|
||||
|
||||
|
||||
def read_scope(
|
||||
supplied: str | Mapping[str, Any],
|
||||
catalogue: Catalogue,
|
||||
*,
|
||||
provider_values: Mapping[str, Sequence[str]] | None = None,
|
||||
) -> Scope:
|
||||
"""Read a scope spec. Raises ScopeError naming every problem: YAML that does not
|
||||
parse, a field the catalogue does not have, a value of the wrong type."""
|
||||
text = supplied if isinstance(supplied, str) else yaml.safe_dump(dict(supplied))
|
||||
try:
|
||||
data = yaml.safe_load(text) if isinstance(supplied, str) else dict(supplied)
|
||||
except yaml.YAMLError as exc:
|
||||
raise ScopeError([f"not valid YAML ({exc})"]) from exc
|
||||
if not isinstance(data, Mapping):
|
||||
raise ScopeError(["the spec must be a mapping of sections"])
|
||||
|
||||
problems: list[str] = []
|
||||
leaves: dict[str, Any] = {}
|
||||
aliases = catalogue.aliases()
|
||||
prefixes = {
|
||||
".".join(path.split(".")[:depth])
|
||||
for path in catalogue.fields
|
||||
for depth in range(1, len(path.split(".")))
|
||||
}
|
||||
|
||||
def walk(node: Any, path: str) -> None:
|
||||
target = aliases.get(path, path)
|
||||
if path in _AGENT_FIELDS or path in _AGENT_WRITTEN:
|
||||
return
|
||||
if target in catalogue.fields:
|
||||
if target in leaves and leaves[target] != node and path != target:
|
||||
problems.append(f"{path} and {target} name the same thing and differ")
|
||||
leaves[target] = node
|
||||
return
|
||||
if path in prefixes:
|
||||
if isinstance(node, Mapping):
|
||||
for key, value in node.items():
|
||||
walk(value, f"{path}.{key}")
|
||||
elif node in MARKERS or node is None:
|
||||
# A whole section marked at once applies to every field in it.
|
||||
for name in catalogue.fields:
|
||||
if name.startswith(path + "."):
|
||||
leaves.setdefault(name, node)
|
||||
else:
|
||||
problems.append(f"{path} is a section, not a value")
|
||||
return
|
||||
problems.append(f"{path} is not a field of the scope spec")
|
||||
|
||||
for key, value in data.items():
|
||||
walk(value, str(key))
|
||||
|
||||
kind = leaves.get(KIND_FIELD)
|
||||
kind = str(kind) if kind in catalogue.kinds else None
|
||||
scope = Scope(catalogue, kind, supplied_text=text)
|
||||
extra = dict(provider_values or {})
|
||||
for path, spec in catalogue.fields.items():
|
||||
value = leaves.get(path)
|
||||
if value is None or value == ASK:
|
||||
scope.states[path] = ASK
|
||||
elif value in (DECIDE, NOT_APPLICABLE):
|
||||
scope.states[path] = value
|
||||
else:
|
||||
fault = _fault(spec, value, extra.get(path, ()))
|
||||
if fault:
|
||||
problems.append(f"{path}: {fault}")
|
||||
else:
|
||||
scope.values[path] = _plain(value)
|
||||
case = scope.values.get(CASE_FIELD)
|
||||
if case is not None:
|
||||
scope.values[CASE_FIELD] = slug(str(case))
|
||||
if not scope.values[CASE_FIELD]:
|
||||
problems.append(f"{CASE_FIELD}: needs letters or digits")
|
||||
if problems:
|
||||
raise ScopeError(problems)
|
||||
return scope
|
||||
|
||||
|
||||
def _plain(value: Any) -> Any:
|
||||
"""A value as plain data: YAML reads a date as a date, and state must be JSON."""
|
||||
if isinstance(value, (dt.date, dt.datetime)):
|
||||
return value.isoformat()
|
||||
if isinstance(value, Mapping):
|
||||
return {str(k): _plain(v) for k, v in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_plain(item) for item in value]
|
||||
return value
|
||||
|
||||
|
||||
def _fault(spec: FieldSpec, value: Any, provider_values: Sequence[str]) -> str:
|
||||
"""Why a value does not fit its field; "" when it does."""
|
||||
kind = spec.type
|
||||
if kind in ("string", "text"):
|
||||
return "" if isinstance(value, (str, int, float)) else "must be text"
|
||||
if kind == "integer":
|
||||
ok = isinstance(value, int) and not isinstance(value, bool)
|
||||
return "" if ok else "must be a whole number"
|
||||
if kind == "number":
|
||||
ok = isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
return "" if ok else "must be a number"
|
||||
if kind == "boolean":
|
||||
return "" if isinstance(value, bool) else "must be true or false"
|
||||
if kind == "date":
|
||||
if isinstance(value, (dt.date, dt.datetime)):
|
||||
return ""
|
||||
try:
|
||||
dt.date.fromisoformat(str(value))
|
||||
except ValueError:
|
||||
return "must be a date, as YYYY-MM-DD"
|
||||
return ""
|
||||
if kind == "enum":
|
||||
allowed = (*spec.values, *provider_values)
|
||||
if not allowed or str(value) in allowed:
|
||||
return ""
|
||||
return f"must be one of {', '.join(allowed)}"
|
||||
if kind == "list":
|
||||
return "" if isinstance(value, list) else "must be a list"
|
||||
if kind == "records":
|
||||
if not isinstance(value, list):
|
||||
return "must be a list of entries"
|
||||
known = {*spec.keys, *spec.optional_keys}
|
||||
for number, entry in enumerate(value, start=1):
|
||||
if not isinstance(entry, Mapping):
|
||||
return f"entry {number} must have named fields"
|
||||
missing = [key for key in spec.keys if not entry.get(key)]
|
||||
if missing:
|
||||
return f"entry {number} needs {', '.join(missing)}"
|
||||
unknown = sorted(set(map(str, entry)) - known)
|
||||
if unknown:
|
||||
return f"entry {number} has unknown fields {', '.join(unknown)}"
|
||||
return ""
|
||||
return f"has an unknown type {kind} in the catalogue"
|
||||
|
||||
|
||||
def render_template(catalogue: Catalogue, kind: str | None = None) -> str:
|
||||
"""A scope spec with every field set to ask, to fill in."""
|
||||
document: dict[str, Any] = {}
|
||||
for path, spec in catalogue.fields.items():
|
||||
if path == KIND_FIELD and kind:
|
||||
_put(document, path, kind)
|
||||
elif spec.tier_for(kind) != OPTIONAL or kind is None:
|
||||
_put(document, path, ASK)
|
||||
header = (
|
||||
"# Scope spec. Give each field a value, or: ask (not known yet), decide\n"
|
||||
"# (you choose), not_applicable. A field left out is read as ask.\n"
|
||||
)
|
||||
return header + yaml.safe_dump(document, sort_keys=False, allow_unicode=True)
|
||||
|
||||
|
||||
def render_reading(scope: Scope) -> str:
|
||||
"""What a spec settles and what it leaves open, for a person."""
|
||||
counts = scope.readiness()
|
||||
lines = [
|
||||
f"Engagement kind: {scope.kind or 'not stated'}",
|
||||
(
|
||||
f"{counts[SUPPLIED]} supplied, {counts[ASSUMED]} left to the agent, "
|
||||
f"{counts[UNKNOWN]} unknown, {counts[NOT_APPLICABLE]} not applicable."
|
||||
),
|
||||
]
|
||||
for severity, title in (
|
||||
("HIGH", "Essential, needed before work can start"),
|
||||
("MEDIUM", "Important, asked when someone is present"),
|
||||
):
|
||||
gaps = [gap for gap in scope.gaps() if gap["severity"] == severity]
|
||||
if gaps:
|
||||
lines += ["", f"{title}:"]
|
||||
lines += [f" {gap['field']}: {gap['prompt']}" for gap in gaps]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(arguments: Sequence[str]) -> int:
|
||||
from app.config import get_settings
|
||||
from app.providers import load_provider
|
||||
|
||||
command, rest = (arguments[0], arguments[1:]) if arguments else ("", [])
|
||||
if command == "template":
|
||||
bundle = load_provider(get_settings().CLOUD_PROVIDER)
|
||||
print(render_template(bundle.scope_catalogue(), rest[0] if rest else None))
|
||||
return 0
|
||||
if command == "check" and rest:
|
||||
bundle = load_provider(
|
||||
rest[1] if len(rest) > 1 else get_settings().CLOUD_PROVIDER
|
||||
)
|
||||
try:
|
||||
scope = read_scope(
|
||||
Path(rest[0]).read_text(encoding="utf-8"),
|
||||
bundle.scope_catalogue(),
|
||||
provider_values=bundle.scope_values,
|
||||
)
|
||||
except ScopeError as refused:
|
||||
print("This spec cannot be used:")
|
||||
for problem in refused.problems:
|
||||
print(f" {problem}")
|
||||
return 1
|
||||
print(render_reading(scope))
|
||||
return 0
|
||||
print(__doc__, file=sys.stderr)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -59,6 +59,9 @@ class ExecutionState(TypedDict, total=False):
|
||||
risks: list[dict[str, Any]] # flags on what the user wrote, for a person to see
|
||||
overlay_skills: dict[str, str] # overlay Skills this Execution loads -> version
|
||||
terraform_modules: bool # opted into published Terraform modules
|
||||
# The customer's scope spec as read at the start: its Skill, status, readiness
|
||||
# and what a Specialist is shown of it. None when no spec was supplied.
|
||||
scope: dict[str, Any] | None
|
||||
baseline: dict[str, Any] | None # a Change request's Baseline, read from its files
|
||||
pre_approved: bool
|
||||
allow_cloud_read: bool | None
|
||||
|
||||
50
docs/adr/0008-scoping-starts-from-a-scope-spec.md
Normal file
50
docs/adr/0008-scoping-starts-from-a-scope-spec.md
Normal file
@@ -0,0 +1,50 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Scoping starts from a scope spec
|
||||
|
||||
An Execution began from the customer's request, a sentence or two saved word for word, and the requirements analyst filled everything else with assumptions. Most of what went wrong downstream traced to what nobody had stated, and nothing told the customer how much of their design rested on assumption. We decide that an engagement is scoped from a **scope spec**: a structured file that states what the customer wants and the facts a design depends on, which may arrive complete, partly filled or not at all, and which the customer approves before requirements begin.
|
||||
|
||||
The Intent stays the customer's own words. The scope spec sits beside it as the agreed, structured form.
|
||||
|
||||
## What the spec is
|
||||
|
||||
- **A file of fields from a catalogue.** A Skill holds the catalogue: each field's type, when it must be settled, the question to ask for it, and guidance for a default. The catalogue names no Cloud provider. A provider's bundle supplies the allowed values for the fields it leaves to a provider, in that provider's terms.
|
||||
- **Every field says one of four things.** A value is the customer's statement. `ask`, or the field left out, means not known. `decide` leaves the choice to the agent, recorded as an assumption. `not_applicable` asks and assumes nothing.
|
||||
- **Whether a field is settled is read by code.** No model decides what is missing.
|
||||
- **Fields are essential, important or optional,** and which is which depends on the kind of engagement: a workload, a platform foundation or a migration.
|
||||
- **It carries the customer's own words and requirements:** the statement verbatim, the requirements they stated with their own ids, and what they excluded.
|
||||
- **It carries how to run the agent,** so one file starts an Execution.
|
||||
|
||||
## Decisions
|
||||
|
||||
Accepted by the owner on 2026-10-11, with each of these:
|
||||
|
||||
1. **The file supplied is never altered.** The agent writes a completed copy beside it, with what it assumed, what is still unknown, where each value came from, and a status.
|
||||
2. **A choice the customer states binds the design.** A product, a topology or a setting written in the spec is a constraint every Specialist honours. The rule that requirements name no products is about what the agents introduce, not what the customer has already decided.
|
||||
3. **A platform foundation is in scope,** alongside the application workloads the agent was built on.
|
||||
4. **An engagement and a Customer case are the same thing.** A spec may name either; an engagement reference is turned into the case's id.
|
||||
5. **The spec is provider-agnostic in shape.** Provider vocabulary enters only as values a provider's bundle allows.
|
||||
6. **An essential field that is not settled refuses a start that has nobody to ask.** With someone present it is asked for.
|
||||
7. **Scoping becomes a Phase the customer approves,** before requirements discovery.
|
||||
8. **Where the spec and a prose request disagree, the spec wins.**
|
||||
|
||||
## Considered Options
|
||||
|
||||
- **Judge the free-text request against a list of topics** with a model or Jev: the first proposal. Rejected once a spec was on the table: a field that is filled or not is a fact, and a judgement about prose is not.
|
||||
- **One file the agent completes in place:** rejected. An author must be able to see that what they wrote is what was used.
|
||||
- **Keep product names out of the spec, as out of requirements:** rejected. A customer who has decided on an interconnect or a named appliance has stated a constraint; refusing to record it would lose it.
|
||||
- **A separate spec for platform foundations:** rejected. One catalogue with tiers by kind serves both, and a migration is usually some of each.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The Requirements chain gains a start: a requirement traces to the customer's own requirement id or to a spec field.
|
||||
- The requirements analyst stops being where questions are asked and assumptions made. That moves to scoping.
|
||||
- A Change request may be an edited spec, compared field by field in code.
|
||||
- An evaluation case may be a spec, in place of answers matched by pattern.
|
||||
- The glossary gains **Scope spec** and **Scoping**, and Customer case says that an engagement is the same thing.
|
||||
- A platform foundation is new work for the agents. It needs its own evaluation case before it is relied on.
|
||||
- The spec holds client names and details. Like lesson candidates it is client data: kept with the Customer case, never in the repository, apart from the fictional benchmark specs.
|
||||
|
||||
This is built in stages; `.agents/plans/iter-09-scoping/` says which are done.
|
||||
@@ -545,5 +545,21 @@
|
||||
]
|
||||
],
|
||||
"terraform_modules": true
|
||||
},
|
||||
{
|
||||
"id": "case-08-landing-zone",
|
||||
"name": "A regulated platform foundation, started from a scope spec",
|
||||
"scope": "landing-zone.yaml",
|
||||
"required_products": [
|
||||
"Cloud Interconnect",
|
||||
"Shared VPC",
|
||||
"Cloud KMS"
|
||||
],
|
||||
"scripted_answers": [
|
||||
[
|
||||
".*",
|
||||
"Proceed on the scope spec as supplied. Where it says decide, choose a sensible default and record it as an assumption."
|
||||
]
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
117
eval/datasets/specs/landing-zone.yaml
Normal file
117
eval/datasets/specs/landing-zone.yaml
Normal file
@@ -0,0 +1,117 @@
|
||||
# A benchmark scope spec: a regulated platform foundation. A fictional customer.
|
||||
schema_version: 1
|
||||
|
||||
metadata:
|
||||
engagement_id: case-08-landing-zone
|
||||
client_name: Example Payments UK
|
||||
author: Evaluation
|
||||
revision: "1.0.0"
|
||||
|
||||
engagement:
|
||||
kind: platform_foundation
|
||||
|
||||
intent:
|
||||
statement: |
|
||||
We are a UK-regulated payment platform expanding into Google Cloud. We need an
|
||||
enterprise-grade landing zone ready in under 10 weeks to support our core payments
|
||||
engine. All payment processing must comply with DORA and PCI-DSS v4.0. We have
|
||||
existing on-premises systems that require private, low-latency connectivity over a
|
||||
10Gbps dedicated interconnect. We use Entra ID for corporate single sign-on and
|
||||
GitHub Actions for our delivery pipelines. Security requires that all egress
|
||||
traffic flows through centralised inspection appliances before touching the public
|
||||
internet, and customer data must be encrypted with keys we control.
|
||||
requirements:
|
||||
- id: REQ-01
|
||||
category: Timeline
|
||||
statement: Platform foundation must be ready for workload ingestion within 10 weeks.
|
||||
- id: REQ-02
|
||||
category: Compliance
|
||||
statement: Must achieve DORA resiliency alignment and PCI-DSS v4.0 perimeter isolation.
|
||||
- id: REQ-03
|
||||
category: Networking
|
||||
statement: 10Gbps on-premises hybrid connectivity; centralised inspection appliances for egress.
|
||||
- id: REQ-04
|
||||
category: Identity and security
|
||||
statement: Workload identity federation with GitHub Actions; Entra ID single sign-on; customer-managed encryption keys.
|
||||
exclusions:
|
||||
- Application container deployment; workloads are managed by application squads.
|
||||
- An active-active secondary region in phase 1; warm standby is acceptable.
|
||||
success_criteria: decide
|
||||
|
||||
estate:
|
||||
cloud_provider: gcp
|
||||
mode: greenfield
|
||||
regions:
|
||||
primary: europe-west2
|
||||
secondary: europe-west1
|
||||
resiliency_tier: warm_standby
|
||||
data_residency: United Kingdom and European Union
|
||||
existing_systems: On-premises payment systems that stay in place and connect over the interconnect.
|
||||
|
||||
workload:
|
||||
users: not_applicable
|
||||
availability:
|
||||
target: decide
|
||||
hours: always
|
||||
recovery_time: decide
|
||||
recovery_point: decide
|
||||
|
||||
compliance:
|
||||
industry: financial_services
|
||||
frameworks: [DORA, ISO27001, PCI_DSS_v4]
|
||||
data_classification: confidential
|
||||
encryption:
|
||||
type: cmek
|
||||
key_rotation_days: 90
|
||||
audit_logging:
|
||||
retention_days: 365
|
||||
immutable_retention_policy: true
|
||||
|
||||
networking:
|
||||
exposure: internal_only
|
||||
topology: shared_vpc_hub_and_spoke
|
||||
hybrid_connectivity:
|
||||
enabled: true
|
||||
type: cloud_interconnect
|
||||
bandwidth_gbps: 10
|
||||
egress_control:
|
||||
model: centralized_appliance
|
||||
appliance_vendor: decide
|
||||
cidr_allocations: decide
|
||||
|
||||
governance:
|
||||
identity_source_of_truth: Entra ID
|
||||
pipeline_authentication: workload identity federation
|
||||
environment_tiers:
|
||||
- {name: development, archetype: corp_workload}
|
||||
- {name: staging, archetype: corp_workload}
|
||||
- {name: production, archetype: pci_regulated_workload}
|
||||
|
||||
operations:
|
||||
operating_model: central platform team brokering access for application squads
|
||||
support_hours: always
|
||||
monitoring: decide
|
||||
change_process: decide
|
||||
|
||||
integrations: decide
|
||||
|
||||
delivery:
|
||||
window_weeks: 10
|
||||
commercial_model: fixed_price_milestone
|
||||
iac_tooling: terraform
|
||||
ci_cd: GitHub Actions
|
||||
team_size: 3
|
||||
working_days_per_week: 5
|
||||
|
||||
cost:
|
||||
monthly_budget: decide
|
||||
currency: GBP
|
||||
|
||||
constraints:
|
||||
products_required: decide
|
||||
products_forbidden: decide
|
||||
|
||||
run:
|
||||
pre_approved: true
|
||||
allow_cloud_read: false
|
||||
terraform_modules: false
|
||||
@@ -18,6 +18,8 @@ from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from app.adk.runtime import ToolResult
|
||||
from app.agent import Interaction
|
||||
from app.tools.assessment import QUALITY_BAR, QUALITY_RUBRIC, checklist_question
|
||||
@@ -73,6 +75,8 @@ class CaseSpec:
|
||||
overlay_skills: tuple[str, ...] = ()
|
||||
# Whether the case's Executions opt into published Terraform modules.
|
||||
terraform_modules: bool = False
|
||||
# A scope spec (ADR-0008), as YAML text, that the case's Execution starts from.
|
||||
scope: str = ""
|
||||
# What the infrastructure code must and must not contain, as regular
|
||||
# expressions over all of it: how a case checks that standards from an overlay
|
||||
# Skill reached the code. Each is {"name", "require"} or {"name", "forbid"}.
|
||||
@@ -106,12 +110,27 @@ class CaseResult:
|
||||
return self.score >= PASS_THRESHOLD
|
||||
|
||||
|
||||
def _scope_of(path: Path, item: Mapping[str, Any]) -> str:
|
||||
"""The text of the scope spec a case names, from the specs folder beside the
|
||||
dataset; "" when it names none."""
|
||||
name = item.get("scope")
|
||||
return (path.parent / "specs" / str(name)).read_text("utf-8") if name else ""
|
||||
|
||||
|
||||
def _request_of(item: Mapping[str, Any], scope: str) -> str:
|
||||
"""A case's request: its own, or the statement of intent in its scope spec."""
|
||||
if item.get("workflow_request"):
|
||||
return str(item["workflow_request"])
|
||||
spec = yaml.safe_load(scope) if scope else {}
|
||||
return str(((spec or {}).get("intent") or {}).get("statement", "")).strip()
|
||||
|
||||
|
||||
def load_cases(path: Path) -> list[CaseSpec]:
|
||||
return [
|
||||
CaseSpec(
|
||||
id=item["id"],
|
||||
name=item["name"],
|
||||
workflow_request=item["workflow_request"],
|
||||
workflow_request=_request_of(item, _scope_of(path, item)),
|
||||
required_products=tuple(item["required_products"]),
|
||||
scripted_answers=tuple(
|
||||
tuple(pair) for pair in item.get("scripted_answers", [])
|
||||
@@ -121,6 +140,7 @@ def load_cases(path: Path) -> list[CaseSpec]:
|
||||
change=str(item.get("change", "")),
|
||||
overlay_skills=tuple(item.get("overlay_skills", ())),
|
||||
terraform_modules=bool(item.get("terraform_modules", False)),
|
||||
scope=_scope_of(path, item),
|
||||
code_standards=tuple(item.get("code_standards", ())),
|
||||
)
|
||||
for item in json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
@@ -133,6 +133,7 @@ async def run_with_scripted_user(
|
||||
execution_id=execution_id,
|
||||
overlay_skills=case.overlay_skills,
|
||||
terraform_modules=case.terraform_modules,
|
||||
scope=case.scope or None,
|
||||
)
|
||||
for _ in range(max_user_turns):
|
||||
if interaction.status != "awaiting_user" or interaction.prompt is None:
|
||||
@@ -196,6 +197,7 @@ async def run_case(
|
||||
scripted_answers=case.scripted_answers,
|
||||
overlay_skills=case.overlay_skills,
|
||||
terraform_modules=case.terraform_modules,
|
||||
scope=case.scope or None,
|
||||
)
|
||||
baseline_tokens = token_report(adk_runtime.usage)["total"].get("total")
|
||||
baseline_seconds = round(time.monotonic() - started, 1)
|
||||
@@ -211,6 +213,7 @@ async def run_case(
|
||||
scripted_answers=case.scripted_answers,
|
||||
overlay_skills=case.overlay_skills,
|
||||
terraform_modules=case.terraform_modules,
|
||||
scope=case.scope or None,
|
||||
baseline=base.execution_id,
|
||||
)
|
||||
elif case.interactive:
|
||||
@@ -227,6 +230,7 @@ async def run_case(
|
||||
scripted_answers=case.scripted_answers,
|
||||
overlay_skills=case.overlay_skills,
|
||||
terraform_modules=case.terraform_modules,
|
||||
scope=case.scope or None,
|
||||
)
|
||||
elapsed = time.monotonic() - started
|
||||
write_progress()
|
||||
|
||||
@@ -57,6 +57,17 @@ role_skills:
|
||||
# once it holds lessons for that Specialist's Deliverable.
|
||||
lessons_skill: google-cloud-lessons
|
||||
|
||||
# The scope spec (ADR-0008). `skill` holds its catalogue, which names no Cloud
|
||||
# provider. `values` are what this provider allows for the fields the catalogue
|
||||
# leaves to a provider, in this provider's own terms.
|
||||
scope:
|
||||
skill: scoping
|
||||
values:
|
||||
estate.cloud_provider: [gcp]
|
||||
compliance.encryption.type: [cmek, cmek_hsm]
|
||||
networking.topology: [single_vpc, shared_vpc, shared_vpc_hub_and_spoke]
|
||||
networking.hybrid_connectivity.type: [cloud_vpn, cloud_interconnect, partner_interconnect]
|
||||
|
||||
# One advisor per architecture-framework pillar; each loads only its pillar's Skill.
|
||||
pillar_skills:
|
||||
security: google-cloud-waf-security
|
||||
|
||||
36
skills/scoping/SKILL.md
Normal file
36
skills/scoping/SKILL.md
Normal file
@@ -0,0 +1,36 @@
|
||||
---
|
||||
name: scoping
|
||||
description: >-
|
||||
The scope spec: the structured statement of what a customer wants and the
|
||||
facts a solution architecture depends on, with the catalogue of its fields.
|
||||
Use when reading a scope spec in your context, to know what a field means,
|
||||
whether the customer stated it or left it to you, and what binds your
|
||||
Deliverable.
|
||||
metadata:
|
||||
scope: every Deliverable
|
||||
---
|
||||
|
||||
# Scoping
|
||||
|
||||
A scope spec is a YAML file the customer's architect supplies, complete or partly filled. It sits beside the Intent, which stays in the customer's words. `assets/catalogue.yaml` lists every field it may hold: its type, when it must be settled, the question to ask for it, and guidance for a default.
|
||||
|
||||
## What a field says
|
||||
|
||||
| In the spec | Meaning |
|
||||
| --- | --- |
|
||||
| a value | The customer states this. It is theirs: use it as given. |
|
||||
| `ask`, or left out | Not known yet. |
|
||||
| `decide` | The customer leaves the choice to you. Choose, and record it as an assumption with what it affects. |
|
||||
| `not_applicable` | It does not apply. Ask nothing and assume nothing. |
|
||||
|
||||
## Rules for every Specialist
|
||||
|
||||
- **A stated value binds you.** A region, a topology, a product, a setting or a number the spec states is the customer's decision. Honour it. If you believe it is wrong, say so with `needs_input`; do not quietly replace it.
|
||||
- **A stated choice is not yours to generalise.** The rule that requirements name no products is about what you introduce. What the customer wrote stays as they wrote it.
|
||||
- **Build nothing the customer excluded.** `intent.exclusions` is out of scope for every Deliverable.
|
||||
- **The customer's own requirements carry their ids.** Each entry in `intent.requirements` must be met. When your work serves one, name its id.
|
||||
- **Do not fill a gap silently.** A field under `to_ask` is unknown: ask for it when you may ask the user, and otherwise assume a default and say so. A field under `to_decide` is yours to choose: record the choice as an assumption.
|
||||
|
||||
## The kinds of engagement
|
||||
|
||||
`engagement.kind` is `workload` (an application), `platform_foundation` (a landing zone or shared platform that workloads will later run on) or `migration`. A platform foundation has no users or response times of its own: its subject is the organisation's structure, shared networking, identity, environments and guardrails.
|
||||
318
skills/scoping/assets/catalogue.yaml
Normal file
318
skills/scoping/assets/catalogue.yaml
Normal file
@@ -0,0 +1,318 @@
|
||||
# The scope spec's catalogue: every field a scope spec may hold.
|
||||
#
|
||||
# A field is a dotted path. `type` is one of: string, text, integer, number, boolean,
|
||||
# date, list, enum, records. `tier` says when it must be settled: essential (always,
|
||||
# before any work), important (asked when someone is present) or optional. A tier
|
||||
# may differ by the kind of engagement. `question` is what to ask the customer for
|
||||
# it. `default` guides the choice when the customer says "decide". `affects` says
|
||||
# what a wrong assumption would change.
|
||||
#
|
||||
# This catalogue names no Cloud provider. A provider's bundle adds allowed values
|
||||
# for the fields marked `provider_values`.
|
||||
schema_version: 1
|
||||
kinds: [workload, platform_foundation, migration]
|
||||
|
||||
fields:
|
||||
metadata.customer_case:
|
||||
type: string
|
||||
tier: essential
|
||||
aliases: [metadata.engagement_id]
|
||||
question: What short reference identifies this engagement?
|
||||
metadata.engagement_reference:
|
||||
type: string
|
||||
metadata.client_name:
|
||||
type: string
|
||||
metadata.author:
|
||||
type: string
|
||||
metadata.revision:
|
||||
type: string
|
||||
|
||||
engagement.kind:
|
||||
type: enum
|
||||
values: [workload, platform_foundation, migration]
|
||||
tier: essential
|
||||
question: >-
|
||||
Is this an application workload, a platform foundation such as a landing zone,
|
||||
or a migration of existing systems?
|
||||
|
||||
intent.statement:
|
||||
type: text
|
||||
tier: essential
|
||||
question: In the customer's own words, what do they want?
|
||||
intent.requirements:
|
||||
type: records
|
||||
keys: [id, statement]
|
||||
optional_keys: [category]
|
||||
question: Which requirements has the customer stated outright?
|
||||
intent.exclusions:
|
||||
type: list
|
||||
tier: important
|
||||
question: What has the customer said is out of scope?
|
||||
default: Nothing is excluded beyond what the statement implies.
|
||||
affects: what is designed and planned
|
||||
intent.success_criteria:
|
||||
type: list
|
||||
question: How will the customer judge this a success?
|
||||
default: The stated requirements are met.
|
||||
|
||||
estate.cloud_provider:
|
||||
type: enum
|
||||
provider_values: true
|
||||
tier: essential
|
||||
question: Which Cloud provider is this for?
|
||||
estate.mode:
|
||||
type: enum
|
||||
values: [greenfield, brownfield_expansion, migration]
|
||||
tier: essential
|
||||
question: Is this a new estate, an extension of an existing one, or a migration?
|
||||
estate.regions.primary:
|
||||
type: string
|
||||
tier: essential
|
||||
question: Which region should it run in?
|
||||
estate.regions.secondary:
|
||||
type: string
|
||||
question: Is there a second region, and which?
|
||||
default: None, unless availability or recovery targets need one.
|
||||
affects: resilience design and cost
|
||||
estate.regions.resiliency_tier:
|
||||
type: enum
|
||||
values: [single_region, warm_standby, active_active]
|
||||
tier: important
|
||||
question: How should it survive the loss of a region?
|
||||
default: single_region
|
||||
affects: resilience design and cost
|
||||
estate.data_residency:
|
||||
type: string
|
||||
tier: essential
|
||||
question: Where must data stay?
|
||||
estate.existing_systems:
|
||||
type: text
|
||||
tier: {migration: essential, platform_foundation: important, workload: important}
|
||||
question: What exists today that this replaces, extends or must connect to?
|
||||
default: Nothing exists today.
|
||||
affects: connectivity, migration work and the plan
|
||||
|
||||
workload.users.who:
|
||||
type: enum
|
||||
values: [employees, customers, partners, public, systems]
|
||||
tier: {workload: essential, migration: essential}
|
||||
question: Who uses it?
|
||||
workload.users.count:
|
||||
type: integer
|
||||
tier: {workload: essential, migration: essential}
|
||||
question: How many users?
|
||||
workload.users.peak_concurrent:
|
||||
type: integer
|
||||
tier: {workload: important, migration: important}
|
||||
question: How many use it at once at peak?
|
||||
default: A tenth of the user count.
|
||||
affects: scaling limits and cost
|
||||
workload.availability.target:
|
||||
type: string
|
||||
tier: {workload: essential, migration: essential, platform_foundation: important}
|
||||
question: What availability is required, as a percentage?
|
||||
default: "99.9%"
|
||||
affects: resilience design and cost
|
||||
workload.availability.hours:
|
||||
type: enum
|
||||
values: [business_hours, extended, always]
|
||||
tier: important
|
||||
question: During which hours must it be available?
|
||||
default: always
|
||||
affects: scaling to zero and support
|
||||
workload.availability.recovery_time:
|
||||
type: string
|
||||
tier: important
|
||||
question: How quickly must it recover from a failure?
|
||||
default: 4 hours
|
||||
affects: backup and standby design
|
||||
workload.availability.recovery_point:
|
||||
type: string
|
||||
tier: important
|
||||
question: How much recent data may be lost in a failure?
|
||||
default: 1 hour
|
||||
affects: backup frequency
|
||||
|
||||
compliance.industry:
|
||||
type: string
|
||||
question: "Which industry is the customer in?"
|
||||
compliance.frameworks:
|
||||
type: list
|
||||
tier: essential
|
||||
question: Which regulations or standards apply? An empty list if none.
|
||||
compliance.data_classification:
|
||||
type: enum
|
||||
values: [public, internal, confidential, restricted]
|
||||
tier: essential
|
||||
question: How sensitive is the data?
|
||||
compliance.encryption.type:
|
||||
type: enum
|
||||
values: [provider_managed, customer_managed]
|
||||
provider_values: true
|
||||
tier: important
|
||||
question: Must the customer control the encryption keys?
|
||||
default: provider_managed
|
||||
affects: key management design
|
||||
compliance.encryption.key_rotation_days:
|
||||
type: integer
|
||||
question: How often must keys rotate, in days?
|
||||
default: 90
|
||||
compliance.audit_logging.retention_days:
|
||||
type: integer
|
||||
tier: important
|
||||
question: How long must audit logs be kept, in days?
|
||||
default: 365
|
||||
affects: storage cost and log design
|
||||
compliance.audit_logging.immutable_retention_policy:
|
||||
type: boolean
|
||||
question: Must audit logs be undeletable for that time?
|
||||
default: false
|
||||
|
||||
networking.exposure:
|
||||
type: enum
|
||||
values: [internal_only, internet_facing]
|
||||
tier: important
|
||||
question: Is it reached from the internet or only from inside?
|
||||
default: internal_only
|
||||
affects: perimeter design
|
||||
networking.topology:
|
||||
type: enum
|
||||
provider_values: true
|
||||
tier: {platform_foundation: important}
|
||||
question: Is a network topology already decided?
|
||||
default: The simplest topology that meets the requirements.
|
||||
affects: the whole network design
|
||||
networking.hybrid_connectivity.enabled:
|
||||
type: boolean
|
||||
tier: {platform_foundation: important, migration: important}
|
||||
question: Must it connect privately to on-premises or another cloud?
|
||||
default: false
|
||||
affects: network design and lead time
|
||||
networking.hybrid_connectivity.type:
|
||||
type: enum
|
||||
provider_values: true
|
||||
question: By what kind of connection?
|
||||
networking.hybrid_connectivity.bandwidth_gbps:
|
||||
type: number
|
||||
question: At what bandwidth, in Gbps?
|
||||
networking.egress_control.model:
|
||||
type: enum
|
||||
values: [direct, provider_managed, centralized_appliance]
|
||||
question: How must traffic to the internet be controlled?
|
||||
default: provider_managed
|
||||
affects: network design and cost
|
||||
networking.egress_control.appliance_vendor:
|
||||
type: string
|
||||
question: Which inspection appliance, if one is mandated?
|
||||
networking.cidr_allocations:
|
||||
type: text
|
||||
tier: {platform_foundation: important}
|
||||
question: Which IP ranges are allocated, and how large per environment?
|
||||
default: Private ranges that do not overlap, sized for growth.
|
||||
affects: every network resource
|
||||
|
||||
governance.identity_source_of_truth:
|
||||
type: string
|
||||
tier: important
|
||||
question: Which identity provider do people sign in with?
|
||||
default: The Cloud provider's own identities.
|
||||
affects: access design
|
||||
governance.pipeline_authentication:
|
||||
type: string
|
||||
question: How do delivery pipelines authenticate?
|
||||
default: Short-lived federated credentials, no stored keys.
|
||||
governance.environment_tiers:
|
||||
type: records
|
||||
keys: [name]
|
||||
optional_keys: [archetype]
|
||||
tier: important
|
||||
question: Which environments are needed?
|
||||
default: development, test and production.
|
||||
affects: the number of environments built and planned
|
||||
|
||||
operations.operating_model:
|
||||
type: string
|
||||
question: "Who runs the platform day to day?"
|
||||
operations.support_hours:
|
||||
type: enum
|
||||
values: [business_hours, extended, always]
|
||||
question: During which hours is it supported?
|
||||
default: business_hours
|
||||
operations.monitoring:
|
||||
type: text
|
||||
question: What monitoring and alerting is expected?
|
||||
default: The Cloud provider's own monitoring, with alerts on availability and errors.
|
||||
operations.change_process:
|
||||
type: text
|
||||
question: "How are changes approved and released?"
|
||||
|
||||
integrations:
|
||||
type: records
|
||||
keys: [name]
|
||||
optional_keys: [direction, criticality]
|
||||
question: Which other systems must it exchange data with?
|
||||
default: None.
|
||||
|
||||
delivery.target_start_date:
|
||||
type: date
|
||||
question: "When should work start?"
|
||||
delivery.window_weeks:
|
||||
type: integer
|
||||
tier: important
|
||||
question: Is there a deadline, in weeks from the start?
|
||||
default: No deadline; the plan gives the time the work needs.
|
||||
affects: the plan and whether the scope fits
|
||||
delivery.commercial_model:
|
||||
type: enum
|
||||
values: [time_and_materials, fixed_price_milestone, managed_service]
|
||||
question: How is the work contracted?
|
||||
delivery.iac_tooling:
|
||||
type: string
|
||||
question: "Which infrastructure-as-code tool?"
|
||||
default: terraform
|
||||
delivery.ci_cd:
|
||||
type: string
|
||||
question: "Which delivery pipeline tool?"
|
||||
delivery.team_size:
|
||||
type: integer
|
||||
tier: important
|
||||
question: How many engineers will build it?
|
||||
default: 2
|
||||
affects: the timeline
|
||||
delivery.working_days_per_week:
|
||||
type: integer
|
||||
default: 5
|
||||
|
||||
cost.monthly_budget:
|
||||
type: number
|
||||
tier: {workload: essential, migration: essential, platform_foundation: important}
|
||||
question: What is the monthly running budget?
|
||||
default: No ceiling; cost is optimised within the requirements.
|
||||
affects: product tiers and resilience choices
|
||||
cost.currency:
|
||||
type: string
|
||||
default: "USD"
|
||||
|
||||
constraints.products_required:
|
||||
type: list
|
||||
question: Which products has the customer mandated?
|
||||
default: None.
|
||||
constraints.products_forbidden:
|
||||
type: list
|
||||
question: Which products are ruled out?
|
||||
default: None.
|
||||
constraints.other:
|
||||
type: list
|
||||
|
||||
run.pre_approved:
|
||||
type: boolean
|
||||
default: false
|
||||
run.allow_cloud_read:
|
||||
type: boolean
|
||||
default: false
|
||||
run.overlay_skills:
|
||||
type: list
|
||||
default: []
|
||||
run.terraform_modules:
|
||||
type: boolean
|
||||
default: false
|
||||
198
tests/execution/test_scope_spec.py
Normal file
198
tests/execution/test_scope_spec.py
Normal file
@@ -0,0 +1,198 @@
|
||||
"""An Execution started from a scope spec (ADR-0008)."""
|
||||
|
||||
import json
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
from langgraph.checkpoint.memory import InMemorySaver
|
||||
|
||||
from app.adk.fake_runtime import FakeAgentRuntime, sample_submission
|
||||
from app.adk.runtime import NoDecider, RoleSpec
|
||||
from app.agent import ExecutionEngine
|
||||
from app.nodes.router import Router, Thresholds
|
||||
from app.providers import load_provider
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
from app.workflows.deliverables import DeliverableKind as K
|
||||
|
||||
ROLES = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
**{
|
||||
spec.specialist: RoleSpec(
|
||||
spec.specialist, "Produce.", budget=15, skills=("base",)
|
||||
)
|
||||
for spec in DEPENDENCY_GRAPH.values()
|
||||
},
|
||||
}
|
||||
ANALYST = DEPENDENCY_GRAPH[K.REQUIREMENTS].specialist
|
||||
SPEC = """
|
||||
metadata: {engagement_id: ENG-77}
|
||||
engagement: {kind: platform_foundation}
|
||||
intent:
|
||||
statement: Build us a landing zone.
|
||||
exclusions: [application workloads]
|
||||
estate:
|
||||
cloud_provider: gcp
|
||||
mode: greenfield
|
||||
regions: {primary: europe-west2}
|
||||
data_residency: United Kingdom
|
||||
compliance:
|
||||
frameworks: [DORA]
|
||||
data_classification: confidential
|
||||
networking: {topology: shared_vpc_hub_and_spoke, cidr_allocations: decide}
|
||||
run: {pre_approved: true, terraform_modules: false}
|
||||
"""
|
||||
|
||||
|
||||
class Recording(FakeAgentRuntime):
|
||||
def __init__(self, turns):
|
||||
super().__init__(turns)
|
||||
self.seen: dict[str, dict] = {}
|
||||
|
||||
async def run(self, role, task, tools):
|
||||
self.seen[role.name] = {
|
||||
"skills": role.skills,
|
||||
"scope": task.context.get("scope"),
|
||||
}
|
||||
return await super().run(role, task, tools)
|
||||
|
||||
|
||||
def engine(tmp_path) -> tuple[ExecutionEngine, Recording]:
|
||||
bundle = load_provider("gcp")
|
||||
runtime = Recording(
|
||||
{
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
}
|
||||
)
|
||||
built = ExecutionEngine(
|
||||
runtime=runtime,
|
||||
roles=ROLES,
|
||||
recorder=InMemoryRecorder(),
|
||||
checkpointer=InMemorySaver(),
|
||||
output_root=tmp_path,
|
||||
orchestrator=Router(NoDecider(), None, Thresholds()),
|
||||
scope_catalogue=bundle.scope_catalogue(),
|
||||
scope_values=bundle.scope_values,
|
||||
scope_skill=bundle.scope_skill,
|
||||
)
|
||||
return built, runtime
|
||||
|
||||
|
||||
async def test_a_spec_alone_starts_an_execution_and_names_its_customer_case(tmp_path):
|
||||
built, _ = engine(tmp_path)
|
||||
|
||||
result = await built.start("", scope=SPEC, execution_id="v1")
|
||||
|
||||
assert result.status == "completed"
|
||||
state = result.state
|
||||
# The spec said pre-approved, named the case, and gave the request.
|
||||
assert state["pre_approved"] is True and state["customer_case"] == "eng-77"
|
||||
assert state["workflow_request"] == "Build us a landing zone."
|
||||
assert state["scope"]["status"] == "APPROVED"
|
||||
|
||||
|
||||
async def test_the_supplied_file_is_kept_unaltered_beside_a_completed_copy(tmp_path):
|
||||
built, _ = engine(tmp_path)
|
||||
|
||||
await built.start("", scope=SPEC, execution_id="v1")
|
||||
|
||||
intent = tmp_path / "deliverables" / "cases" / "eng-77" / "intent"
|
||||
assert (intent / "001.scope.yaml").read_text() == SPEC
|
||||
completed = yaml.safe_load((intent / "001.scope.completed.yaml").read_text())
|
||||
assert completed["metadata"]["status"] == "APPROVED"
|
||||
assert completed["networking"]["cidr_allocations"] == "decide"
|
||||
assert completed["provenance"]["estate.regions.primary"] == "supplied"
|
||||
assert [a["field"] for a in completed["inferred_assumptions"]] == [
|
||||
"networking.cidr_allocations"
|
||||
]
|
||||
history = json.loads((intent / "history.json").read_text())
|
||||
assert history["intents"][0]["scope"] == {
|
||||
"completed": "intent/001.scope.completed.yaml",
|
||||
"supplied": "intent/001.scope.yaml",
|
||||
}
|
||||
manifest = json.loads(next(tmp_path.rglob("execution.json")).read_text())
|
||||
assert manifest["scope"]["status"] == "APPROVED"
|
||||
assert manifest["scope"]["readiness"]["assumed"] == 1
|
||||
|
||||
|
||||
async def test_every_specialist_is_shown_the_spec_and_given_the_skill(tmp_path):
|
||||
built, runtime = engine(tmp_path)
|
||||
|
||||
await built.start("", scope=SPEC, execution_id="v1")
|
||||
|
||||
for name, seen in runtime.seen.items():
|
||||
assert seen["skills"] == ("base", "scoping"), name
|
||||
assert seen["scope"]["kind"] == "platform_foundation"
|
||||
stated = runtime.seen[ANALYST]["scope"]["stated"]
|
||||
assert stated["networking"]["topology"] == "shared_vpc_hub_and_spoke"
|
||||
assert stated["intent"]["exclusions"] == ["application workloads"]
|
||||
assert runtime.seen[ANALYST]["scope"]["to_decide"][0]["field"] == (
|
||||
"networking.cidr_allocations"
|
||||
)
|
||||
|
||||
|
||||
async def test_without_a_spec_nothing_changes(tmp_path):
|
||||
built, runtime = engine(tmp_path)
|
||||
|
||||
result = await built.start("Ingest webhooks", pre_approved=True)
|
||||
|
||||
assert result.state["scope"] is None
|
||||
assert runtime.seen[ANALYST] == {"skills": ("base",), "scope": None}
|
||||
manifest = json.loads(next(tmp_path.rglob("execution.json")).read_text())
|
||||
assert manifest["scope"] is None
|
||||
assert not list(tmp_path.rglob("*.scope*.yaml"))
|
||||
|
||||
|
||||
async def test_an_essential_gap_refuses_a_start_with_nobody_to_ask(tmp_path):
|
||||
built, runtime = engine(tmp_path)
|
||||
gappy = SPEC.replace(" data_residency: United Kingdom\n", "")
|
||||
|
||||
with pytest.raises(ValueError, match="estate.data_residency") as refused:
|
||||
await built.start("", scope=gappy)
|
||||
|
||||
assert "nobody to ask" in str(refused.value)
|
||||
assert runtime.seen == {} and not list(tmp_path.rglob("history.json"))
|
||||
# With someone present the same spec starts, and the gap is theirs to answer.
|
||||
waiting = await built.start("", scope=gappy, pre_approved=False, execution_id="v2")
|
||||
assert waiting.state["scope"]["status"] == "DRAFT"
|
||||
asked = waiting.state["scope"]["context"]["to_ask"]
|
||||
assert asked[0] == {
|
||||
"field": "estate.data_residency",
|
||||
"prompt": "Where must data stay?",
|
||||
"severity": "HIGH",
|
||||
}
|
||||
|
||||
|
||||
async def test_a_spec_that_cannot_be_read_refuses_the_start(tmp_path):
|
||||
built, runtime = engine(tmp_path)
|
||||
|
||||
with pytest.raises(ValueError, match="networking.topolgy is not a field"):
|
||||
await built.start("", scope=SPEC.replace("topology", "topolgy"))
|
||||
with pytest.raises(ValueError, match="must be one of"):
|
||||
await built.start(
|
||||
"", scope=SPEC.replace("cloud_provider: gcp", "cloud_provider: aws")
|
||||
)
|
||||
with pytest.raises(ValueError, match="started for another-case"):
|
||||
await built.start("", scope=SPEC, customer_case="another-case")
|
||||
assert runtime.seen == {}
|
||||
|
||||
|
||||
async def test_what_the_caller_says_outright_wins_over_the_specs_run_options(tmp_path):
|
||||
built, _ = engine(tmp_path)
|
||||
|
||||
waiting = await built.start("", scope=SPEC, pre_approved=False, execution_id="v1")
|
||||
|
||||
assert waiting.state["pre_approved"] is False
|
||||
assert waiting.status == "awaiting_user"
|
||||
|
||||
|
||||
async def test_a_request_is_still_needed_when_the_spec_has_no_statement(tmp_path):
|
||||
built, _ = engine(tmp_path)
|
||||
|
||||
with pytest.raises(ValueError, match="A request is needed"):
|
||||
await built.start(
|
||||
"",
|
||||
scope=SPEC.replace(" statement: Build us a landing zone.\n", ""),
|
||||
pre_approved=False,
|
||||
)
|
||||
290
tests/test_scope.py
Normal file
290
tests/test_scope.py
Normal file
@@ -0,0 +1,290 @@
|
||||
"""The scope spec (ADR-0008): read against the catalogue, with no model involved."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import yaml
|
||||
|
||||
from app.providers import load_provider
|
||||
from app.scope import (
|
||||
APPROVED,
|
||||
ASK,
|
||||
DRAFT,
|
||||
ESSENTIAL,
|
||||
UNDER_REVIEW,
|
||||
Catalogue,
|
||||
FieldSpec,
|
||||
ScopeError,
|
||||
read_scope,
|
||||
render_reading,
|
||||
render_template,
|
||||
slug,
|
||||
)
|
||||
|
||||
CATALOGUE = Catalogue(
|
||||
schema_version=1,
|
||||
kinds=("workload", "platform_foundation"),
|
||||
fields={
|
||||
spec.path: spec
|
||||
for spec in (
|
||||
FieldSpec(
|
||||
"metadata.customer_case",
|
||||
tier=ESSENTIAL,
|
||||
aliases=("metadata.engagement_id",),
|
||||
),
|
||||
FieldSpec(
|
||||
"engagement.kind",
|
||||
type="enum",
|
||||
values=("workload", "platform_foundation"),
|
||||
tier=ESSENTIAL,
|
||||
),
|
||||
FieldSpec("intent.statement", type="text", tier=ESSENTIAL),
|
||||
FieldSpec(
|
||||
"intent.requirements",
|
||||
type="records",
|
||||
keys=("id", "statement"),
|
||||
optional_keys=("category",),
|
||||
),
|
||||
FieldSpec(
|
||||
"users.count",
|
||||
type="integer",
|
||||
tier={"workload": ESSENTIAL},
|
||||
question="How many users?",
|
||||
),
|
||||
FieldSpec(
|
||||
"users.peak",
|
||||
type="integer",
|
||||
tier="important",
|
||||
question="How many at once?",
|
||||
default="a tenth of the users",
|
||||
affects="scaling limits",
|
||||
),
|
||||
FieldSpec("estate.region", tier="important", question="Which region?"),
|
||||
FieldSpec("network.topology", type="enum", provider_values=True),
|
||||
FieldSpec("delivery.start", type="date"),
|
||||
FieldSpec("run.pre_approved", type="boolean"),
|
||||
)
|
||||
},
|
||||
)
|
||||
COMPLETE = """
|
||||
metadata: {customer_case: acme-bookings}
|
||||
engagement: {kind: workload}
|
||||
intent:
|
||||
statement: Staff book rooms.
|
||||
requirements:
|
||||
- {id: REQ-01, statement: Rooms can be booked.}
|
||||
users: {count: 400, peak: 40}
|
||||
estate: {region: europe-west2}
|
||||
"""
|
||||
|
||||
|
||||
def read(text: str, **options):
|
||||
return read_scope(text, CATALOGUE, **options)
|
||||
|
||||
|
||||
def test_a_complete_spec_has_nothing_to_ask():
|
||||
scope = read(COMPLETE)
|
||||
|
||||
assert scope.kind == "workload" and scope.customer_case == "acme-bookings"
|
||||
assert scope.statement == "Staff book rooms."
|
||||
assert scope.gaps(ESSENTIAL, "important") == []
|
||||
assert scope.values["users.count"] == 400
|
||||
assert (
|
||||
scope.status() == UNDER_REVIEW and scope.status(pre_approved=True) == APPROVED
|
||||
)
|
||||
|
||||
|
||||
def test_a_field_left_out_or_marked_ask_is_a_gap_with_its_question():
|
||||
scope = read(COMPLETE.replace("count: 400, peak: 40", "peak: ask"))
|
||||
|
||||
assert scope.gaps() == [
|
||||
{"field": "users.count", "prompt": "How many users?", "severity": "HIGH"},
|
||||
{"field": "users.peak", "prompt": "How many at once?", "severity": "MEDIUM"},
|
||||
# Optional fields nobody filled are gaps too, the least pressing.
|
||||
*[gap for gap in scope.gaps() if gap["severity"] == "LOW"],
|
||||
]
|
||||
assert scope.status(pre_approved=True) == DRAFT
|
||||
|
||||
|
||||
def test_decide_is_an_assumption_and_not_applicable_is_neither():
|
||||
scope = read(
|
||||
COMPLETE.replace("peak: 40", "peak: decide").replace(
|
||||
"estate: {region: europe-west2}", "estate: not_applicable"
|
||||
)
|
||||
)
|
||||
|
||||
(assumed,) = scope.assumptions()
|
||||
assert assumed["id"] == "ASSUMP-01" and assumed["field"] == "users.peak"
|
||||
assert "a tenth of the users" in assumed["statement"]
|
||||
assert assumed["impact_if_invalid"] == "Changes scaling limits."
|
||||
# A whole section marked at once covers every field in it.
|
||||
assert scope.states["estate.region"] == "not_applicable"
|
||||
assert all(gap["field"] != "estate.region" for gap in scope.gaps())
|
||||
|
||||
|
||||
def test_which_fields_are_essential_depends_on_the_kind_of_engagement():
|
||||
foundation = read(
|
||||
COMPLETE.replace("workload", "platform_foundation").replace(
|
||||
"users: {count: 400, peak: 40}", ""
|
||||
)
|
||||
)
|
||||
|
||||
assert foundation.gaps(ESSENTIAL) == []
|
||||
assert foundation.tier("users.count") == "optional"
|
||||
|
||||
|
||||
def test_an_unknown_field_is_named_never_ignored():
|
||||
with pytest.raises(ScopeError) as refused:
|
||||
read(COMPLETE + "userz: {count: 1}\nestate: {regoin: x}\n")
|
||||
|
||||
assert "userz is not a field of the scope spec" in refused.value.problems
|
||||
# The second estate key replaced the first; its typo is reported too.
|
||||
assert any("estate.regoin" in problem for problem in refused.value.problems)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("change", "fault"),
|
||||
[
|
||||
(("count: 400", "count: many"), "users.count: must be a whole number"),
|
||||
(("kind: workload", "kind: holiday"), "must be one of workload"),
|
||||
(
|
||||
("{id: REQ-01, statement: Rooms can be booked.}", "{id: REQ-01}"),
|
||||
"needs statement",
|
||||
),
|
||||
(("estate: {region: europe-west2}", "estate: europe"), "estate is a section"),
|
||||
],
|
||||
)
|
||||
def test_a_value_of_the_wrong_kind_is_refused_with_the_reason(change, fault):
|
||||
with pytest.raises(ScopeError) as refused:
|
||||
read(COMPLETE.replace(*change))
|
||||
|
||||
assert any(fault in problem for problem in refused.value.problems)
|
||||
|
||||
|
||||
def test_text_that_is_not_yaml_is_refused():
|
||||
with pytest.raises(ScopeError, match="not valid YAML"):
|
||||
read("intent: [unclosed")
|
||||
with pytest.raises(ScopeError, match="must be a mapping"):
|
||||
read("- just\n- a list\n")
|
||||
|
||||
|
||||
def test_a_provider_supplies_the_values_the_catalogue_leaves_to_it():
|
||||
spec = COMPLETE + "network: {topology: hub_and_spoke}\n"
|
||||
|
||||
with pytest.raises(ScopeError):
|
||||
read(spec, provider_values={"network.topology": ["flat"]})
|
||||
allowed = read(spec, provider_values={"network.topology": ["hub_and_spoke"]})
|
||||
|
||||
assert allowed.values["network.topology"] == "hub_and_spoke"
|
||||
|
||||
|
||||
def test_an_engagement_reference_and_a_customer_case_name_the_same_thing():
|
||||
by_engagement = read(
|
||||
COMPLETE.replace(
|
||||
"customer_case: acme-bookings", "engagement_id: ENG-2026-UK-084"
|
||||
)
|
||||
)
|
||||
|
||||
assert by_engagement.customer_case == "eng-2026-uk-084"
|
||||
assert slug("ENG 2026/UK 084") == "eng-2026-uk-084"
|
||||
with pytest.raises(ScopeError, match="name the same thing and differ"):
|
||||
read(
|
||||
COMPLETE.replace(
|
||||
"{customer_case: acme-bookings}", "{customer_case: a, engagement_id: b}"
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def test_the_completed_copy_holds_every_field_and_the_agents_sections():
|
||||
scope = read(
|
||||
COMPLETE.replace("peak: 40", "peak: decide") + "delivery: {start: 2026-11-01}\n"
|
||||
)
|
||||
|
||||
completed = scope.completed(DRAFT)
|
||||
|
||||
assert completed["metadata"]["status"] == DRAFT
|
||||
assert completed["users"] == {"count": 400, "peak": "decide"}
|
||||
assert completed["delivery"]["start"] == "2026-11-01" # plain data, not a date
|
||||
assert completed["network"]["topology"] == ASK
|
||||
assert completed["provenance"]["users.count"] == "supplied"
|
||||
assert completed["provenance"]["users.peak"] == "assumed"
|
||||
assert completed["provenance"]["network.topology"] == "unknown"
|
||||
assert completed["readiness"]["supplied"] == len(scope.values)
|
||||
assert completed["inferred_assumptions"][0]["field"] == "users.peak"
|
||||
assert yaml.safe_load(yaml.safe_dump(completed)) == completed
|
||||
|
||||
|
||||
def test_a_completed_copy_can_be_read_back_as_a_spec():
|
||||
"""The agent's own sections are replaced, not rejected, on the way back in."""
|
||||
scope = read(COMPLETE.replace("peak: 40", "peak: decide"))
|
||||
again = read(yaml.safe_dump(scope.completed(DRAFT)))
|
||||
|
||||
assert again.values == scope.values and again.states == scope.states
|
||||
|
||||
|
||||
def test_a_specialist_is_shown_what_is_stated_and_what_is_open():
|
||||
scope = read(
|
||||
COMPLETE.replace("peak: 40", "peak: decide") + "run: {pre_approved: true}\n"
|
||||
)
|
||||
|
||||
context = scope.for_context()
|
||||
|
||||
assert context["kind"] == "workload"
|
||||
assert context["stated"]["users"] == {"count": 400}
|
||||
assert context["stated"]["intent"]["requirements"][0]["id"] == "REQ-01"
|
||||
assert context["to_decide"][0]["field"] == "users.peak"
|
||||
# How to run the agent, and the case's name, are not the Specialists' business.
|
||||
assert "run" not in context["stated"] and "metadata" not in context["stated"]
|
||||
|
||||
|
||||
def test_a_template_and_a_reading_are_written_for_a_person():
|
||||
template = yaml.safe_load(render_template(CATALOGUE, "workload"))
|
||||
reading = render_reading(read(COMPLETE.replace("count: 400, peak: 40", "peak: 40")))
|
||||
|
||||
assert template["engagement"]["kind"] == "workload"
|
||||
assert template["users"] == {"count": ASK, "peak": ASK}
|
||||
assert "delivery" not in template # optional fields are left out of a template
|
||||
assert "Essential, needed before work can start:" in reading
|
||||
assert "users.count: How many users?" in reading
|
||||
|
||||
|
||||
# --- the real catalogue and the benchmark spec ------------------------------------------
|
||||
|
||||
|
||||
def test_the_gcp_bundle_reads_the_benchmark_landing_zone_spec():
|
||||
bundle = load_provider("gcp")
|
||||
text = (
|
||||
Path(__file__).resolve().parent.parent
|
||||
/ "eval"
|
||||
/ "datasets"
|
||||
/ "specs"
|
||||
/ "landing-zone.yaml"
|
||||
).read_text()
|
||||
|
||||
scope = read_scope(
|
||||
text, bundle.scope_catalogue(), provider_values=bundle.scope_values
|
||||
)
|
||||
|
||||
assert scope.kind == "platform_foundation"
|
||||
assert scope.customer_case == "case-08-landing-zone"
|
||||
# Complete enough to start with nobody to ask.
|
||||
assert scope.gaps(ESSENTIAL) == []
|
||||
assert scope.values["networking.topology"] == "shared_vpc_hub_and_spoke"
|
||||
assert scope.states["workload.users.count"] == "not_applicable"
|
||||
assert [item["id"] for item in scope.values["intent.requirements"]] == [
|
||||
"REQ-01",
|
||||
"REQ-02",
|
||||
"REQ-03",
|
||||
"REQ-04",
|
||||
]
|
||||
|
||||
|
||||
def test_every_essential_field_of_the_catalogue_has_a_question_to_ask():
|
||||
catalogue = load_provider("gcp").scope_catalogue()
|
||||
|
||||
for kind in catalogue.kinds:
|
||||
for path, spec in catalogue.fields.items():
|
||||
if spec.tier_for(kind) != "optional":
|
||||
assert spec.question, (
|
||||
f"{path} is {spec.tier_for(kind)} and has no question"
|
||||
)
|
||||
Reference in New Issue
Block a user