feat: a stated choice must reach a requirement, and only pinned providers (iteration 9 step 3)
Some checks failed
validation / verify (push) Failing after 11s
Some checks failed
validation / verify (push) Failing after 11s
Case 08 named the customer's stated topology in its design and did not build it; every Check passed because none asked for it. - The catalogue marks the 26 fields that bind a design. The agreed scope lists the ones the customer stated; what was assumed for them does not bind. - A requirement says what it comes from. A submission of requirements is refused when one of the customer's own requirements, or a stated binding field, is named by none, or when the requirement carrying a choice is not a constraint with a condition. The existing chain then owes it a Check. - Generated code may use only the providers the bundle pins: a resource or a required_providers entry of any other is refused before Terraform runs. - The guide opens with how much of the design the customer stated, how much was assumed and how much is unknown. Not run with real models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,7 +14,7 @@ One directory per iteration of this agent. Each holds the plan for that iteratio
|
||||
| 6 | `iter-06-target-architecture-graph/` | complete: all four cases pass with real models, each run twice on 2026-10-08; thresholds set | The Target architecture as one explicit graph of Elements, Relationships and Checks, established before infrastructure code, with validation as a verdict per Check. |
|
||||
| 7 | `iter-07-change-requests/` | complete (2026-10-09): the change case ran with real models and passed | A user restates a request to say how requirements changed; a new Execution starts from a Baseline Execution of the same Customer case and delivers the updated architecture with what changed. |
|
||||
| 8 | `iter-08-terraform-modules/` | steps 1–5 built and run once each way on 2026-10-09, both at 100%; steps 6 and 7 wait on Azure and AWS provider bundles | Infrastructure code is plain resources by default. An Execution may opt into published Terraform modules: only from registry namespaces the provider bundle lists, at one exact version, with a lookup of what a module accepts and a Skill of four Google modules tested against the pinned provider. |
|
||||
| 9 | `iter-09-scoping/` | steps 1, 2 and 6 built on 2026-10-11, not run with real models; steps 3–5 planned; ADR-0008 accepted | An engagement is scoped from a scope spec: a structured file, complete or partly filled, read against a catalogue in code, completed in conversation and approved before requirements begin. Covers workloads and platform foundations. |
|
||||
| 9 | `iter-09-scoping/` | steps 1–3 and 6 built on 2026-10-11; cases 03 and 08 ran on steps 1 and 2; steps 4 and 5 planned; ADR-0008 accepted | An engagement is scoped from a scope spec: a structured file, complete or partly filled, read against a catalogue in code, completed in conversation and approved before requirements begin. Covers workloads and platform foundations. |
|
||||
| 10 | `iter-10-work-breakdown/` | planned; nothing built | A work breakdown structure as a Deliverable: work packages traced to Elements and Checks, activities in sequence, and a timeline on a relative day axis computed and drawn by code. |
|
||||
|
||||
Numbers record when an iteration was planned, not the order to build in. Build iteration 6 before the remaining steps of iteration 4 (10 to 14) and before iteration 5: it changes the Deliverables, Phases and validation format they build on. Its `MAP.md` lists the effect on each of their steps.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 9, Scoping
|
||||
**Intent:** Scope an engagement from a scope spec: a structured file that may arrive complete, partly filled or not at all, completed in conversation and approved by the customer before requirements begin. So that a design's quality no longer rests on what the agents had to assume.
|
||||
**Status:** steps 1, 2 and 6 built on 2026-10-11, not run with real models; steps 3 to 5 planned
|
||||
**Status:** steps 1, 2, 3 and 6 built on 2026-10-11; steps 1 and 2 ran in cases 03 and 08; step 3 not run; steps 4 and 5 planned
|
||||
**This directory covers:** steps 1 to 6
|
||||
|
||||
## Sources of truth
|
||||
@@ -17,7 +17,7 @@
|
||||
|---|---|---|
|
||||
| 1 | The catalogue, reading and validating a spec, the gap list, the two files beside the Intent, run options from the spec, the spec shown to every Specialist | built 2026-10-11 (`steps/step-1.md`); not run with real models |
|
||||
| 2 | The conversation that fills gaps and writes answers into the completed copy; scoping as a Phase with its Deliverable and Approval; a spec drafted from a sentence | built 2026-10-11 (`steps/step-2.md`); not run with real models |
|
||||
| 3 | Requirements traced to the customer's requirement ids and to spec fields, in `requirements.json`; readiness in the solution guide | planned |
|
||||
| 3 | Requirements traced to the customer's requirement ids and to spec fields, in `requirements.json`, with a stated choice refused unless a requirement carries it; readiness in the solution guide | built 2026-10-11 (`steps/step-3.md`); not run with real models |
|
||||
| 4 | An edited spec as a Change request, compared field by field | planned |
|
||||
| 5 | Evaluation cases as specs, replacing answers matched by pattern | planned; `case-08-landing-zone` is the first |
|
||||
| 6 | The status `BASELINE_LOCKED` set when the Execution completes | built with step 2 |
|
||||
|
||||
31
.agents/plans/iter-09-scoping/steps/step-3.md
Normal file
31
.agents/plans/iter-09-scoping/steps/step-3.md
Normal file
@@ -0,0 +1,31 @@
|
||||
# Step 3 — Implementation Detail
|
||||
|
||||
Built on 2026-10-11. **Not run with real models.**
|
||||
|
||||
## What a stated choice now has to reach
|
||||
|
||||
Case 08 showed a stated topology named in a design and not built, with every Check passed. A stated choice was an instruction; nothing held a design to it.
|
||||
|
||||
1. **The catalogue marks the fields that bind a design** (`binds: true`, 26 of them): regions and residency, availability and recovery targets, compliance frameworks and classification, encryption, audit logging, every networking field, identity, environments, and required and forbidden products.
|
||||
2. **The agreed scope lists the binding fields the customer stated,** as `binding` in `scope.json`. A field assumed for them is not among them: an assumption is not the customer's decision.
|
||||
3. **A requirement says what it comes from,** in `from`: the customer's own requirement ids and the scope fields it carries.
|
||||
4. **A submission of requirements is refused** (`_carries_the_scope` in `app/workflows/chain.py`) when:
|
||||
- one of the customer's own requirements is named by no requirement;
|
||||
- a binding field is named by no requirement;
|
||||
- a requirement that carries a binding field is not non-functional, or has no entry in `conditions`.
|
||||
5. **From there the existing chain does the rest.** A non-functional requirement's conditions each need a Check on the Target architecture, and validation gives each Check a verdict. So a stated choice now reaches a Check, in code, at every link.
|
||||
|
||||
What this does not prove: that the condition the analyst writes says what the customer meant, or that the Check the architect writes verifies it. It guarantees a stated choice cannot be silently dropped; whether it is honoured well is still judged by the validator.
|
||||
|
||||
## Readiness in the guide
|
||||
|
||||
The packager is told to open the executive summary with what the design rests on: how much the customer stated, how much was assumed, how much is unknown. An instruction, not a check.
|
||||
|
||||
## Also closed, from the same run
|
||||
|
||||
- **Only the pinned providers.** `check_providers` in `app/tools/local.py` refuses, before Terraform runs, a resource or data source of a provider the bundle does not pin, and a `required_providers` entry for one. The target architect is told that something outside the Cloud provider's own provider is not an Element. A module's own providers are untouched: they are declared inside the module.
|
||||
- **The scope statement names only the customer's exclusions.**
|
||||
|
||||
Not done: `intent.exclusions` are free text, so nothing in code checks that an excluded thing was not built. The Specialists are told; in case 08 they complied.
|
||||
|
||||
Tests: `tests/execution/test_requirements_chain.py` (6 more), `tests/tools/test_local.py` (5 more).
|
||||
@@ -10,6 +10,7 @@ You produce the **solution guide** Deliverable: the Skill's packaging of every a
|
||||
- Consolidate the content of every Deliverable in `dependencies`; rewrite for flow, but keep every requirement, product decision, citation, recommendation, command and validation verdict.
|
||||
- Include the architecture diagram, which is the `architecture.mmd` attachment of the Target architecture, unchanged as a `mermaid` code block, and the Terraform as code blocks.
|
||||
- List the Target architecture's Checks with their validation verdicts in the validation section.
|
||||
- Open the executive summary with what the design rests on, from the scope: how many of its facts the customer stated, how many were assumed for them, and how many are still unknown, naming the assumptions that most shape the design. A reader must be able to tell an agreed design from an indicative one.
|
||||
- When the design recommendations hold a "Questions for the customer" section, carry every question into the guide's design and configuration recommendations section, under the same heading and with the same wording. They are what the request did not say enough to settle; say so, and do not present them as faults or as recommendations.
|
||||
|
||||
In a Change request (`baseline` is set), open the executive summary with what changed and why, from `baseline.changes`: the requirements added, changed and removed with their reasons, then what that changed in the architecture. Name separately anything listed under `unrequested`: it changed without a changed requirement reaching it.
|
||||
|
||||
@@ -13,7 +13,7 @@ You produce the **requirements** Deliverable: the Skill's requirements-discovery
|
||||
- Name every ambiguity or contradiction, explain why the conflicting goals can't both hold, and ask the user to choose. When they delegate the choice, propose one and ask them to approve it.
|
||||
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
|
||||
- Name no products, services or architectures of your own choosing. This Deliverable describes the problem, not the solution. A product or design choice the customer stated in `scope` is theirs: record it as a constraint, in their words.
|
||||
- Cover every entry of `scope.stated.intent.requirements`, the customer's own requirements, and name its id in the requirement that covers it. List what `scope.to_ask` still holds under `Open questions`.
|
||||
- Give each requirement its `from`, as the requirements-chain Skill describes. Cover every entry of `scope.stated.intent.requirements`, the customer's own requirements, by id. Carry every field under `scope.binding`, a choice the customer stated, in a non-functional requirement whose conditions say what must exist or be set for it to have been honoured. A stated choice that reaches no requirement reaches no Check, and can then be named in the design without being built. List what `scope.to_ask` still holds under `Open questions`.
|
||||
- Give every requirement an id and attach them all as `requirements.json`, in the format the requirements-chain Skill gives. Use the same ids in your content.
|
||||
- Give every non-functional requirement its `conditions`: one entry for each separate thing that must be true for it to be met and that the configuration of infrastructure can show. "Data is encrypted in transit and at rest" has two. Each becomes something the design is checked against, so a condition left out is a condition nobody verifies.
|
||||
- Put under `outside_code` what the requirement asks for that no infrastructure configuration can show: an outcome measured in operation (an availability percentage, a response time), behaviour of the application's own code, a process people follow, or a property of a region or product that only its provider publishes (a carbon rating, a service level). For the last, the condition is the choice configuration can show, such as which region is set. Those are recorded and reported, and are not checked against the infrastructure code.
|
||||
|
||||
@@ -17,7 +17,7 @@ The scope spec is kept for you. `scope_read` shows what is settled, what is not,
|
||||
- A field marked `decide` is yours: choose, and record it as assumed with a note.
|
||||
- A field that cannot apply to this kind of engagement is `not_applicable`.
|
||||
4. Call `scope_read` again. Every essential field must be settled before you can submit.
|
||||
5. Submit. Your content is the scope statement under its heading: three to six sentences saying what the customer wants, what is in scope, what is out of scope, and what success looks like, in plain words and using only what the spec holds. The completed spec, the information register, the assumptions and the questions still open are written for you from what you recorded. Pass no attachments.
|
||||
5. Submit. Your content is the scope statement under its heading: three to six sentences saying what the customer wants, what is in scope, what is out of scope, and what success looks like, in plain words and using only what the spec holds. Name as out of scope only what `intent.exclusions` lists; do not add an exclusion of your own. The completed spec, the information register, the assumptions and the questions still open are written for you from what you recorded. Pass no attachments.
|
||||
|
||||
Never record a product, a region, a number or a regulation the customer did not state as though they had: that is `assumed`, with its note. An assumption the customer can see is useful; one presented as their decision is a fault.
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ includes: [specialist-protocol]
|
||||
You produce the **Target architecture** Deliverable: the solution as one explicit graph of Elements, Relationships and Checks. It is agreed before any infrastructure code is written, and the diagram, the description, the code and the validation are all derived from it.
|
||||
|
||||
- Load the requirements-chain Skill for the graph's format and how to derive it.
|
||||
- Every Element is a resource of the Cloud provider's own Terraform provider. Something outside it that the design depends on, such as a source repository or a third party's service, is not an Element: say in the design rationale that it is needed and who provides it.
|
||||
- Build the graph with `graph_put` in a handful of calls of ten to twenty items each: all the Elements, then all the Relationships, then the Checks in batches. A call for every Element, or for every few Checks, uses up your tool calls before the graph is finished; a single call with everything is a reply long enough to time out.
|
||||
- Give every component an Element with the product selected for it, and add the supporting Elements the design needs, such as identities, policies and networks.
|
||||
- Give every connection between Elements a Relationship.
|
||||
|
||||
11
app/scope.py
11
app/scope.py
@@ -82,6 +82,9 @@ class FieldSpec:
|
||||
keys: tuple[str, ...] = ()
|
||||
optional_keys: tuple[str, ...] = ()
|
||||
aliases: tuple[str, ...] = ()
|
||||
# A choice a design must honour. Stated by the customer, it has to be carried
|
||||
# by a requirement, and so by a Check on the Target architecture.
|
||||
binds: bool = False
|
||||
|
||||
def tier_for(self, kind: str | None) -> str:
|
||||
if isinstance(self.tier, Mapping):
|
||||
@@ -118,6 +121,7 @@ def load_catalogue(path: Path) -> Catalogue:
|
||||
keys=tuple(raw.get("keys") or ()),
|
||||
optional_keys=tuple(raw.get("optional_keys") or ()),
|
||||
aliases=tuple(raw.get("aliases") or ()),
|
||||
binds=bool(raw.get("binds")),
|
||||
)
|
||||
return Catalogue(
|
||||
schema_version=int(data.get("schema_version", 1)),
|
||||
@@ -248,6 +252,13 @@ class Scope:
|
||||
return {
|
||||
"kind": self.kind,
|
||||
"stated": stated,
|
||||
# The stated fields a design must honour: each has to be carried by a
|
||||
# requirement, which the requirements are checked for.
|
||||
"binding": [
|
||||
path
|
||||
for path in self.values
|
||||
if self.catalogue.fields[path].binds and self.source(path) != ASSUMED
|
||||
],
|
||||
# What the agent chose where the customer did not say. Not the
|
||||
# customer's decision: it may be revisited when they answer.
|
||||
"assumed": [
|
||||
|
||||
@@ -143,8 +143,10 @@ class TerraformRunner:
|
||||
)
|
||||
|
||||
async def validate(self, files: Mapping[str, str]) -> ToolResult:
|
||||
refusal = _check_files(files) or check_module_sources(
|
||||
files, allowed_namespaces(self.module_namespaces)
|
||||
refusal = (
|
||||
_check_files(files)
|
||||
or check_providers(files, self.providers)
|
||||
or check_module_sources(files, allowed_namespaces(self.module_namespaces))
|
||||
)
|
||||
if refusal:
|
||||
return refusal
|
||||
@@ -160,8 +162,10 @@ class TerraformRunner:
|
||||
async def plan(
|
||||
self, files: Mapping[str, str], variables: Mapping[str, str] | None = None
|
||||
) -> ToolResult:
|
||||
refusal = _check_files(files) or check_module_sources(
|
||||
files, allowed_namespaces(self.module_namespaces)
|
||||
refusal = (
|
||||
_check_files(files)
|
||||
or check_providers(files, self.providers)
|
||||
or check_module_sources(files, allowed_namespaces(self.module_namespaces))
|
||||
)
|
||||
if refusal:
|
||||
return refusal
|
||||
@@ -363,6 +367,45 @@ class TerraformRunner:
|
||||
return process.returncode or 0, stdout.decode(errors="replace")
|
||||
|
||||
|
||||
_BLOCK_TYPE = re.compile(
|
||||
r'^\s*(?:resource|data)\s+"([a-z0-9]+)_[a-z0-9_]*"', re.MULTILINE
|
||||
)
|
||||
_PROVIDER_SOURCE = re.compile(r'^\s*source\s*=\s*"([\w.-]+/[\w.-]+)"\s*$', re.MULTILINE)
|
||||
# Built into Terraform itself: terraform_data and the terraform_remote_state source.
|
||||
_BUILT_IN = frozenset({"terraform"})
|
||||
|
||||
|
||||
def check_providers(
|
||||
files: Mapping[str, str], pinned: Mapping[str, Mapping[str, str]]
|
||||
) -> ToolResult | None:
|
||||
"""Refuse code that uses a provider the bundle does not pin.
|
||||
|
||||
`terraform init` downloads whatever provider the code names, at whatever
|
||||
version is current. A design with a source repository in it brought the GitHub
|
||||
provider into generated code that way on 2026-10-11. With nothing pinned there
|
||||
is nothing to hold code to, and it passes.
|
||||
"""
|
||||
if not pinned:
|
||||
return None
|
||||
names = set(pinned) | _BUILT_IN
|
||||
sources = {pin["source"] for pin in pinned.values()}
|
||||
code = "\n".join(text for name, text in files.items() if name.endswith(".tf"))
|
||||
foreign = sorted({m.group(1) for m in _BLOCK_TYPE.finditer(code)} - names)
|
||||
undeclared = sorted({m.group(1) for m in _PROVIDER_SOURCE.finditer(code)} - sources)
|
||||
if not foreign and not undeclared:
|
||||
return None
|
||||
found = [f"resources of the {name} provider" for name in foreign]
|
||||
found += [f"the provider {source}" for source in undeclared]
|
||||
return ToolResult(
|
||||
"Refused: the code uses "
|
||||
+ ", ".join(found)
|
||||
+ ". Only these providers may be used: "
|
||||
+ ", ".join(sorted(sources))
|
||||
+ ". Write resources of those alone, and describe anything outside them in "
|
||||
"the deployment guidance as a step the user takes."
|
||||
)
|
||||
|
||||
|
||||
def _check_files(files: Mapping[str, str]) -> ToolResult | None:
|
||||
if not files:
|
||||
return ToolResult("Refused: provide at least one Terraform file.")
|
||||
|
||||
@@ -213,9 +213,87 @@ def check_requirements(
|
||||
problems += _lacking_conditions(
|
||||
[item for item in items if item["kind"] == "non_functional"], name
|
||||
)
|
||||
problems += _carries_the_scope(items, name, upstream)
|
||||
return problems
|
||||
|
||||
|
||||
def agreed_scope(upstream: Upstream | None) -> dict[str, Any]:
|
||||
"""The agreed scope, as every Specialist reads it; empty when there is none."""
|
||||
text = (
|
||||
((upstream or {}).get("scope") or {})
|
||||
.get("attachments", {})
|
||||
.get(SCOPE_DATA_FILE, "")
|
||||
)
|
||||
try:
|
||||
data = json.loads(text) if text else {}
|
||||
except ValueError:
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def sources_of(item: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
"""What a requirement comes from: the customer's own requirement ids and the
|
||||
scope spec's fields, as its `from` lists them."""
|
||||
listed = item.get("from")
|
||||
return tuple(str(s) for s in listed) if isinstance(listed, list) else ()
|
||||
|
||||
|
||||
def _carries_the_scope(
|
||||
items: Sequence[Mapping[str, Any]], name: str, upstream: Upstream | None
|
||||
) -> list[str]:
|
||||
"""What the customer stated has to reach the requirements, by name (ADR-0008).
|
||||
|
||||
Each requirement the customer wrote, and each choice they stated that a design
|
||||
must honour, is named in the `from` of a requirement. A choice is carried by a
|
||||
non-functional requirement with a condition, so the Target architecture owes it
|
||||
a Check. Before this a stated topology could be named in a design and not built:
|
||||
case 08 on 2026-10-11 passed all 101 Checks with no Check asking for it.
|
||||
"""
|
||||
scope = agreed_scope(upstream)
|
||||
if not scope:
|
||||
return []
|
||||
stated = scope.get("stated") or {}
|
||||
theirs = [
|
||||
str(entry.get("id"))
|
||||
for entry in ((stated.get("intent") or {}).get("requirements") or [])
|
||||
if isinstance(entry, Mapping) and entry.get("id")
|
||||
]
|
||||
binding = [str(path) for path in scope.get("binding") or []]
|
||||
cited = {source for item in items for source in sources_of(item)}
|
||||
problems = [
|
||||
f"{name}: the customer's requirement {reference} is covered by no "
|
||||
"requirement; name it in the `from` of the one that covers it"
|
||||
for reference in theirs
|
||||
if reference not in cited
|
||||
]
|
||||
problems += [
|
||||
f"{name}: the customer stated {path} and no requirement carries it; add a "
|
||||
f"non-functional requirement whose `from` names {path}, with the condition "
|
||||
"that shows it was honoured"
|
||||
for path in binding
|
||||
if path not in cited
|
||||
]
|
||||
for item in items:
|
||||
carried = [source for source in sources_of(item) if source in binding]
|
||||
if carried and (
|
||||
item.get("kind") != "non_functional" or not conditions_of_listed(item)
|
||||
):
|
||||
problems.append(
|
||||
f"{name}: {item.get('id')} carries {', '.join(carried)}, a choice "
|
||||
"the customer stated; it must be non_functional with at least one "
|
||||
"entry in `conditions`, so the design is checked against it"
|
||||
)
|
||||
return problems
|
||||
|
||||
|
||||
def conditions_of_listed(item: Mapping[str, Any]) -> tuple[str, ...]:
|
||||
"""The conditions a requirement lists outright, with no fallback to its text."""
|
||||
listed = item.get("conditions")
|
||||
if not isinstance(listed, list):
|
||||
return ()
|
||||
return tuple(c.strip() for c in listed if isinstance(c, str) and c.strip())
|
||||
|
||||
|
||||
def check_components(
|
||||
attachments: Mapping[str, str], upstream: Upstream | None
|
||||
) -> list[str]:
|
||||
|
||||
@@ -37,6 +37,19 @@ Every non-functional requirement lists its `conditions`: one entry for each sepa
|
||||
|
||||
`kind` is `functional` or `non_functional`. Write one requirement per entry, so each can be checked on its own.
|
||||
|
||||
When the customer's scope is in your context, a requirement says what it comes from in `from`: the ids of the customer's own requirements, and the scope fields it carries.
|
||||
|
||||
```json
|
||||
{"id": "NFR-4", "kind": "non_functional",
|
||||
"text": "The network is a hub with spokes, joined to on-premises over a private 10Gbps connection.",
|
||||
"conditions": ["A hub network is connected to every spoke network.", "A dedicated 10Gbps connection joins the hub to on-premises."],
|
||||
"from": ["REQ-03", "networking.topology", "networking.hybrid_connectivity.type", "networking.hybrid_connectivity.bandwidth_gbps"]}
|
||||
```
|
||||
|
||||
- Every requirement the customer wrote (`scope.stated.intent.requirements`) is named in the `from` of the requirement that covers it.
|
||||
- Every field under `scope.binding` is a choice the customer stated that the design must honour. Each is named in the `from` of a non-functional requirement with at least one condition that shows it was honoured. Write the condition about what must exist or be set, in terms a Check can verify: "a hub network is connected to every spoke network", not "the topology is hub and spoke".
|
||||
- One requirement may carry several fields that belong together. A submission is refused while any is carried by none.
|
||||
|
||||
### `components.json` (technical decomposition)
|
||||
|
||||
A list of components and the requirement ids each serves. Every functional requirement must be served by at least one component.
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
# before any work), important (asked when someone is present) or optional. A tier
|
||||
# may differ by the kind of engagement. `question` is what to ask the customer for
|
||||
# it. `default` guides the choice when the customer says "decide". `affects` says
|
||||
# what a wrong assumption would change.
|
||||
# what a wrong assumption would change. `binds: true` marks a choice a design must
|
||||
# honour: stated by the customer, it has to reach a requirement and its Checks.
|
||||
#
|
||||
# This catalogue names no Cloud provider. A provider's bundle adds allowed values
|
||||
# for the fields marked `provider_values`.
|
||||
@@ -66,15 +67,18 @@ fields:
|
||||
tier: essential
|
||||
question: Is this a new estate, an extension of an existing one, or a migration?
|
||||
estate.regions.primary:
|
||||
binds: true
|
||||
type: string
|
||||
tier: essential
|
||||
question: Which region should it run in?
|
||||
estate.regions.secondary:
|
||||
binds: true
|
||||
type: string
|
||||
question: Is there a second region, and which?
|
||||
default: None, unless availability or recovery targets need one.
|
||||
affects: resilience design and cost
|
||||
estate.regions.resiliency_tier:
|
||||
binds: true
|
||||
type: enum
|
||||
values: [single_region, warm_standby, active_active]
|
||||
tier: important
|
||||
@@ -82,6 +86,7 @@ fields:
|
||||
default: single_region
|
||||
affects: resilience design and cost
|
||||
estate.data_residency:
|
||||
binds: true
|
||||
type: string
|
||||
tier: essential
|
||||
question: Where must data stay?
|
||||
@@ -108,6 +113,7 @@ fields:
|
||||
default: A tenth of the user count.
|
||||
affects: scaling limits and cost
|
||||
workload.availability.target:
|
||||
binds: true
|
||||
type: string
|
||||
tier: {workload: essential, migration: essential, platform_foundation: important}
|
||||
question: What availability is required, as a percentage?
|
||||
@@ -121,12 +127,14 @@ fields:
|
||||
default: always
|
||||
affects: scaling to zero and support
|
||||
workload.availability.recovery_time:
|
||||
binds: true
|
||||
type: string
|
||||
tier: important
|
||||
question: How quickly must it recover from a failure?
|
||||
default: 4 hours
|
||||
affects: backup and standby design
|
||||
workload.availability.recovery_point:
|
||||
binds: true
|
||||
type: string
|
||||
tier: important
|
||||
question: How much recent data may be lost in a failure?
|
||||
@@ -137,15 +145,18 @@ fields:
|
||||
type: string
|
||||
question: "Which industry is the customer in?"
|
||||
compliance.frameworks:
|
||||
binds: true
|
||||
type: list
|
||||
tier: essential
|
||||
question: Which regulations or standards apply? An empty list if none.
|
||||
compliance.data_classification:
|
||||
binds: true
|
||||
type: enum
|
||||
values: [public, internal, confidential, restricted]
|
||||
tier: essential
|
||||
question: How sensitive is the data?
|
||||
compliance.encryption.type:
|
||||
binds: true
|
||||
type: enum
|
||||
values: [provider_managed, customer_managed]
|
||||
provider_values: true
|
||||
@@ -154,21 +165,25 @@ fields:
|
||||
default: provider_managed
|
||||
affects: key management design
|
||||
compliance.encryption.key_rotation_days:
|
||||
binds: true
|
||||
type: integer
|
||||
question: How often must keys rotate, in days?
|
||||
default: 90
|
||||
compliance.audit_logging.retention_days:
|
||||
binds: true
|
||||
type: integer
|
||||
tier: important
|
||||
question: How long must audit logs be kept, in days?
|
||||
default: 365
|
||||
affects: storage cost and log design
|
||||
compliance.audit_logging.immutable_retention_policy:
|
||||
binds: true
|
||||
type: boolean
|
||||
question: Must audit logs be undeletable for that time?
|
||||
default: false
|
||||
|
||||
networking.exposure:
|
||||
binds: true
|
||||
type: enum
|
||||
values: [internal_only, internet_facing]
|
||||
tier: important
|
||||
@@ -176,6 +191,7 @@ fields:
|
||||
default: internal_only
|
||||
affects: perimeter design
|
||||
networking.topology:
|
||||
binds: true
|
||||
type: enum
|
||||
provider_values: true
|
||||
tier: {platform_foundation: important}
|
||||
@@ -183,28 +199,34 @@ fields:
|
||||
default: The simplest topology that meets the requirements.
|
||||
affects: the whole network design
|
||||
networking.hybrid_connectivity.enabled:
|
||||
binds: true
|
||||
type: boolean
|
||||
tier: {platform_foundation: important, migration: important}
|
||||
question: Must it connect privately to on-premises or another cloud?
|
||||
default: false
|
||||
affects: network design and lead time
|
||||
networking.hybrid_connectivity.type:
|
||||
binds: true
|
||||
type: enum
|
||||
provider_values: true
|
||||
question: By what kind of connection?
|
||||
networking.hybrid_connectivity.bandwidth_gbps:
|
||||
binds: true
|
||||
type: number
|
||||
question: At what bandwidth, in Gbps?
|
||||
networking.egress_control.model:
|
||||
binds: true
|
||||
type: enum
|
||||
values: [direct, provider_managed, centralized_appliance]
|
||||
question: How must traffic to the internet be controlled?
|
||||
default: provider_managed
|
||||
affects: network design and cost
|
||||
networking.egress_control.appliance_vendor:
|
||||
binds: true
|
||||
type: string
|
||||
question: Which inspection appliance, if one is mandated?
|
||||
networking.cidr_allocations:
|
||||
binds: true
|
||||
type: text
|
||||
tier: {platform_foundation: important}
|
||||
question: Which IP ranges are allocated, and how large per environment?
|
||||
@@ -212,16 +234,19 @@ fields:
|
||||
affects: every network resource
|
||||
|
||||
governance.identity_source_of_truth:
|
||||
binds: true
|
||||
type: string
|
||||
tier: important
|
||||
question: Which identity provider do people sign in with?
|
||||
default: The Cloud provider's own identities.
|
||||
affects: access design
|
||||
governance.pipeline_authentication:
|
||||
binds: true
|
||||
type: string
|
||||
question: How do delivery pipelines authenticate?
|
||||
default: Short-lived federated credentials, no stored keys.
|
||||
governance.environment_tiers:
|
||||
binds: true
|
||||
type: records
|
||||
keys: [name]
|
||||
optional_keys: [archetype]
|
||||
@@ -294,10 +319,12 @@ fields:
|
||||
default: "USD"
|
||||
|
||||
constraints.products_required:
|
||||
binds: true
|
||||
type: list
|
||||
question: Which products has the customer mandated?
|
||||
default: None.
|
||||
constraints.products_forbidden:
|
||||
binds: true
|
||||
type: list
|
||||
question: Which products are ruled out?
|
||||
default: None.
|
||||
|
||||
@@ -767,3 +767,113 @@ def test_a_constraint_with_conditions_still_needs_a_check():
|
||||
)
|
||||
|
||||
assert gaps == ("Requirement NFR-2 has no Check",)
|
||||
|
||||
|
||||
# --- what the customer stated reaches the requirements (ADR-0008) ------------------------
|
||||
|
||||
|
||||
def scoped_upstream(**scope) -> dict:
|
||||
import json
|
||||
|
||||
return {"scope": {"attachments": {"scope.json": json.dumps(scope)}}}
|
||||
|
||||
|
||||
def requirement_problems(requirements: list[dict], **scope) -> list[str]:
|
||||
import json
|
||||
|
||||
from app.workflows.chain import check_requirements
|
||||
|
||||
return check_requirements(
|
||||
{"requirements.json": json.dumps(requirements)}, scoped_upstream(**scope)
|
||||
)
|
||||
|
||||
|
||||
STATED = {
|
||||
"stated": {
|
||||
"intent": {
|
||||
"requirements": [{"id": "REQ-01", "statement": "Ready in 10 weeks."}]
|
||||
},
|
||||
"networking": {"topology": "hub_and_spoke"},
|
||||
},
|
||||
"binding": ["networking.topology"],
|
||||
}
|
||||
FUNCTIONAL = {"id": "FR-1", "kind": "functional", "text": "Accept events."}
|
||||
CARRIED = {
|
||||
"id": "NFR-1",
|
||||
"kind": "non_functional",
|
||||
"text": "The network is hub and spoke, ready in ten weeks.",
|
||||
"conditions": ["A hub network is connected to each spoke network."],
|
||||
"from": ["REQ-01", "networking.topology"],
|
||||
}
|
||||
|
||||
|
||||
def test_requirements_that_carry_everything_the_customer_stated_are_accepted():
|
||||
assert requirement_problems([FUNCTIONAL, CARRIED], **STATED) == []
|
||||
|
||||
|
||||
def test_a_customers_own_requirement_must_be_covered_by_name():
|
||||
uncited = {**CARRIED, "from": ["networking.topology"]}
|
||||
|
||||
(problem,) = requirement_problems([FUNCTIONAL, uncited], **STATED)
|
||||
|
||||
assert "the customer's requirement REQ-01 is covered by no requirement" in problem
|
||||
|
||||
|
||||
def test_a_stated_choice_must_be_carried_by_a_requirement():
|
||||
"""Case 08: a stated topology was named in the design and not built, and every
|
||||
Check passed, because nothing asked for it."""
|
||||
uncarried = {**CARRIED, "from": ["REQ-01"]}
|
||||
|
||||
(problem,) = requirement_problems([FUNCTIONAL, uncarried], **STATED)
|
||||
|
||||
assert "the customer stated networking.topology and no requirement carries it" in (
|
||||
problem
|
||||
)
|
||||
|
||||
|
||||
def test_a_stated_choice_is_carried_as_a_constraint_the_design_is_checked_against():
|
||||
as_functional = {**CARRIED, "kind": "functional", "conditions": []}
|
||||
outside_code_only = {
|
||||
**CARRIED,
|
||||
"conditions": [],
|
||||
"outside_code": ["The topology is reviewed by the network team."],
|
||||
}
|
||||
|
||||
for weak in (as_functional, outside_code_only):
|
||||
problems = requirement_problems([FUNCTIONAL, weak], **STATED)
|
||||
assert any(
|
||||
"must be non_functional with at least one entry in `conditions`" in p
|
||||
for p in problems
|
||||
), weak
|
||||
|
||||
|
||||
def test_with_no_scope_or_nothing_binding_requirements_are_as_before():
|
||||
from app.workflows.chain import check_requirements
|
||||
|
||||
plain = [FUNCTIONAL]
|
||||
import json
|
||||
|
||||
assert check_requirements({"requirements.json": json.dumps(plain)}, None) == []
|
||||
assert requirement_problems(plain, stated={"intent": {"statement": "x"}}) == []
|
||||
|
||||
|
||||
def test_what_was_assumed_for_the_customer_does_not_bind_the_design():
|
||||
from app.scope import Catalogue, FieldSpec, Scope
|
||||
|
||||
catalogue = Catalogue(
|
||||
1,
|
||||
("workload",),
|
||||
{
|
||||
"net.topology": FieldSpec("net.topology", binds=True),
|
||||
"net.region": FieldSpec("net.region", binds=True),
|
||||
"notes": FieldSpec("notes"),
|
||||
},
|
||||
)
|
||||
scope = Scope(
|
||||
catalogue,
|
||||
"workload",
|
||||
values={"net.topology": "hub", "net.region": "x", "notes": "n"},
|
||||
sources={"net.region": "assumed"},
|
||||
)
|
||||
|
||||
assert scope.for_context()["binding"] == ["net.topology"]
|
||||
|
||||
@@ -14,6 +14,7 @@ from app.providers import load_provider
|
||||
from app.recorder import InMemoryRecorder
|
||||
from app.workflows.deliverables import DEPENDENCY_GRAPH
|
||||
from app.workflows.deliverables import DeliverableKind as K
|
||||
from tests.scoped import carrying
|
||||
|
||||
ROLES = {
|
||||
"orchestrator": RoleSpec("orchestrator", "Unused.", budget=40),
|
||||
@@ -45,6 +46,15 @@ run: {pre_approved: true, terraform_modules: false}
|
||||
"""
|
||||
|
||||
|
||||
STATED_CHOICES = [
|
||||
"estate.regions.primary",
|
||||
"estate.data_residency",
|
||||
"compliance.frameworks",
|
||||
"compliance.data_classification",
|
||||
"networking.topology",
|
||||
]
|
||||
|
||||
|
||||
class Recording(FakeAgentRuntime):
|
||||
def __init__(self, turns):
|
||||
super().__init__(turns)
|
||||
@@ -62,8 +72,13 @@ def engine(tmp_path) -> tuple[ExecutionEngine, Recording]:
|
||||
bundle = load_provider("gcp")
|
||||
runtime = Recording(
|
||||
{
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
**{
|
||||
spec.specialist: [[sample_submission(kind)]]
|
||||
for kind, spec in DEPENDENCY_GRAPH.items()
|
||||
},
|
||||
# The requirements carry every choice SPEC states, and a Check holds
|
||||
# the design to them.
|
||||
**carrying(STATED_CHOICES),
|
||||
}
|
||||
)
|
||||
built = ExecutionEngine(
|
||||
|
||||
@@ -78,3 +78,61 @@ def scoped_ledger(**options: Any): # type: ignore[no-untyped-def]
|
||||
if ledger.phase_status(PhaseId.SCOPING) is PhaseStatus.AWAITING_APPROVAL:
|
||||
ledger.approve(PhaseId.SCOPING, actor="user")
|
||||
return ledger
|
||||
|
||||
|
||||
def carrying(sources: list[str]) -> dict[str, list]:
|
||||
"""Scripted turns for the Specialists along the Requirements chain, in which one
|
||||
more requirement, NFR-2, carries what the customer stated (`sources`), and a
|
||||
Check, K-3, holds the design to it."""
|
||||
import json
|
||||
|
||||
from app.adk.fake_runtime import SAMPLE_DATA, Call
|
||||
|
||||
def submission(kind: DeliverableKind, **files: str) -> Call:
|
||||
call = sample_submission(kind)
|
||||
return Call(
|
||||
call.tool,
|
||||
{**call.args, "attachments": {**call.args["attachments"], **files}},
|
||||
)
|
||||
|
||||
requirements = json.loads(SAMPLE_DATA["requirements.json"])
|
||||
requirements.append(
|
||||
{
|
||||
"id": "NFR-2",
|
||||
"kind": "non_functional",
|
||||
"text": "The design honours what the customer stated.",
|
||||
"conditions": ["Each stated choice is configured as stated."],
|
||||
"from": sources,
|
||||
}
|
||||
)
|
||||
components = json.loads(SAMPLE_DATA["components.json"])
|
||||
components[0]["requirements"].append("NFR-2")
|
||||
graph = json.loads(SAMPLE_DATA["target-architecture.json"])
|
||||
graph["checks"].append(
|
||||
{
|
||||
"id": "K-3",
|
||||
"applies_to": "E-1",
|
||||
"source": "NFR-2",
|
||||
"statement": "the stated choices are configured",
|
||||
}
|
||||
)
|
||||
implementation = json.loads(SAMPLE_DATA["implementation.json"])
|
||||
implementation.append(
|
||||
{"check": "K-3", "file": "terraform/main.tf", "address": "example.run"}
|
||||
)
|
||||
verdicts = json.loads(SAMPLE_DATA["verdicts.json"])
|
||||
verdicts["checks"].append(
|
||||
{"check": "K-3", "verdict": "pass", "evidence": "Configured as stated."}
|
||||
)
|
||||
K = DeliverableKind
|
||||
files = {
|
||||
K.REQUIREMENTS: {"requirements.json": json.dumps(requirements)},
|
||||
K.TECHNICAL_DECOMPOSITION: {"components.json": json.dumps(components)},
|
||||
K.TARGET_ARCHITECTURE: {"target-architecture.json": json.dumps(graph)},
|
||||
K.DEPLOYMENT_GUIDANCE: {"implementation.json": json.dumps(implementation)},
|
||||
K.VALIDATION_RESULTS: {"verdicts.json": json.dumps(verdicts)},
|
||||
}
|
||||
return {
|
||||
DEPENDENCY_GRAPH[kind].specialist: [[submission(kind, **attached)]]
|
||||
for kind, attached in files.items()
|
||||
}
|
||||
|
||||
@@ -383,3 +383,72 @@ def test_the_cache_setting_resolves_under_the_repository_and_can_be_disabled():
|
||||
).terraform_plugin_cache()
|
||||
== "/var/cache/tf"
|
||||
)
|
||||
|
||||
|
||||
# --- only the providers the bundle pins ---------------------------------------------------
|
||||
|
||||
PINNED = {"cloud": {"source": "example/cloud", "version": "1.2.3"}}
|
||||
|
||||
|
||||
def provider_refusal(code: str, pinned=PINNED) -> str:
|
||||
from app.tools.local import check_providers
|
||||
|
||||
result = check_providers({"terraform/main.tf": code}, pinned)
|
||||
return result.message if result else ""
|
||||
|
||||
|
||||
def test_code_of_the_pinned_provider_alone_is_accepted():
|
||||
code = """
|
||||
terraform {
|
||||
required_providers {
|
||||
cloud = {
|
||||
source = "example/cloud"
|
||||
version = "1.2.3"
|
||||
}
|
||||
}
|
||||
}
|
||||
resource "cloud_service" "web" {}
|
||||
data "cloud_project" "this" {}
|
||||
resource "terraform_data" "marker" {}
|
||||
module "network" {
|
||||
source = "example-modules/network/cloud"
|
||||
version = "1.0.0"
|
||||
}
|
||||
"""
|
||||
|
||||
assert provider_refusal(code) == ""
|
||||
|
||||
|
||||
def test_a_resource_of_a_provider_that_is_not_pinned_is_refused():
|
||||
message = provider_refusal('resource "sourcehost_repository" "infra" {}')
|
||||
|
||||
assert message.startswith("Refused: the code uses resources of the sourcehost")
|
||||
assert "Only these providers may be used: example/cloud" in message
|
||||
|
||||
|
||||
def test_declaring_another_provider_is_refused():
|
||||
declared = """
|
||||
terraform {
|
||||
required_providers {
|
||||
sourcehost = {
|
||||
source = "integrations/sourcehost"
|
||||
}
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
assert "the provider integrations/sourcehost" in provider_refusal(declared)
|
||||
|
||||
|
||||
def test_with_nothing_pinned_there_is_nothing_to_hold_code_to():
|
||||
assert provider_refusal('resource "anything_at_all" "x" {}', pinned={}) == ""
|
||||
|
||||
|
||||
async def test_validation_refuses_a_foreign_provider_before_terraform_runs():
|
||||
runner = TerraformRunner(binary="no-such-binary", providers=PINNED)
|
||||
|
||||
result = await runner.validate(
|
||||
{"terraform/main.tf": 'resource "sourcehost_repository" "infra" {}'}
|
||||
)
|
||||
|
||||
assert result.message.startswith("Refused: the code uses")
|
||||
|
||||
Reference in New Issue
Block a user