fix: a product the customer's scope spec names is not a leak; record the first scoping runs

Case 08 failed requirements_name_no_products for "Shared VPC", which the
spec's stated topology names. Records cases 03 and 08 on the scoping branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-11 19:27:51 +01:00
parent 545097530b
commit c26b744b26
3 changed files with 57 additions and 2 deletions

View File

@@ -26,3 +26,30 @@ Not done here:
- A re-supplied file part-way through a conversation is not handled; the customer answers in chat. - A re-supplied file part-way through a conversation is not handled; the customer answers in chat.
Tests: `tests/execution/test_scope_spec.py` (12), and the existing suites as updated. Tests: `tests/execution/test_scope_spec.py` (12), and the existing suites as updated.
## First runs with real models, 2026-10-11
Two cases side by side (`20261011T172837Z` case 03, `20261011T172840Z` case 08). Both completed.
| | Case 03, a one-sentence request | Case 08, the landing-zone spec |
|---|---|---|
| Score | 100% | 91%; the one failure was the evaluation's, now fixed |
| Minutes | 38 | 55 |
| Cost, about | $5.42 | $8.76 |
| Revisions | 1 of the Target architecture | 2 of the Target architecture, 1 of the code |
| The scope analyst | 64 seconds, 12 requests, $0.07 | 96 seconds, 9 requests, $0.08 |
| The scope | 6 stated, 21 assumed | 44 stated, 11 assumed, 4 not applicable |
| The design | 22 Elements, 81 Checks | 49 Elements, 101 Checks, 718 lines of Terraform |
**Scoping works and is cheap:** about a minute and under ten cents. From one sentence the analyst recorded the six things the sentence says as the customer's and 21 as assumptions, each with a reason; nothing was presented as stated that was not. No essential field was left unknown in either.
**The landing zone, the agent's first platform foundation, came out as one:** folders, organisation policies, a service perimeter, an interconnect with four attachments, routers, customer-managed keys, workload identity federation for the pipelines, a workforce pool for sign-in, audit sinks. The customer's four requirement ids are each cited in the requirements. Nothing excluded was built: no container workloads. Regions and encryption are as stated. Sixteen questions for the customer were written out.
**What it shows is missing:**
- **A stated choice is not checked.** The spec says `shared_vpc_hub_and_spoke`. The design names a "Spoke Shared VPC" and builds a transit network and a spoke network, with no Shared VPC host project and nothing connecting hub to spoke. All 101 Checks passed, because no Check asks for the topology the customer stated. "A stated choice binds" is an instruction; nothing in code holds a design to it. This is what step 3 has to close: a stated field that shapes the design must be traceable to a Check.
- **The code brought in a provider the bundle does not pin.** A `github_repository` Element led to `integrations/github` in the Terraform. Validation downloads whatever provider the code names, as it once did modules.
- **The scope statement added an exclusion the customer did not state** (changes to on-premises systems). The analyst is told to use only what the spec holds.
- **The evaluation scored a stated choice as a leaked product.** "Shared VPC" in the requirements failed `requirements_name_no_products`, though the customer's spec names that topology. The check now counts what the spec states as the customer's.
Case 03's cost and its one Revision are in the range seen before scoping existed; one run cannot say what scoping changed downstream.

View File

@@ -285,11 +285,13 @@ async def score_execution(
) )
requirements = content(DeliverableKind.REQUIREMENTS) requirements = content(DeliverableKind.REQUIREMENTS)
# What the customer named themselves, in the request or in their scope spec. A
# choice they stated is a constraint to record, not a product leaking in.
stated = case.workflow_request + "\n" + case.scope.replace("_", " ")
leaked = [ leaked = [
product product
for product in case.required_products for product in case.required_products
if _mentions(requirements, product) if _mentions(requirements, product) and not _mentions(stated, product)
and not _mentions(case.workflow_request, product)
] ]
checks.append( checks.append(
Check( Check(

View File

@@ -820,3 +820,29 @@ async def test_a_case_with_no_standards_has_no_such_check():
result = await score_execution(CASE, execution(), HEADINGS, passing_terraform) result = await score_execution(CASE, execution(), HEADINGS, passing_terraform)
assert all(c.name != "code_follows_standards" for c in result.checks) assert all(c.name != "code_follows_standards" for c in result.checks)
async def test_a_product_the_customers_scope_spec_names_may_appear_in_requirements():
"""Case 08: the spec said shared_vpc_hub_and_spoke, and "Shared VPC" in the
requirements was scored as a product leaking in."""
named = {
"content": "## Functional requirements\nUse the Shared VPC topology stated.\n"
"## Current state\nNone.",
"attachments": {"requirements.json": SAMPLE_DATA["requirements.json"]},
}
with_spec = dataclasses.replace(
CASE,
required_products=("Shared VPC",),
scope="networking:\n topology: shared_vpc_hub_and_spoke\n",
)
without = dataclasses.replace(CASE, required_products=("Shared VPC",))
stated = await score_execution(
with_spec, execution(requirements=named), HEADINGS, passing_terraform
)
leaked = await score_execution(
without, execution(requirements=named), HEADINGS, passing_terraform
)
assert check(stated, "requirements_name_no_products").passed
assert not check(leaked, "requirements_name_no_products").passed