fix: a product the customer's scope spec names is not a leak; record the first scoping runs
Case 08 failed requirements_name_no_products for "Shared VPC", which the spec's stated topology names. Records cases 03 and 08 on the scoping branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -26,3 +26,30 @@ Not done here:
|
||||
- A re-supplied file part-way through a conversation is not handled; the customer answers in chat.
|
||||
|
||||
Tests: `tests/execution/test_scope_spec.py` (12), and the existing suites as updated.
|
||||
|
||||
## First runs with real models, 2026-10-11
|
||||
|
||||
Two cases side by side (`20261011T172837Z` case 03, `20261011T172840Z` case 08). Both completed.
|
||||
|
||||
| | Case 03, a one-sentence request | Case 08, the landing-zone spec |
|
||||
|---|---|---|
|
||||
| Score | 100% | 91%; the one failure was the evaluation's, now fixed |
|
||||
| Minutes | 38 | 55 |
|
||||
| Cost, about | $5.42 | $8.76 |
|
||||
| Revisions | 1 of the Target architecture | 2 of the Target architecture, 1 of the code |
|
||||
| The scope analyst | 64 seconds, 12 requests, $0.07 | 96 seconds, 9 requests, $0.08 |
|
||||
| The scope | 6 stated, 21 assumed | 44 stated, 11 assumed, 4 not applicable |
|
||||
| The design | 22 Elements, 81 Checks | 49 Elements, 101 Checks, 718 lines of Terraform |
|
||||
|
||||
**Scoping works and is cheap:** about a minute and under ten cents. From one sentence the analyst recorded the six things the sentence says as the customer's and 21 as assumptions, each with a reason; nothing was presented as stated that was not. No essential field was left unknown in either.
|
||||
|
||||
**The landing zone, the agent's first platform foundation, came out as one:** folders, organisation policies, a service perimeter, an interconnect with four attachments, routers, customer-managed keys, workload identity federation for the pipelines, a workforce pool for sign-in, audit sinks. The customer's four requirement ids are each cited in the requirements. Nothing excluded was built: no container workloads. Regions and encryption are as stated. Sixteen questions for the customer were written out.
|
||||
|
||||
**What it shows is missing:**
|
||||
|
||||
- **A stated choice is not checked.** The spec says `shared_vpc_hub_and_spoke`. The design names a "Spoke Shared VPC" and builds a transit network and a spoke network, with no Shared VPC host project and nothing connecting hub to spoke. All 101 Checks passed, because no Check asks for the topology the customer stated. "A stated choice binds" is an instruction; nothing in code holds a design to it. This is what step 3 has to close: a stated field that shapes the design must be traceable to a Check.
|
||||
- **The code brought in a provider the bundle does not pin.** A `github_repository` Element led to `integrations/github` in the Terraform. Validation downloads whatever provider the code names, as it once did modules.
|
||||
- **The scope statement added an exclusion the customer did not state** (changes to on-premises systems). The analyst is told to use only what the spec holds.
|
||||
- **The evaluation scored a stated choice as a leaked product.** "Shared VPC" in the requirements failed `requirements_name_no_products`, though the customer's spec names that topology. The check now counts what the spec states as the customer's.
|
||||
|
||||
Case 03's cost and its one Revision are in the range seen before scoping existed; one run cannot say what scoping changed downstream.
|
||||
|
||||
Reference in New Issue
Block a user