ALZ-F01 Discovery: package step, value scorecard, application grouping, cases

- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent f659564208
commit 9e503885cc
211 changed files with 18170 additions and 158 deletions

View File

@@ -4,6 +4,10 @@ For people who run, review or operate CP1 Discovery with the individual-agents p
| Document | For |
|---|---|
| [deployment.md](../../../docs/deployment.md) | Platform operators: building, pushing and deploying the eight agents with kustomize |
| [deployment.md](../../../docs/deployment.md) | Platform operators: building, pushing and deploying the agents with kustomize |
| [adr/](adr/README.md) | Product owner and reviewers: every change to the original ALZ-F01 requirements in this pattern |
| [run-context-field-review.md](run-context-field-review.md) | Product owner: ADR 0016's 51 run context fields, one by one, with a recommendation and a decision column |
| [discovery-value-plan.md](discovery-value-plan.md) | Product owner: what the output must give the Engagement Lead and Assessment, the scorecard baseline, and the phases to raise it |
| `<agent>/docs/user-guide.md` | Engagement leads: what each step does and how to read its output |
- [MCP tools and context](../../../docs/platform-issues/agent-tools-and-context/individual-agents-report-2026-11-01.md): why each agent holds its tools, how context carries between steps, and what to watch for.

View File

@@ -3,15 +3,42 @@
| | |
|---|---|
| Status | Proposed |
| Date | 2026-10-29 |
| Date | 2026-10-29, changed 2026-11-01 and 2026-11-03 |
| Decided by | Built as a default; awaiting product owner and platform confirmation |
| Requirements changed | DSC-08's note on sharing a model with any executor |
## Decision
- **Seven agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
- **The critic uses `claude-sonnet-5-5`.** Both models are deployed on the Azure AI resource behind the gateway.
- **All eight agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
- **The critic shares a model with the executors.** It was to use `claude-sonnet-5-5`, but the cluster's LiteLLM doesn't serve that (checked 2026-11-01), so the product owner moved it to `gpt-5.6-sol`.
## Change, 2026-11-03
**The critic now runs on `gpt-5.4`,** which the cluster's LiteLLM serves, and the eight other agents stay on
`gpt-5.6-sol`. The product owner asked for this after the first live GCP run (`exec_0f8629066e13`), whose critic
noted that it shared the executors' model. The critic states the models it and the executors use in `note`.
## Change, 2026-11-04
**The network agent also runs on `gpt-5.4`.** Two live Azure runs on an 81-record subscription
(`exec_c57354b25f8e`, `exec_beb8879632b8`) failed at the network step: `gpt-5.6-sol` returned HTTP 429 for its token
rate limit, with no fallback configured. The network agent sends the whole inventory on each of up to 25 model calls
while it pages through topology, which uses the per-minute budget in one step. `gpt-5.4` has its own quota. The
product owner approved this. As a result the critic shares a model with the network agent, and says so in `note`.
Follow-up: send the network agent only network-relevant records, which fixes the cause.
## Consequences
Confirm both names on the cluster's LiteLLM gateway before the first run.
- **The critic can share blind spots with the agents it judges.** It states this in its `note`, as DSC-08 requires, and the Engagement Lead's approval still gates its verdict (ADR 0006).
- **Revisit this when another model is on the gateway.** A different model for the critic remains the better design.
## Change, 2026-11-04: output limit 80,000 on the `gpt-5.6-sol` agents
At the product owner's direction ("do not change the model; increase the token limits within the agents to 80,000"),
`LITELLM_MAX_TOKENS` is `80000` (was `32000`) on the eight `gpt-5.6-sol` agents. Models are unchanged.
- The network agent and the critic stay at `16384`: `gpt-5.4` refuses a larger value ("max_tokens is too large",
`exec_b929f06cbb0c`).
- Not yet confirmed that `gpt-5.6-sol` accepts 80,000. A refusal would fail the planner, the first model step, at once.
- Azure OpenAI counts a request's `max_tokens` against the deployment's tokens-per-minute quota before the reply is
written, so a higher limit can bring the 429 sooner if `gpt-5.6-sol` is served from Azure.

View File

@@ -0,0 +1,17 @@
# ADR 0012: the Discovery report comes from code, and the approval gate moves to it
Status: Proposed. Date: 2026-11-01.
## Context
The workflow ended at the critic's verdict and produced no report for the Engagement Lead, who approved from the verdict alone.
## Decision
Add `cp1-discovery-report-agent` as the last step. Its `pre_process` hook renders the report in code, as Markdown and as a self-contained HTML page, and replies without calling a model, so it can only repeat what the steps recorded. The approval gate moves from the critic to this step, so approval is given on the whole report. This updates ADR 0006.
A Format for UI step after the gate takes `final_html`, so an approved run ends with an artifact of the clean page. This is on trial.
## Consequences
The report has no written narrative. The verdict still isn't enforced: a person can approve a `fail`, and the report marks one at the top. The saved workflow must be imported again, and a run that fails before the report step produces no report.

View File

@@ -0,0 +1,41 @@
# 0013. A `provider` input lets one workflow read the sample estate or a live cloud
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-03 |
| Decided by | Product owner, 2026-11-03: adapt both patterns, with a workflow value choosing the hyperscaler; GCP first |
| Requirements changed | DSC-01 to DSC-09: the scope, scan, classification, network and critic sources in a live run |
## Context
The Vault Service is running on kab-dev, and the CP1 tenant's Default Project holds connections for
`azure-discovery` and `gcp-discovery`. The agents called only the sample server.
The runtime offers every declared MCP tool to the model, and its `before_model` hook can only add instructions, so
a code-only choice of cloud isn't possible in a declarative agent (see the repository's
`docs/hyperscaler-interchange-review-note-2026-11-01.md`). The cloud servers stay standalone.
## Decision
- **A `provider` run input**, `sample`, `azure` or `gcp` (default `sample`), goes to every step. The runtime
enforces the enum. With `sample`, nothing changes.
- **Live tools.** The planner, scan, classification, network and critic agents also declare the live tools they
need from `azure-discovery` and `gcp-discovery`. A "Live provider" section in each prompt confines a live run to
`{provider}_*` tools. Inventory, CMDB, dependency and report agents call no live tool.
- **Live scope.** Scope comes from the cloud's own listings: servers, workloads, clusters and data stores, every
kind, every page. Each resource's own name or id is its scope entry id, and its region is its site.
- **Missing sources.** CMDB, ITSM, flows and schemas are declared `source_gaps` (`no_connection`).
- **Wrong cloud.** The critic fails a run in which any identifier belongs to the other cloud.
- **AWS later.** Its tools take `account_id` and `region`, and no AWS connection is stored yet.
## Consequences
- **Choosing the cloud rests on the prompt.** Both connections are in one project, so a wrong-cloud call would
succeed. The critic's identifier check is the backstop.
- **Large estates.** A live estate's tool output goes straight into each model's context. The GCP trial project
is about 40 KB; the Azure subscription was about 5.3 MB (most of it Advisor recommendations, which no agent
calls), and needs testing before it's relied on.
- **No call-log check.** The critic can't check tool calls in a live run (`call_log` and `guard_hits` go in
`not_checked`).
- **Every declared server must be registered and healthy** for the agents to start, the cloud servers included.

View File

@@ -0,0 +1,44 @@
# 0014. A Well-Architected review step in Discovery, judged in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-03 |
| Decided by | Product owner, 2026-11-03: expand Discovery to support the Well-Architected Framework (advised to place it in ALZ-F02; chose Discovery) |
| Requirements changed | New story ALZ-DSC-10 (to be created); the Discovery report's content for live runs |
## Context
The first live GCP run produced a thin report, and the owner wants a Well-Architected review of the provider as a
file of its own. Discovery's agents collected inventory, metadata and topology, but no policies or recommendations.
Azure's subscription has about 1,776 Advisor recommendations (3.2 MB), more than a model step can take.
## Decision
- **A new step, `cp1-well-architected-review-agent`** (ALZ-DSC-10), after the critic and before the report. It
answers from code in `pre_process` (`app/waf.py`), like the report agent (ADR 0012):
- It reads the in-scope resources, firewall or NSG rules, organisation policies, Azure resource locks and every
page of the provider's recommendations, through read-only tools on the existing servers.
- It applies fixed checks per pillar, rolls up recommendations by pillar and impact, and adds a guidance
reference per pillar from the provider's own server (`<provider>_search_well_architected`; changed on 2026-11-04,
when `cloud-knowledge` was retired).
- It renders `waf_markdown`, `waf_html` and `final_waf_html`.
- No model is involved. A pillar with no evidence is "not assessed", never a pass.
- With `provider` `sample`, it's "not applicable" and calls nothing.
- **Its page is released by the existing approval gate**, through its own Format for UI step (`s12`). The report
shows a pillar summary.
- **The report gains live sections:** estate overview, asset details from a new `provider_attributes` field (scan,
then inventory, which also fills Site), network topology from the network agent's `topology_summary`, the
Well-Architected summary, and a Mermaid dependency diagram. Sections empty only because their source was missing
collapse into one "Not examined" list.
## Consequences
- **The individual pattern has a second agent with code.** Its checks are unit-tested (`tests/test_waf.py`), and a
check change needs a test.
- **Judgement in Discovery.** This review judges configuration, which ALZ-F02's ALZ-ASM-05 also does for security.
ALZ-ASM-05 should take this review as input rather than repeat it.
- **Partial pillars.** Performance and sustainability rest on configuration and recommendations only: there's no
utilisation telemetry.
- **No automatic `.md` file.** The platform registers step outputs as `output.json`, so the Markdown is uploaded
after approval with `scripts/kab_publish_report.py` until the platform can save a field as a file.

View File

@@ -0,0 +1,57 @@
# 0015. A live run's inventory is built in batches, with at most three tries per batch
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-04 |
| Decided by | Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step" |
| Requirements changed | ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged |
## Context
The inventory agent built a live run's whole inventory in one model reply. Azure run `exec_cf87d8d3a563`
(2026-11-04) scanned 97 resources:
- The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token
output limit. Run `exec_beb8879632b8`'s 81 records had fitted.
- The output contract's retry returned `{"records": [], "sme_review_queue": []}` in 15 seconds. That met the schema,
so every later step ran on an empty estate and nothing failed.
A single reply caps the estate a run can hold, and a shortfall is not detected.
## Decision
- **Live runs (`provider` `azure` or `gcp`) answer from code in `pre_process`** (`app/business_logic.py`,
`app/inventory.py`), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and
`prompt.md` unchanged.
- **One model call per batch** of findings, through the `normalise_batch` sub-agent (`as_tool: false`, prompt
`prompts/normalise_batch.md`), one batch at a time. The batch size is `INVENTORY_BATCH_SIZE` (20).
- **The model judges; code copies and counts.**
- The model judges, per finding: `hostname`, `fqdn`, `asset_class`, `environment`, `os`, `services`, `facts`,
`fact_source` and `conflicts`.
- Code sets: `canonical_id` (`CI-001`… in the findings' order), `scope_entry_id`, `sources`, `match_basis`,
`provider_attributes`, `facts.site` (the region, with a GCP zone reduced to its region) and `confidence` (0.85
for one source; 0.79 with a conflict).
- Code also builds `sme_review_queue` (every record below 0.8).
- **Every reply is checked before it is kept.** It must be valid JSON (a markdown fence is tolerated). It must hold
exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's
types and enums. Each fact value must be found in the finding or the run context file.
- **At most three tries per batch** (`INVENTORY_MAX_TRIES`, 3 in all). A rejected reply or a failed model call (for
example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the
last reply was rejected.
- **A batch that fails all three tries fails the step.** The error names the batch and its findings. Later batches
are not asked, and no partial inventory is returned.
- **The whole inventory is checked** before the reply: one record per (de-duplicated) finding, and the output schema
in code, because the output contract cannot repair a `pre_process` reply.
## Consequences
- An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input
tokens each, well inside the model's per-minute token limit.
- A shortfall is a failed step, never an empty inventory passed as complete.
- Code-set fields can no longer drift from the finding, which also fixes `environment` being left `unknown` when the
context file gives one (the batch prompt reads `context.assets.<name>.environment`).
- Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's
19-minute budget (`AGENT_MAX_DURATION_SECONDS`).
- **Not changed:** the sample estate still builds its inventory in one reply. Batching it means merging one asset's
scan and CMDB records across applications, which needs its own design.

View File

@@ -0,0 +1,88 @@
# 0016. The run context file carries an intent, in the solution-architecture scope spec's format
| | |
|---|---|
| Status | Proposed |
| Date | 2026-11-04 |
| Decided by | Proposed at the product owner's request, 2026-11-04 ("create an ADR locally"). The owner described the idea and pointed to the solution-architecture agent's scope spec as a model. Not yet accepted, and nothing is built |
| Requirements changed | ALZ-DSC-01 (the planner reads the intent); ALZ-DSC-10 (the Well-Architected review is weighted and checked against it); the Discovery package and ALZ-F02 (both carry the intent forward). The run context file's layout ([run-context.template.yaml](../../workflows/run-context.template.yaml)) |
## Context
- Discovery always runs towards an intent: what the customer wants the estate to become. Today the run context file
holds that only as loose text that no step acts on: `plan.target` ("Azure landing zone"), `plan.objectives` and
`customer.regulatory`.
- The owner wants an explicit intent. It would describe the current architecture against it, and feed the
Well-Architected review with each hyperscaler's patterns and compliance policies, so that the gap to a target
architecture is clear.
- The solution-architecture agent (outcome pattern) already reads such a file: its **scope spec** (ADR-0008 in
`outcome-pattern/solution_architecture/`; reader `app/core/scope.py`; field catalogue
`skills/scoping/assets/catalogue.yaml`; example `eval/datasets/specs/landing-zone.yaml`). It has:
- fixed-value fields: `engagement.kind`, `estate.mode` (`greenfield`, `brownfield_expansion`, `migration`),
`compliance.data_classification`, `compliance.encryption.type`, `networking.exposure` and
`networking.egress_control.model`;
- a marker per unsettled field: `ask` (not known yet), `decide` (the agent may assume, and records it) or
`not_applicable`;
- requirements with ids (`REQ-01` …);
- a completed copy written beside the spec, with `discovery_gaps`, `provenance` and `readiness`.
- Two constraints apply:
- The `*_search_well_architected` tools refuse free-text topics, so customer details never reach a vendor's
search.
- Under charter rule 3, scope, budgets and approvals are platform-supplied facts.
## Decision (proposed)
1. **The intent uses the scope spec's format.** The run context file gains the scope spec's sections, read against
the same catalogue: `metadata`, `engagement`, `intent`, `estate`, `workload`, `compliance`, `networking`,
`governance` and `operations`, each at the catalogue's `schema_version`.
- Discovery's own sections stay as they are: `workflow_inputs`, `generated`, and `context` (`assets`, `owners`,
`business_services`, `scope`, `sources`).
- `customer.regulatory` and `plan.target` give way to `compliance.frameworks` and `intent`.
- One file then runs from Discovery through the package and Assessment to solution architecture.
2. **Discovery describes the current state against the intent; it doesn't design the target.**
- It fills or confirms the fields its evidence can settle, such as `estate.regions`,
`estate.existing_systems`, `networking.topology`, `networking.cidr_allocations`, `networking.exposure`,
`compliance.encryption.type` and `operations.monitoring`.
- It writes each settled value with its evidence and `provenance`.
- It lists what it couldn't settle under `discovery_gaps`.
- The target architecture is drafted downstream (solution architecture), not in Discovery.
3. **Only fixed-value fields drive the review.**
- The Well-Architected review weights pillars from the intent's requirements and fields.
- It checks each `compliance.frameworks` entry against the provider's own policy sets: Azure Policy
initiatives, GCP organisation policies, AWS Config conformance packs.
- It cites the `REQ-` id behind each gap.
- `intent.statement` and other free text are data. They are never sent in a tool call and never followed as
instructions (charter rule 1).
4. **The file never grants anything.**
- `run.pre_approved`, `run.allow_cloud_read` and any similar field are ignored by Discovery.
- What Discovery may read is the connection's scope root, and approval is the workflow's review gate.
- The context may narrow the scope (`context.scope`), never widen it.
5. **The patterns stay separate.** F01 adopts the file format and the catalogue as a shared contract. It doesn't
import the solution-architecture agent's code. The catalogue moves to a neutral shared location, or F01 pins a
copy with its `schema_version` checked.
## Consequences
- **What it enables:**
- The Discovery report and the Well-Architected review can state the gap between the current estate and the
intent, requirement by requirement.
- Assessment and solution architecture start from the same intent, with Discovery's evidence attached.
- Fields marked `ask` give the engagement a concrete question list.
- **Work if accepted:**
- the planner reads the intent and plans evidence for it (DSC-01);
- the review maps frameworks and pillars to the provider's checks and policy sets (DSC-10);
- the report gains a "current vs intent" section;
- the package carries the completed spec;
- `check_context_doc.py` validates the new sections against the catalogue;
- the template, the user guides, and stories DSC-01 and DSC-10 are updated, plus the F02 handover.
- **Risks:**
- The catalogue is owned by the solution-architecture pattern, so a change there now affects Discovery. The
shared `schema_version` check catches a mismatch.
- A long free-text statement adds prompt length and an injection surface. It's kept as data and summarised by
code, not passed to every step.
- **Open questions for the owner:**
- Where does the shared catalogue live?
- Must every Discovery run have an intent, or is it optional, with the review falling back to equal pillar
weights?
- Does Discovery write the completed copy of the spec (the scope spec's own convention), or return it only in
the package?

View File

@@ -0,0 +1,61 @@
# 0017. A live run's scope is every resource under the scope root, planned and checked in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-04 |
| Decided by | Product owner, 2026-11-04: "that scope needs to be extend to all resources" |
| Requirements changed | ALZ-DSC-01 (ADO 3214451): what a live run's scope entries are, and how the plan is made. ALZ-DSC-08 (ADO 3214458): what live coverage is measured against |
## Context
Azure run `exec_29ec31ba92ab` (2026-11-04) was scoped to the resource group `crucible-kab-dev`. That group holds 11
resources, but the run scanned, inventoried and reported 3: the AKS cluster, a storage account and a PostgreSQL
server.
- The planner built a live scope only from `list_servers`, `list_workloads`, `list_kubernetes_clusters` and
`list_data_stores`. Two Key Vaults, two managed identities, a DNS zone, an Event Grid topic, a private endpoint and
a network interface were never scope entries. They weren't declared as gaps either.
- The critic measured coverage with the same four tools, so it reported "coverage complete" and passed the run.
## Decision
- **Every resource under the connection's scope root is accounted for.** The planner's code lists the scope with
`<provider>_list_resources`, every page at 200 a page, and puts each resource in exactly one place
(`app/scope.py`):
- **a scope entry:** any resource that isn't structure or a child component;
- **folded into its parent, with the rule recorded:**
- Azure: disks, network interfaces, VM extensions, scale-set instances;
- GCP: disks, GKE node pools;
- **structure:** subscriptions, resource groups, management groups, projects, folders and organisations, which
are counted but aren't assets.
- **A live plan is built in code, with no model call** (`app/plan.py`, `pre_process`):
- one `scan_target` per scope entry, in batches of 10;
- a `classify_data` per data store;
- one `collect_network` and one `trace_dependencies`;
- `source_gaps` set by the same rules as before.
The run context still only narrows (`context.scope.regions`, `context.scope.exclude`), as declared gaps. The new
`scope_accounting` output gives `listed`, `scope_entries`, `folded` (each with its rule) and `structure`. The sample
estate keeps its model and prompt unchanged.
- **An incomplete listing fails the planner step.** A refused or failed page, or more than 100 pages, means no plan
is made, rather than a plan of part of the estate.
- **The critic counts coverage itself.** Its code lists the scope again with the same rules: `app/scope.py` is
identical in both agents, and a test compares the two files. It compares that listing with the findings and the
planner's declared gaps, and gives the model `live_scope`: counts, the uncovered ids (up to 200, with the exact
count) and any listing issues. The model copies it into `coverage_gaps` and `coverage_complete`. A listing issue
fails coverage.
## Consequences
- Supporting services are discovered, inventoried and reported: Key Vault, identities, DNS, messaging, private
endpoints, registries, monitoring and so on. The Well-Architected review and Assessment (ALZ-ASM-05) see the same
estate.
- The critic can't pass a run that skipped a resource type.
- **Scale:**
- At subscription scope, about 650 resources become about 600 scan tasks. The planner handles that in code.
- The scan executor (ALZ-DSC-02) still has a model call each resource's metadata in one request. That fits a
resource group, but not a large subscription.
- Batching the scan in code, as ADR 0015 did for the inventory, is the next change before a subscription-wide run.
- Folding is a fixed list. A new child-component type shows up as a scope entry until it's added, which over-counts
but never drops anything.

View File

@@ -0,0 +1,79 @@
# 0018. Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "the model should ingest the data and normalise it to the data schema that we own and judge". Plan and decisions D1–D4: `engineering_hub_responsiblities/reports/cp1/cmdb-upload-ingestion-plan-2026-11-05.md` |
| Requirements changed | ALZ-DSC-04 (ADO 3214454): a live run reads CMDB exports uploaded with the run. ALZ-DSC-01 (ADO 3214451): an uploaded CMDB is a source, not a gap |
## Context
- **A live run never read a CMDB.** `cmdb` was always a source gap, and the reconciliation step returned nothing.
- **A second uploaded file broke the run.** The upload step joins files with `\n\n---\n\n`. The planner, inventory
and report parsed the whole text as one YAML document, failed, and carried on as if no run context had been
given.
- **Exports differ in layout.** CMDB exports differ by customer and tool, so no column can be fixed in code.
## Decision
- **Uploads are listed in the run context.** The context's `uploads:` list names the files uploaded after it, in
order, each with a `source` (e.g. `cmdb`). `kab_run_workflow.py` uploads them. The shared `app/run_context.py`
(identical in the planner, inventory, reconciliation and report agents) splits the text into the context and
each listed file. It cuts a CSV, or each XLSX sheet, into a header line and one line per row. It knows no
column.
- **The model normalises; code judges.** The `normalise_cmdb_batch` sub-agent maps each batch of rows (25 by
default, each batch with its header) to `schemas/cmdb_ci.json`. Every string is copied verbatim; `environment`
and `lifecycle` are the only judgments; any other column can be kept in `other_attributes`. Code rejects a reply
unless:
- every row has exactly one result (or a reasoned `skip`);
- every value is found in its row;
- every key is a header column;
- every judgment is an allowed value.
A batch gets at most three tries (ADR 0015's rule). One that fails all three, or an upload the platform couldn't
parse, fails the step.
- **Injections:** a row is flagged when the model or a code pattern sees an instruction. The quoted text goes to
the critic (`cmdb_injections`) and to the report.
- **Reconciliation is code** (`app/reconcile.py`), against the step's own scope listing (`app/scope.py`, the
planner's rules) and the inventory:
- **match:** by cloud resource id, otherwise by name when exactly one listed resource has it;
- **stale attributes:**
- environment, compared at enum level (D1);
- location, compared with the record's site;
- a retired CI whose resource exists (`investigate`);
- owner and business service, when the record carries them;
- **ghost:** an unmatched, not-retired CI whose id lies in a scope container the listing shows (D2). It's
`investigate` outside `context.scope.regions`, in `exclude`, or with an unknown lifecycle, and otherwise
`retire_ci`. A CI with no cloud id is `unplaced` (D3), and one in another container is `out_of_scope`. Neither
is drift.
- **orphan:** an inventory record no CI matched (`add_ci`). Folded components are never orphans.
- **The planner** adds one `ingest_cmdb` task per uploaded CMDB file, and drops the `cmdb` gap when
`include_cmdb` is `yes` and a file is uploaded.
## Consequences
- **New layouts need no code change.** A new export layout, whether a ServiceNow CSV or an asset register XLSX
with other column names, is handled by the model's mapping. Code checks that mapping on every row.
- **The model never decides drift.** A manipulated normaliser can only return values that are in the row.
- **The step makes model calls** (one per batch) on a live run with a CMDB. Without one, it makes none.
- **Subscription scope is cautious.** At subscription scope, a CI in a resource group with no listed resources is
`out_of_scope`, not a ghost. Returning the connection's scope root from the existing discovery servers would
make this exact; that is left until it's needed.
- **Planner and step listings can differ.** The reconciliation step lists the scope again, so a resource created
between the planner's listing and this one is matched, not a ghost.
## Changes after the first live run (2026-11-05)
Run `exec_a0a3c486ec9b` (case `azure-crucible-kab-dev-cmdb`) passed 13 of 19 checks. Every planted CMDB result
was found except the environment difference.
- **`env` tags were redacted** by the discovery servers' shared `_scrub` (`env` is a secret key, for a container's
environment variables). The inventory then took `env: [redacted]` over `Environment: dev`, and 8 of 10 records
had environment `unknown`, so the planted environment difference couldn't be found. A single value under
`env` is now kept; a list or map under it is still redacted (azure-discovery 0.1.6, gcp-discovery 0.1.4,
aws-discovery 0.1.2).
- **Orphans name their resource** (`field` `resource`, `discovered_value` the resource id). Before, they named only
the inventory's canonical id (reconciliation 0.5.1).
- **The critic** no longer raises "drift with no CMDB source" when `source_gaps` has no `cmdb` (critic 0.5.2).
- **The report's drift table** shows both the asset and the CMDB record (report 0.7.2).

View File

@@ -0,0 +1,69 @@
# 0019. A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "the cmdb nodes should be separate from the hyperscaler but still be built in an equivalent way with children and parents at an edge"; "it's the same graph but different edge nodes". Plan and decisions D1–D7: `engineering_hub_responsiblities/reports/cp1/live-dependency-map-plan-2026-11-05.md` |
| Requirements changed | ALZ-DSC-06 (dependency mapping): a live run's graph comes from the scan's recorded dependencies and the CMDB's relationships. ALZ-DSC-04: CMDB relationships are read. ALZ-DSC-08: the critic checks dependencies in code |
## Context
Run `exec_93742bdd7e29` (case `azure-crucible-kab-dev-cmdb`) passed all 19 checks, and the critic failed it.
- **The dependency step never received the scan's findings.** The scan recorded 7 dependencies. The step rebuilt
edges from each record's network attributes, dropped 4 of them and added 2 VNet edges the scan never recorded.
- **Out-of-scope targets were bare names.** Dependencies outside the scope root (subnets in `crucible-net-dev`, a
public IP in `crucible-kab-dev-nodes`) came through as names, so the critic reported them as unsupported.
- **The critic was inconsistent.** It passed the same estate the day before, so the verdict depended on the
model's judgment that day.
- **The CMDB's own relationships were never read.**
## Decision
- **One graph, three kinds of node** (`app/graph.py` in the dependency agent, from `pre_process`, with no model
call):
- `resource`: a scope entry, i.e. its inventory record;
- `external`: a dependency target outside the scope root or not inventoried, with its container and what depends
on it (D1);
- `ci`: a judged CMDB CI, with its reconciliation result.
- **Every edge runs parent (`source`) to child (`target`):**
- **resource → resource or external:** one edge per entry in a finding's `dependencies`. They're typed by the
target's resource type, with confidence 0.8 and `auto_approved`, since they're the provider's own record (D3).
- **ci → ci:** a CMDB relationship. Confidence 0.7 and `sme_review_recommended`.
- **ci → resource, `represents`:** the reconciliation's match. `auto_approved` when matched by cloud id;
`sme_review_recommended` when matched by name.
- **Folded components:** a dependency on a resource's own child component (a NIC, a disk) is a
`folded_dependencies` entry, not an edge (D2).
- **Relationships are judged like CIs** (ADR 0018). A relationship export (`source: cmdb_relationships` in the
context's `uploads:`, D6), or a CI row's own parent or child columns (`related`), is normalised by the model to
`schemas/cmdb_relationship.json`. It's typed with the graph's edge types (D7), and every value is checked in code
against its row. An end that resolves to no CI (by `ci_ref`, or by unique name) is dangling, never a node. This
is done in the CMDB step (D5).
- **The comparison is a walk over the one graph.** For each ci → ci edge whose two CIs both represent resources:
- the cloud has the same resource → resource edge, in the same direction: `confirmed`. The cloud edge gains
the CMDB evidence, and its confidence becomes 0.95;
- otherwise: `cmdb_only`.
A cloud edge between two represented resources that no ci → ci edge matches is `discovered_only`. A relationship
with an end that has no resource is `not_comparable`.
- **The critic counts dependencies in code** (`live_dependencies`): every recorded dependency must be an edge (to a
record, or to an external node) or a folded dependency. It also treats `external`, `represents` and ci → ci edges
as supported by their own evidence.
- **The report:**
- one diagram, with each node drawn by kind: a resource as a box, external with a dashed outline, a CI as a
hexagon, and `represents` as dotted;
- tables of dependencies outside this discovery (with where to widen the scope, D4), the CMDB comparison, and
dangling relationships.
## Consequences
- No recorded dependency is dropped and none is invented. The critic's dependency verdict is the same on every run.
- An assessment sees what a landing zone depends on outside the scope, for example the shared VNet in
`crucible-net-dev`, and which resource group to add to discover it.
- The CMDB's picture and the cloud's are in one graph, so a CI's dependents, and an asset's dependents in either
source, are one walk.
- `discovered_only` is reported in the dependency step and the report. It doesn't add a drift item, because the
CMDB step (s4) runs before the graph (s7).
- Folding is a fixed list, so a dependency on another asset's child component (another VM's disk) is still folded,
not redirected to its parent: the listing doesn't say which parent owns it.

View File

@@ -0,0 +1,75 @@
# 0020. One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "lets go ahead with the enhancements when these are not combined and then later that can be decided"; the solution architecture agents are a separate stream, and Discovery's outputs are used only by the Assessment workflow. Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phases 0 and 1 |
| Requirements changed | ALZ-F02 ASM-01 (Discovery hand-off): F02 selects one package artifact rather than one per Discovery step (F02 ADR 0002). New step `s13`, agent `cp1-discovery-package-agent` |
## Context
- **F02 reads Discovery by hand.** Its Artifacts step needs each Discovery step's artifact id typed in, and every F02
agent receives all of them as one text, joined by `---` (engineering hub review
`reports/cp1/discovery-to-assessment-requirements-review-2026-11-04.md`, section 1).
- **Nothing measured whether a run was useful.** A case either passed its checks or it didn't. Nothing showed how
much of what F02 needs a run gives, how many decisions it leaves for a person, or what it cost.
- **KAB can't write a .zip.** It can expand a selected zip, but no step type creates one. Every step's output is kept
as a JSON artifact, `<execution_id>_<step_id>`.
- **Discovery and Assessment stay separate workflows** for now; combining them is a later decision.
- **The idea is borrowed, not the code.** The original `gcp_solution_architecture_agent` writes a deliverable folder
with an `execution.json` and measures its runs with limits per case. Those ideas are used here; nothing is ported,
and no hand-over to the solution architecture stream is built.
## Decision
1. **A package step, `s13`, answered in code** (`cp1-discovery-package-agent`, `app/package.py`, no model, no tools):
- It runs after the report's review point, beside the two page steps, so a package exists only for a run the
Engagement Lead let through.
- Its output holds every step's output fields, **copied unchanged**, as named parts: `plan`, `scan`, `inventory`,
`cmdb`, `classification`, `network`, `dependencies`, `critic`, `well_architected`, `report`. Parts are named by
meaning, because KAB renames step ids on import. The report part keeps `counts` and `report_markdown`, not the
HTML.
- Its `manifest` holds the engagement, run, execution id, provider, time, critic verdict, each agent's version
(a static input set when the workflow is built), the planner's source gaps, a `sha256` and item count per part,
the parts absent, `complete`, `problems` and the scorecard.
- `complete` is true only when the required parts (plan, inventory, classification, network, dependencies,
critic, report) are present and the critic passed. The CMDB and Well-Architected parts are optional.
- Each `sha256` is over the part's canonical JSON (`sort_keys`, `separators=(",", ":")`, UTF-8), so a consumer
can recompute it before trusting the part.
2. **A value scorecard** (`app/scorecard.py`), in the manifest and in `scripts/evaluate_workflow_run.py`, from the
same code:
- `f02_fields`: each F02 field (application, owner, criticality, environment, support status, end of support,
compliance scope, annual cost, data class), per record, as `filled`, `declared_gap`, `missing` or
`not_applicable`;
- `f02_run`: the run-level facts F02's ASM-01 checks;
- `decisions`: what the run leaves for a person, by kind;
- `findings`: how many findings name an owner and an action.
`filled` and `filled_or_declared` are both reported: a declared gap is honest, but it isn't value.
3. **Cases carry a value bar and limits.** `value:` (for example `f02_filled_min`) holds the baseline so a change
can't lower it; `limits:` bound tokens, model calls and agent seconds, read from the agents' logs by
`scripts/collect_run_usage.py`. Both fail as major.
4. **F02 is unchanged by this ADR.** Its Artifacts step can select the package today. Teaching F02 to verify the
manifest in code is F02's own change, with its own ADR.
## Consequences
- **First live runs** (2026-11-05, both after the Engagement Lead continued): `exec_14956d524a8b`
(`azure-crucible-kab-dev-cmdb`, 46 of 46 checks) and `exec_d1c242609f82` (`azure-crucible-kab-dev`, 19 of 19).
Both packages are complete, and every part's `sha256` recomputes.
- **Baseline:** 24% of F02's fields filled in both cases; 27% filled or declared with a CMDB, 51% without one (the
CMDB gap declares owners and applications); 6 of 7 run checks met (no applications list); 9 decisions with a CMDB;
findings 0% with an owner, 100% with an action. Owners are in the CMDB export but never reach the inventory
records, and nothing reads cost, lifecycle or compliance. These are phases 3 and 4 of the plan.
- **Size:** about 100 KB with a CMDB and 78 KB without, against KAB's 5 MB inline limit. A large estate will need
per-application summaries (plan phase 3) or a platform zip step.
- **The review record is implied, not explicit.** The package proves the run continued past review, but not who
approved it or their comments. That still needs the platform to expose review records to later steps.
- **Every Discovery output field now has a reader:** the context report's "unread" fields reach F02 through the
package, except the run's own ids.
- **The package input schema has no enums**, so the build-time contract check can never refuse a producer's value
at this step. A new field needs a row in `INPUTS`, its wiring in `build_workflow.py`, and a test checks the two
match.
- **ADR 0016 (Proposed) needs the owner's review:** it carries one run context file through to solution
architecture, which the decision of 2026-11-05 to keep the streams independent rules out. Plan phase 2 borrows
only the scope spec's field meanings, within F01.

View File

@@ -0,0 +1,59 @@
# 0021. A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "CMDB→tag→RG, review model" (code groups by CMDB business service, then the application tag, then resource group; the model proposes groups only for what's left, code judges them, and model-proposed groups are flagged for the Engagement Lead's review). Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phase 3 |
| Requirements changed | ALZ-DSC-03 and ALZ-F02 ASM-01 ("inventory (applications and assets)"): a live run lists applications, from the dependency step rather than the inventory. ALZ-DSC-04: each CI's result carries its owner, business service and support group. ALZ-DSC-06: the dependency step groups applications and names owners |
## Context
- **F02 needs applications.** Its planner groups by the Discovery `applications` list, else by `facts.business_service`,
and live runs had neither, so every asset was a gap (review of 2026-11-04, item 5). The value scorecard's baseline
([ADR 0020](0020.md)) showed application and owner missing on all 10 records of the CMDB case.
- **The evidence was there but never reached Assessment.** The uploaded CMDB names each CI's `owned_by` and
`business_service`, but the CMDB step's per-CI results dropped them. Every resource carries an `Application` tag.
- **The dependency step is the right place:** it runs after the CMDB step, already joins records to CIs (ADR 0019),
and is answered in code.
## Decision
1. **The CMDB step's `cis`** carry each CI's `owner`, `business_service` and `support_group`, verbatim (0.7.0).
2. **The dependency step groups each record, once,** in this order (`app/applications.py`, 0.6.0):
1. the matched CI's business service (`basis: cmdb_business_service`);
2. the resource's application tag (`Application`, `app`, `application_name`, `app_name`, `business_service`,
`service`, any case; `basis: tag`). **Tag alias:** when every CMDB-grouped member of one application carries
the same application tag, a tag-only record with that tag joins that application, flagged for review. A tag
two applications claim is no alias;
3. its resource group or project (`basis: container`), flagged for review;
4. for what's left only, the `propose_application_groups` sub-agent, from names, types and environments. Code
rejects the whole reply if any group names an asset not asked about, places an asset twice, or has an empty,
over-long or instruction-like name. Every model group is flagged for review; a rejected or failed reply leaves
its assets ungrouped, which the note says.
3. **Owners:** from the matched CI (`owner`, else `support_group`), else an owner tag. An application's owner is set
only when its members' owners agree (`owners_disagree` otherwise). No owner is inferred.
4. **Outputs:** `applications` (id, name, basis, members, owner, business service, evidence, `review_required` and
reasons), `asset_owners`, `ungrouped_assets` and `applications_note`. The report shows them after the inventory
and lists the groups to review (0.9.0); the package carries them as its `applications` part, and the scorecard
reads a record's application and owner from it (package 0.2.0, scorecard version 2).
## Consequences
- **Expected on the CMDB case:** one application, "KAB Platform", of 10 assets: 8 by the CMDB, 2 (the DNS zone and
the Event Grid topic, which have no CI) by the `Application: AppName1` tag every CMDB-grouped member shares,
flagged for review. Owner "Platform Engineering" for the 8 matched assets; the 2 orphans have none, correctly.
Scorecard: F02 fields filled from 24% to about 46%; run checks 7 of 7; findings with an owner from 0% to about 33% (the stale
attributes on owned assets; orphans and ghosts have none).
- **Without a CMDB:** one application, "AppName1", by tag, needing no review, and no owners: the CMDB gap declares
them.
- **The model is rarely asked:** on Azure and Google Cloud every resource has a resource group or project, so step 4
runs only for estates with no container. Its tests use a stand-in model.
- **Applications are not graph nodes.** They're a separate list, so no edge or node enum changed and no other step's
input schema had to. Membership edges can be added later if a consumer needs them in the graph.
- **The review is the Engagement Lead's,** at the report's review point. Assessment reads `review_required` and
can decide how to treat a group the lead didn't change.
- **Live, 2026-11-05:** `exec_58ba6a71d00b` (`azure-crucible-kab-dev-cmdb`) passed 56 of 56 checks: one application,
KAB Platform, of 10 assets, flagged for review; 8 owners; 46% of F02's fields filled; 7 of 7 run checks.
`exec_925194d79c47` (`azure-crucible-kab-dev`) passed 21 of 21: AppName1 by tag, no owners, 37% filled. Neither
run asked the grouping model.

View File

@@ -21,9 +21,19 @@ Each agent's `docs/adr/README.md` lists the records that change its own story.
| [0005](0005.md) | Inventory before CMDB reconciliation; the inventory gets CMDB records from `alz-inventory` | Proposed | DSC-03, 04 |
| [0006](0006.md) | The critic's verdict is gated by human approval, not by code | Proposed | DSC-08 |
| [0007](0007.md) | Agent names | Proposed | All |
| [0008](0008.md) | Models | Proposed | All |
| [0008](0008.md) | Models: executors on `gpt-5.6-sol`, the critic on `gpt-5.4` (2026-11-03) | Accepted | All |
| [0009](0009.md) | Deploy with kustomize; keys stay in Secrets | Accepted | — |
| [0010](0010.md) | Step outputs travel through the workflow; there is no shared store | Proposed | All |
| [0011](0011.md) | Uploaded files and project artifacts reach the planner and the inventory as text | Accepted | DSC-01, 03 |
| [0012](0012.md) | The Discovery report comes from code, and the approval gate moves to it | Accepted | report (new) |
| [0013](0013.md) | A `provider` input lets one workflow read the sample estate or a live cloud | Accepted | All |
| [0014](0014.md) | A Well-Architected review step in Discovery, judged in code | Accepted | DSC-10 (new), report |
| [0015](0015.md) | A live run's inventory is built in batches, at most three tries per batch; a batch that fails all three fails the step | Accepted | DSC-03 |
| [0016](0016.md) | The run context file carries an intent, in the solution-architecture scope spec's format | Proposed | DSC-01, DSC-10, package, F02 |
| [0017](0017.md) | A live run's scope is every resource under the scope root, planned and checked in code | Accepted | DSC-01, DSC-08 |
| [0018](0018.md) | Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code | Accepted | DSC-01, DSC-03, DSC-04, DSC-08, report |
| [0019](0019.md) | A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes | Accepted | DSC-04, DSC-06, DSC-08, report |
| [0020](0020.md) | One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code | Accepted | package (new), F02 ASM-01 |
| [0021](0021.md) | A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal | Accepted | DSC-03, DSC-04, DSC-06, report, package, F02 ASM-01 |
New records copy [template.md](template.md) and take the next number.

View File

@@ -0,0 +1,98 @@
# Discovery output value: plan
| | |
|---|---|
| Date | 2026-11-05 |
| Status | Phases 0 and 1 live ([ADR 0020](adr/0020.md)); the rest is planned |
| Decided by | Product owner, 2026-11-05: go ahead, with Discovery and Assessment kept as separate workflows; combining them is decided later |
## 1. Purpose and boundaries
Make the live Discovery workflow's output useful to the people and workflow that consume it, using ideas from the
original `gcp_solution_architecture_agent`.
- **Consumers:** the Engagement Lead at the review point, and the Assessment workflow (ALZ-F02), assuming F02 is
still required. No other workflow or pattern consumes Discovery's output.
- **Separate streams:** the solution architecture agent (`outcome-pattern/`) and the sub-agent pattern are
independent. No hand-over is built to them; only ideas are borrowed from the original agent, never its code or
engine.
- **The owner's order holds:** Azure working (done 2026-11-05), then the package step, then Discovery meets every
F02 requirement.
- **Discovery and Assessment stay separate workflows.** Whether to also offer one combined run is decided later.
## 2. What "meaningful value" means
| Consumer | The output must answer |
|---|---|
| Engagement Lead | What's there, what's wrong, what's missing, and what must be decided |
| F02 Assessment | For each application: its assets, dependencies, support status, data class, compliance, cost and owner, each with evidence |
The **value scorecard** (`cp1-discovery-package-agent/app/scorecard.py`) measures this for every run, in code: F02
fields filled, declared or missing; F02's run-level checks; decisions left for a person; findings with an owner and
an action. Cases also carry limits on tokens, model calls and agent seconds.
**Baseline**, case `azure-crucible-kab-dev-cmdb`, run `exec_14956d524a8b`, 2026-11-05 (46 of 46 checks):
| Measure | Value |
|---|---|
| F02 fields filled | 24% |
| F02 fields filled or declared | 27% |
| Missing everywhere | application, owner, support status, end of support, compliance scope, annual cost |
| F02 run checks | 6 of 7 (no applications list) |
| Decisions for a person | 9 |
| Findings with an owner / an action | 0% / 100% |
| Cost | about 106k tokens, 10 model calls, 108 agent seconds |
| Package size | about 100 KB |
## 3. Borrowed from the original
| Idea | In the original | In Discovery |
|---|---|---|
| Field meanings: `ask`, `decide`, `not_applicable`, provenance, a catalogue, a `check` command | ADR-0008, `app/scope.py` | The run context (phase 2) |
| An explicit graph, and an id chain code won't break | ADR-0005, the requirements chain | Application nodes in the ADR 0019 graph (phase 3); evidence on every finding (phase 4) |
| A deliverable manifest; customer questions | `execution.json`, `customer-questions.md` | The package manifest (phase 1); decisions needed (phase 4) |
| A Baseline and its changes | ADR-0006, `changes.json` | Re-discovery against the last approved run (phase 5) |
| An evaluation harness | `solution_eval`, judge calibration, limits | Value bars and limits (phase 0); a judge (phase 6) |
| Lessons and overlays | ADR-0004 | Phase 7 |
**Left behind:** its own server, database, LangGraph and `interrupt()` (they conflict with KAB); one model for every
role (Discovery's critic stays on a different model); Jev and unscrubbed free-text search (data would leave the
gateways).
## 4. Phases
Sizes are relative. "F02-only" marks work that only F02 justifies.
| # | Phase | Size | What | Done when | Status |
|---|---|---|---|---|---|
| 0 | Value baseline | S | The scorecard; `value:` bars and `limits:` in both Azure cases; `scripts/collect_run_usage.py` | Both cases have a baseline | **built** (ADR 0020) |
| 1 | Discovery package | M, F02-only | Step `s13`, `cp1-discovery-package-agent`: every part unchanged, a manifest with sha256 per part, completeness and the scorecard | An F02 run starts from one artifact id | **live** (ADR 0020): both Azure cases pass with complete packages; F02's own change to verify the manifest is next |
| 2 | Run context field meanings | M | A catalogue for `context.yaml` (value, `ask`, `decide`, `not_applicable`; essential, important, optional) covering compliance scope, criticality, owner, business services; `check_context.py`; provenance per fact; assumptions shown apart | A `decide` gives a visible assumption; an unset essential field stops the run before it scans | todo. ADR 0016 (Proposed) needs the owner's review first: it carries the file to solution architecture |
| 3 | Close F02's data gaps | L, mostly F02-only | Applications as graph nodes (CMDB business service, then tags, then resource group or naming, then model-proposed and judged in code); CMDB owners onto records; end-of-life rules in code over `provider_attributes`; cost through fixed-shape queries on the existing discovery servers; per-application summaries for size; the repositories source | The Azure case reaches the agreed F02 bar | **applications and owners built** ([ADR 0021](adr/0021.md)): CMDB→tag→container→judged model, owners from the CMDB or an owner tag; expected 46% filled and 7 of 7 run checks. End-of-life, cost and per-application summaries still todo |
| 4 | A report that leads to decisions | M | First page: decisions needed, top risks, open questions. Each finding has a severity, an owner (else "unowned"), an action and its consumer; code refuses a finding without evidence or an action | 100% of findings with an owner and an action | todo |
| 5 | Re-discovery against a Baseline | M | A run may name an earlier approved execution; a code-only diff step gives added, removed and changed assets, edges and drift | Identical runs give an empty diff; a planted change shows | todo |
| 6 | Evaluation harness | M | A report judge on a different model, calibrated on 10–20 labelled sections; runs with and without a feature | Each phase's effect is measured | todo |
| 7 | Lessons and overlays | M | Repeated critic findings become approved rules; customer standards arrive as overlays | — | later |
Phases 0, 2, 4, 5 and 6 are worth doing whatever happens to F02; 1 and 3 are justified by F02.
## 5. Decisions for the owner
1. **Is F02 still required?** It gates the rest of phase 3.
2. **ADR 0016:** the owner will go through its fields one by one before phase 2 starts: [run-context-field-review.md](run-context-field-review.md).
3. ~~**Application grouping**~~ decided 2026-11-05: CMDB, then tag, then resource group, then a judged model
proposal, with model groups reviewed (ADR 0021).
4. **Cost and end-of-life sources:** cost queries on the existing servers (needs Cost Management Reader or the
equivalent) or an uploaded file; end-of-life from a table in code or the provider's lifecycle data.
5. **The value bar** that counts as "Discovery meets F02".
6. **Later:** a combined Discovery and Assessment run, beside the separate workflows.
## 6. Risks
| Risk | Mitigation |
|---|---|
| The package outgrows KAB's 5 MB inline limit | Per-application summaries (phase 3), or a platform zip step |
| Wrong application groups mislead F02's scoring | Checks in code, the critic, and review of model-proposed groups |
| Cost and lifecycle need permissions beyond Reader | Update `docs/customer-cloud-access.md`; a missing permission is a declared gap |
| Borrowing turns into porting | Each borrowing gets its own ADR; ideas only |
| Model cost grows with new steps | The limits in each case; code-only steps wherever possible |

View File

@@ -0,0 +1,99 @@
# Run context: field-by-field review for ADR 0016
| | |
|---|---|
| Date | 2026-11-05 |
| For | The product owner, going through [ADR 0016](adr/0016.md) (Proposed) field by field before value plan phase 2 |
| Source | The scope spec catalogue the ADR borrows (`outcome-pattern/solution_architecture/.../skills/scoping/assets/catalogue.yaml`), 51 fields |
| Status | Recommendations only. Nothing is built from this until the decisions column is filled in |
## How to read it
- **Boundary (decided 2026-11-05):** Discovery's output feeds only Assessment (F02). The solution architecture stream
is separate, so nothing here is a hand-over to it, and F01 would keep its **own** copy of whatever fields it adopts,
with its own catalogue. ADR 0016's points 1 and 5 (one file through to solution architecture; a shared catalogue)
would be dropped.
- **Who settles it:**
- **Evidence:** Discovery can read the current value from the cloud or the CMDB, and records it with its
provenance. The customer may still state a target; Discovery then reports current against stated.
- **Customer:** only the customer can say it. Discovery reads it and uses it; it can be `ask`, `decide` or
`not_applicable`.
- **Not Discovery:** a design or delivery choice. It belongs to a later workflow, not to this file.
- **Used by:** the step or consumer that would act on it today or in a planned phase.
- **Recommendation:** **Keep** (adopt now), **Later** (worth having, no consumer yet), **Drop** (not Discovery's, or
already covered).
Discovery's own sections stay as they are: `workflow_inputs`, `generated`, `uploads`, `customer`, `plan`, and
`context` (`scope`, `sources`, `owners`, `business_services`, `assets`).
## Fields
| # | Field | What it is | Who settles it | Used by | Recommendation | Decision |
|---|---|---|---|---|---|---|
| 1 | `metadata.customer_case` | The engagement's short reference | Customer | All | **Drop**: `workflow_inputs.engagement_id` already is this | |
| 2 | `engagement.kind` | Workload, platform foundation or migration | Customer | Planner (what's essential), F02 | **Keep** | |
| 3 | `intent.statement` | What the customer wants, in their words | Customer | Report (shown as data) | **Keep**, as data only: never sent in a tool call, never passed to every step | |
| 4 | `intent.requirements` | Stated requirements with ids (`REQ-01`) | Customer | Well-Architected review (cite the `REQ-` behind a gap), report | **Keep** | |
| 5 | `intent.exclusions` | What's out of scope | Customer | Planner | **Drop**: `context.scope.exclude` already holds this for Discovery | |
| 6 | `intent.success_criteria` | How the customer judges success | Customer | — | **Later** | |
| 7 | `estate.cloud_provider` | The provider | Customer | — | **Drop**: `workflow_inputs.provider` is this | |
| 8 | `estate.mode` | Greenfield, brownfield expansion or migration | Customer | Planner, F02 | **Keep** | |
| 9 | `estate.regions.primary` | The main region | Evidence | Planner, report | **Keep**, settled from evidence; `context.scope.regions` stays the scan's filter | |
| 10 | `estate.regions.secondary` | A second region | Evidence | Report | **Keep**, settled from evidence | |
| 11 | `estate.regions.resiliency_tier` | Single region, warm standby or active-active | Evidence (current); customer (target) | Well-Architected reliability | **Later** | |
| 12 | `estate.data_residency` | Where data must stay | Customer | F02 (compliance), Well-Architected | **Keep** | |
| 13 | `estate.existing_systems` | What exists today | Evidence | — | **Drop** as a field: it is Discovery's whole output (inventory, applications) | |
| 14 | `workload.users.who` | Who uses it | Customer | — | **Later** | |
| 15 | `workload.users.count` | How many users | Customer | — | **Later** | |
| 16 | `workload.users.peak_concurrent` | Peak concurrent users | Customer | — | **Later** | |
| 17 | `workload.availability.target` | Availability, as a percentage | Customer | Well-Architected reliability, F02 risk | **Keep**, optional | |
| 18 | `workload.availability.hours` | Business hours, extended or always | Customer | F02 risk | **Later** | |
| 19 | `workload.availability.recovery_time` | RTO | Customer | Well-Architected reliability, F02 | **Keep**, optional | |
| 20 | `workload.availability.recovery_point` | RPO | Customer | Well-Architected reliability (backup checks), F02 | **Keep**, optional | |
| 21 | `compliance.industry` | The customer's industry | Customer | Report | **Drop**: `customer.industry` already holds it | |
| 22 | `compliance.frameworks` | Regulations and standards that apply | Customer | F02 `compliance_scope` (gap 11), Well-Architected policy sets | **Keep**, essential: replaces `customer.regulatory` | |
| 23 | `compliance.data_classification` | How sensitive the data is | Customer | Classification step (stated against found), F02 | **Keep** | |
| 24 | `compliance.encryption.type` | Provider- or customer-managed keys | Evidence | Well-Architected security | **Keep**, settled from evidence | |
| 25 | `compliance.encryption.key_rotation_days` | Key rotation period | Evidence (partly) | Well-Architected security | **Later** | |
| 26 | `compliance.audit_logging.retention_days` | Audit log retention | Evidence (partly) | Well-Architected operations | **Later** | |
| 27 | `compliance.audit_logging.immutable_retention_policy` | Undeletable audit logs | Evidence (partly) | Well-Architected security | **Later** | |
| 28 | `networking.exposure` | Internal only or internet-facing | Evidence | Network step, Well-Architected security | **Keep**, settled from evidence | |
| 29 | `networking.topology` | The network topology | Evidence | Network step, report | **Keep**, settled from evidence | |
| 30 | `networking.hybrid_connectivity.enabled` | Private links to on-premises or another cloud | Evidence | Network step, F02 | **Keep**, settled from evidence | |
| 31 | `networking.hybrid_connectivity.type` | VPN, ExpressRoute, Interconnect | Evidence | Network step | **Keep**, settled from evidence | |
| 32 | `networking.hybrid_connectivity.bandwidth_gbps` | Bandwidth | Evidence (partly) | — | **Later** | |
| 33 | `networking.egress_control.model` | Direct, provider-managed or a central appliance | Evidence | Network step, Well-Architected security | **Keep**, settled from evidence | |
| 34 | `networking.egress_control.appliance_vendor` | Which appliance | Evidence (partly) | — | **Later** | |
| 35 | `networking.cidr_allocations` | Allocated IP ranges | Evidence | Network step (address overlap) | **Keep**, settled from evidence | |
| 36 | `governance.identity_source_of_truth` | The identity provider | Customer | — | **Later** | |
| 37 | `governance.pipeline_authentication` | How pipelines authenticate | Not Discovery | — | **Drop** | |
| 38 | `governance.environment_tiers` | Which environments exist | Evidence | Inventory (environment), F02 | **Keep**, settled from evidence (environment tags) | |
| 39 | `operations.operating_model` | Who runs it day to day | Customer | Owners (ADR 0021), F02 | **Later**: owners now come from the CMDB and tags | |
| 40 | `operations.support_hours` | Supported hours | Customer | F02 risk | **Later** | |
| 41 | `operations.monitoring` | Monitoring and alerting | Evidence | Well-Architected operations | **Keep**, settled from evidence | |
| 42 | `operations.change_process` | How changes are approved | Customer, or ITSM | F02 change frequency | **Later**, with an ITSM source | |
| 43 | `integrations` | Other systems it exchanges data with | Evidence | Dependency graph (external nodes) | **Drop** as a field: the dependency graph holds it | |
| 44 | `delivery.target_start_date` | When work starts | Not Discovery | — | **Drop** | |
| 45 | `delivery.window_weeks` | The deadline | Not Discovery | — | **Drop** | |
| 46 | `delivery.commercial_model` | How the work is contracted | Not Discovery | — | **Drop** | |
| 47 | `delivery.iac_tooling` | Infrastructure-as-code tool | Not Discovery | — | **Drop** | |
| 48 | `delivery.ci_cd` | Pipeline tool | Not Discovery | — | **Drop** | |
| 49 | `delivery.team_size` | Engineers building it | Not Discovery | — | **Drop** | |
| 50 | `cost.monthly_budget` | Monthly running budget | Customer | F02 TCO (indicative) | **Later**: F02 uses its rate card | |
| 51 | `constraints.products_required`, `products_forbidden` | Mandated or ruled-out products | Not Discovery | — | **Drop** | |
## Totals, as recommended
| Recommendation | Fields |
|---|---|
| Keep | 21 (10 the customer states, 11 settled from evidence) |
| Later | 16 |
| Drop | 14 (6 already covered by Discovery's own sections or outputs, 8 delivery or design) |
## Also to decide
1. **Must every run have an intent,** or is it optional, with the Well-Architected review weighting pillars equally
when there's none? Recommended: optional.
2. **Where the completed values go:** a completed copy of the context file, or only in the package (a new `context`
part). Recommended: the package only, so there's one artifact for F02.
3. **ADR 0016's fate:** rewrite it to the fields kept here, within F01 and F02, with no shared catalogue.