ALZ-F01 Discovery: package step, value scorecard, application grouping, cases

- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent f659564208
commit 9e503885cc
211 changed files with 18170 additions and 158 deletions

View File

@@ -3,15 +3,42 @@
| | |
|---|---|
| Status | Proposed |
| Date | 2026-10-29 |
| Date | 2026-10-29, changed 2026-11-01 and 2026-11-03 |
| Decided by | Built as a default; awaiting product owner and platform confirmation |
| Requirements changed | DSC-08's note on sharing a model with any executor |
## Decision
- **Seven agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
- **The critic uses `claude-sonnet-5-5`.** Both models are deployed on the Azure AI resource behind the gateway.
- **All eight agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
- **The critic shares a model with the executors.** It was to use `claude-sonnet-5-5`, but the cluster's LiteLLM doesn't serve that (checked 2026-11-01), so the product owner moved it to `gpt-5.6-sol`.
## Change, 2026-11-03
**The critic now runs on `gpt-5.4`,** which the cluster's LiteLLM serves, and the eight other agents stay on
`gpt-5.6-sol`. The product owner asked for this after the first live GCP run (`exec_0f8629066e13`), whose critic
noted that it shared the executors' model. The critic states the models it and the executors use in `note`.
## Change, 2026-11-04
**The network agent also runs on `gpt-5.4`.** Two live Azure runs on an 81-record subscription
(`exec_c57354b25f8e`, `exec_beb8879632b8`) failed at the network step: `gpt-5.6-sol` returned HTTP 429 for its token
rate limit, with no fallback configured. The network agent sends the whole inventory on each of up to 25 model calls
while it pages through topology, which uses the per-minute budget in one step. `gpt-5.4` has its own quota. The
product owner approved this. As a result the critic shares a model with the network agent, and says so in `note`.
Follow-up: send the network agent only network-relevant records, which fixes the cause.
## Consequences
Confirm both names on the cluster's LiteLLM gateway before the first run.
- **The critic can share blind spots with the agents it judges.** It states this in its `note`, as DSC-08 requires, and the Engagement Lead's approval still gates its verdict (ADR 0006).
- **Revisit this when another model is on the gateway.** A different model for the critic remains the better design.
## Change, 2026-11-04: output limit 80,000 on the `gpt-5.6-sol` agents
At the product owner's direction ("do not change the model; increase the token limits within the agents to 80,000"),
`LITELLM_MAX_TOKENS` is `80000` (was `32000`) on the eight `gpt-5.6-sol` agents. Models are unchanged.
- The network agent and the critic stay at `16384`: `gpt-5.4` refuses a larger value ("max_tokens is too large",
`exec_b929f06cbb0c`).
- Not yet confirmed that `gpt-5.6-sol` accepts 80,000. A refusal would fail the planner, the first model step, at once.
- Azure OpenAI counts a request's `max_tokens` against the deployment's tokens-per-minute quota before the reply is
written, so a higher limit can bring the 429 sooner if `gpt-5.6-sol` is served from Azure.

View File

@@ -0,0 +1,17 @@
# ADR 0012: the Discovery report comes from code, and the approval gate moves to it
Status: Proposed. Date: 2026-11-01.
## Context
The workflow ended at the critic's verdict and produced no report for the Engagement Lead, who approved from the verdict alone.
## Decision
Add `cp1-discovery-report-agent` as the last step. Its `pre_process` hook renders the report in code, as Markdown and as a self-contained HTML page, and replies without calling a model, so it can only repeat what the steps recorded. The approval gate moves from the critic to this step, so approval is given on the whole report. This updates ADR 0006.
A Format for UI step after the gate takes `final_html`, so an approved run ends with an artifact of the clean page. This is on trial.
## Consequences
The report has no written narrative. The verdict still isn't enforced: a person can approve a `fail`, and the report marks one at the top. The saved workflow must be imported again, and a run that fails before the report step produces no report.

View File

@@ -0,0 +1,41 @@
# 0013. A `provider` input lets one workflow read the sample estate or a live cloud
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-03 |
| Decided by | Product owner, 2026-11-03: adapt both patterns, with a workflow value choosing the hyperscaler; GCP first |
| Requirements changed | DSC-01 to DSC-09: the scope, scan, classification, network and critic sources in a live run |
## Context
The Vault Service is running on kab-dev, and the CP1 tenant's Default Project holds connections for
`azure-discovery` and `gcp-discovery`. The agents called only the sample server.
The runtime offers every declared MCP tool to the model, and its `before_model` hook can only add instructions, so
a code-only choice of cloud isn't possible in a declarative agent (see the repository's
`docs/hyperscaler-interchange-review-note-2026-11-01.md`). The cloud servers stay standalone.
## Decision
- **A `provider` run input**, `sample`, `azure` or `gcp` (default `sample`), goes to every step. The runtime
enforces the enum. With `sample`, nothing changes.
- **Live tools.** The planner, scan, classification, network and critic agents also declare the live tools they
need from `azure-discovery` and `gcp-discovery`. A "Live provider" section in each prompt confines a live run to
`{provider}_*` tools. Inventory, CMDB, dependency and report agents call no live tool.
- **Live scope.** Scope comes from the cloud's own listings: servers, workloads, clusters and data stores, every
kind, every page. Each resource's own name or id is its scope entry id, and its region is its site.
- **Missing sources.** CMDB, ITSM, flows and schemas are declared `source_gaps` (`no_connection`).
- **Wrong cloud.** The critic fails a run in which any identifier belongs to the other cloud.
- **AWS later.** Its tools take `account_id` and `region`, and no AWS connection is stored yet.
## Consequences
- **Choosing the cloud rests on the prompt.** Both connections are in one project, so a wrong-cloud call would
succeed. The critic's identifier check is the backstop.
- **Large estates.** A live estate's tool output goes straight into each model's context. The GCP trial project
is about 40 KB; the Azure subscription was about 5.3 MB (most of it Advisor recommendations, which no agent
calls), and needs testing before it's relied on.
- **No call-log check.** The critic can't check tool calls in a live run (`call_log` and `guard_hits` go in
`not_checked`).
- **Every declared server must be registered and healthy** for the agents to start, the cloud servers included.

View File

@@ -0,0 +1,44 @@
# 0014. A Well-Architected review step in Discovery, judged in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-03 |
| Decided by | Product owner, 2026-11-03: expand Discovery to support the Well-Architected Framework (advised to place it in ALZ-F02; chose Discovery) |
| Requirements changed | New story ALZ-DSC-10 (to be created); the Discovery report's content for live runs |
## Context
The first live GCP run produced a thin report, and the owner wants a Well-Architected review of the provider as a
file of its own. Discovery's agents collected inventory, metadata and topology, but no policies or recommendations.
Azure's subscription has about 1,776 Advisor recommendations (3.2 MB), more than a model step can take.
## Decision
- **A new step, `cp1-well-architected-review-agent`** (ALZ-DSC-10), after the critic and before the report. It
answers from code in `pre_process` (`app/waf.py`), like the report agent (ADR 0012):
- It reads the in-scope resources, firewall or NSG rules, organisation policies, Azure resource locks and every
page of the provider's recommendations, through read-only tools on the existing servers.
- It applies fixed checks per pillar, rolls up recommendations by pillar and impact, and adds a guidance
reference per pillar from the provider's own server (`<provider>_search_well_architected`; changed on 2026-11-04,
when `cloud-knowledge` was retired).
- It renders `waf_markdown`, `waf_html` and `final_waf_html`.
- No model is involved. A pillar with no evidence is "not assessed", never a pass.
- With `provider` `sample`, it's "not applicable" and calls nothing.
- **Its page is released by the existing approval gate**, through its own Format for UI step (`s12`). The report
shows a pillar summary.
- **The report gains live sections:** estate overview, asset details from a new `provider_attributes` field (scan,
then inventory, which also fills Site), network topology from the network agent's `topology_summary`, the
Well-Architected summary, and a Mermaid dependency diagram. Sections empty only because their source was missing
collapse into one "Not examined" list.
## Consequences
- **The individual pattern has a second agent with code.** Its checks are unit-tested (`tests/test_waf.py`), and a
check change needs a test.
- **Judgement in Discovery.** This review judges configuration, which ALZ-F02's ALZ-ASM-05 also does for security.
ALZ-ASM-05 should take this review as input rather than repeat it.
- **Partial pillars.** Performance and sustainability rest on configuration and recommendations only: there's no
utilisation telemetry.
- **No automatic `.md` file.** The platform registers step outputs as `output.json`, so the Markdown is uploaded
after approval with `scripts/kab_publish_report.py` until the platform can save a field as a file.

View File

@@ -0,0 +1,57 @@
# 0015. A live run's inventory is built in batches, with at most three tries per batch
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-04 |
| Decided by | Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step" |
| Requirements changed | ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged |
## Context
The inventory agent built a live run's whole inventory in one model reply. Azure run `exec_cf87d8d3a563`
(2026-11-04) scanned 97 resources:
- The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token
output limit. Run `exec_beb8879632b8`'s 81 records had fitted.
- The output contract's retry returned `{"records": [], "sme_review_queue": []}` in 15 seconds. That met the schema,
so every later step ran on an empty estate and nothing failed.
A single reply caps the estate a run can hold, and a shortfall is not detected.
## Decision
- **Live runs (`provider` `azure` or `gcp`) answer from code in `pre_process`** (`app/business_logic.py`,
`app/inventory.py`), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and
`prompt.md` unchanged.
- **One model call per batch** of findings, through the `normalise_batch` sub-agent (`as_tool: false`, prompt
`prompts/normalise_batch.md`), one batch at a time. The batch size is `INVENTORY_BATCH_SIZE` (20).
- **The model judges; code copies and counts.**
- The model judges, per finding: `hostname`, `fqdn`, `asset_class`, `environment`, `os`, `services`, `facts`,
`fact_source` and `conflicts`.
- Code sets: `canonical_id` (`CI-001`… in the findings' order), `scope_entry_id`, `sources`, `match_basis`,
`provider_attributes`, `facts.site` (the region, with a GCP zone reduced to its region) and `confidence` (0.85
for one source; 0.79 with a conflict).
- Code also builds `sme_review_queue` (every record below 0.8).
- **Every reply is checked before it is kept.** It must be valid JSON (a markdown fence is tolerated). It must hold
exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's
types and enums. Each fact value must be found in the finding or the run context file.
- **At most three tries per batch** (`INVENTORY_MAX_TRIES`, 3 in all). A rejected reply or a failed model call (for
example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the
last reply was rejected.
- **A batch that fails all three tries fails the step.** The error names the batch and its findings. Later batches
are not asked, and no partial inventory is returned.
- **The whole inventory is checked** before the reply: one record per (de-duplicated) finding, and the output schema
in code, because the output contract cannot repair a `pre_process` reply.
## Consequences
- An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input
tokens each, well inside the model's per-minute token limit.
- A shortfall is a failed step, never an empty inventory passed as complete.
- Code-set fields can no longer drift from the finding, which also fixes `environment` being left `unknown` when the
context file gives one (the batch prompt reads `context.assets.<name>.environment`).
- Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's
19-minute budget (`AGENT_MAX_DURATION_SECONDS`).
- **Not changed:** the sample estate still builds its inventory in one reply. Batching it means merging one asset's
scan and CMDB records across applications, which needs its own design.

View File

@@ -0,0 +1,88 @@
# 0016. The run context file carries an intent, in the solution-architecture scope spec's format
| | |
|---|---|
| Status | Proposed |
| Date | 2026-11-04 |
| Decided by | Proposed at the product owner's request, 2026-11-04 ("create an ADR locally"). The owner described the idea and pointed to the solution-architecture agent's scope spec as a model. Not yet accepted, and nothing is built |
| Requirements changed | ALZ-DSC-01 (the planner reads the intent); ALZ-DSC-10 (the Well-Architected review is weighted and checked against it); the Discovery package and ALZ-F02 (both carry the intent forward). The run context file's layout ([run-context.template.yaml](../../workflows/run-context.template.yaml)) |
## Context
- Discovery always runs towards an intent: what the customer wants the estate to become. Today the run context file
holds that only as loose text that no step acts on: `plan.target` ("Azure landing zone"), `plan.objectives` and
`customer.regulatory`.
- The owner wants an explicit intent. It would describe the current architecture against it, and feed the
Well-Architected review with each hyperscaler's patterns and compliance policies, so that the gap to a target
architecture is clear.
- The solution-architecture agent (outcome pattern) already reads such a file: its **scope spec** (ADR-0008 in
`outcome-pattern/solution_architecture/`; reader `app/core/scope.py`; field catalogue
`skills/scoping/assets/catalogue.yaml`; example `eval/datasets/specs/landing-zone.yaml`). It has:
- fixed-value fields: `engagement.kind`, `estate.mode` (`greenfield`, `brownfield_expansion`, `migration`),
`compliance.data_classification`, `compliance.encryption.type`, `networking.exposure` and
`networking.egress_control.model`;
- a marker per unsettled field: `ask` (not known yet), `decide` (the agent may assume, and records it) or
`not_applicable`;
- requirements with ids (`REQ-01` …);
- a completed copy written beside the spec, with `discovery_gaps`, `provenance` and `readiness`.
- Two constraints apply:
- The `*_search_well_architected` tools refuse free-text topics, so customer details never reach a vendor's
search.
- Under charter rule 3, scope, budgets and approvals are platform-supplied facts.
## Decision (proposed)
1. **The intent uses the scope spec's format.** The run context file gains the scope spec's sections, read against
the same catalogue: `metadata`, `engagement`, `intent`, `estate`, `workload`, `compliance`, `networking`,
`governance` and `operations`, each at the catalogue's `schema_version`.
- Discovery's own sections stay as they are: `workflow_inputs`, `generated`, and `context` (`assets`, `owners`,
`business_services`, `scope`, `sources`).
- `customer.regulatory` and `plan.target` give way to `compliance.frameworks` and `intent`.
- One file then runs from Discovery through the package and Assessment to solution architecture.
2. **Discovery describes the current state against the intent; it doesn't design the target.**
- It fills or confirms the fields its evidence can settle, such as `estate.regions`,
`estate.existing_systems`, `networking.topology`, `networking.cidr_allocations`, `networking.exposure`,
`compliance.encryption.type` and `operations.monitoring`.
- It writes each settled value with its evidence and `provenance`.
- It lists what it couldn't settle under `discovery_gaps`.
- The target architecture is drafted downstream (solution architecture), not in Discovery.
3. **Only fixed-value fields drive the review.**
- The Well-Architected review weights pillars from the intent's requirements and fields.
- It checks each `compliance.frameworks` entry against the provider's own policy sets: Azure Policy
initiatives, GCP organisation policies, AWS Config conformance packs.
- It cites the `REQ-` id behind each gap.
- `intent.statement` and other free text are data. They are never sent in a tool call and never followed as
instructions (charter rule 1).
4. **The file never grants anything.**
- `run.pre_approved`, `run.allow_cloud_read` and any similar field are ignored by Discovery.
- What Discovery may read is the connection's scope root, and approval is the workflow's review gate.
- The context may narrow the scope (`context.scope`), never widen it.
5. **The patterns stay separate.** F01 adopts the file format and the catalogue as a shared contract. It doesn't
import the solution-architecture agent's code. The catalogue moves to a neutral shared location, or F01 pins a
copy with its `schema_version` checked.
## Consequences
- **What it enables:**
- The Discovery report and the Well-Architected review can state the gap between the current estate and the
intent, requirement by requirement.
- Assessment and solution architecture start from the same intent, with Discovery's evidence attached.
- Fields marked `ask` give the engagement a concrete question list.
- **Work if accepted:**
- the planner reads the intent and plans evidence for it (DSC-01);
- the review maps frameworks and pillars to the provider's checks and policy sets (DSC-10);
- the report gains a "current vs intent" section;
- the package carries the completed spec;
- `check_context_doc.py` validates the new sections against the catalogue;
- the template, the user guides, and stories DSC-01 and DSC-10 are updated, plus the F02 handover.
- **Risks:**
- The catalogue is owned by the solution-architecture pattern, so a change there now affects Discovery. The
shared `schema_version` check catches a mismatch.
- A long free-text statement adds prompt length and an injection surface. It's kept as data and summarised by
code, not passed to every step.
- **Open questions for the owner:**
- Where does the shared catalogue live?
- Must every Discovery run have an intent, or is it optional, with the review falling back to equal pillar
weights?
- Does Discovery write the completed copy of the spec (the scope spec's own convention), or return it only in
the package?

View File

@@ -0,0 +1,61 @@
# 0017. A live run's scope is every resource under the scope root, planned and checked in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-04 |
| Decided by | Product owner, 2026-11-04: "that scope needs to be extend to all resources" |
| Requirements changed | ALZ-DSC-01 (ADO 3214451): what a live run's scope entries are, and how the plan is made. ALZ-DSC-08 (ADO 3214458): what live coverage is measured against |
## Context
Azure run `exec_29ec31ba92ab` (2026-11-04) was scoped to the resource group `crucible-kab-dev`. That group holds 11
resources, but the run scanned, inventoried and reported 3: the AKS cluster, a storage account and a PostgreSQL
server.
- The planner built a live scope only from `list_servers`, `list_workloads`, `list_kubernetes_clusters` and
`list_data_stores`. Two Key Vaults, two managed identities, a DNS zone, an Event Grid topic, a private endpoint and
a network interface were never scope entries. They weren't declared as gaps either.
- The critic measured coverage with the same four tools, so it reported "coverage complete" and passed the run.
## Decision
- **Every resource under the connection's scope root is accounted for.** The planner's code lists the scope with
`<provider>_list_resources`, every page at 200 a page, and puts each resource in exactly one place
(`app/scope.py`):
- **a scope entry:** any resource that isn't structure or a child component;
- **folded into its parent, with the rule recorded:**
- Azure: disks, network interfaces, VM extensions, scale-set instances;
- GCP: disks, GKE node pools;
- **structure:** subscriptions, resource groups, management groups, projects, folders and organisations, which
are counted but aren't assets.
- **A live plan is built in code, with no model call** (`app/plan.py`, `pre_process`):
- one `scan_target` per scope entry, in batches of 10;
- a `classify_data` per data store;
- one `collect_network` and one `trace_dependencies`;
- `source_gaps` set by the same rules as before.
The run context still only narrows (`context.scope.regions`, `context.scope.exclude`), as declared gaps. The new
`scope_accounting` output gives `listed`, `scope_entries`, `folded` (each with its rule) and `structure`. The sample
estate keeps its model and prompt unchanged.
- **An incomplete listing fails the planner step.** A refused or failed page, or more than 100 pages, means no plan
is made, rather than a plan of part of the estate.
- **The critic counts coverage itself.** Its code lists the scope again with the same rules: `app/scope.py` is
identical in both agents, and a test compares the two files. It compares that listing with the findings and the
planner's declared gaps, and gives the model `live_scope`: counts, the uncovered ids (up to 200, with the exact
count) and any listing issues. The model copies it into `coverage_gaps` and `coverage_complete`. A listing issue
fails coverage.
## Consequences
- Supporting services are discovered, inventoried and reported: Key Vault, identities, DNS, messaging, private
endpoints, registries, monitoring and so on. The Well-Architected review and Assessment (ALZ-ASM-05) see the same
estate.
- The critic can't pass a run that skipped a resource type.
- **Scale:**
- At subscription scope, about 650 resources become about 600 scan tasks. The planner handles that in code.
- The scan executor (ALZ-DSC-02) still has a model call each resource's metadata in one request. That fits a
resource group, but not a large subscription.
- Batching the scan in code, as ADR 0015 did for the inventory, is the next change before a subscription-wide run.
- Folding is a fixed list. A new child-component type shows up as a scope entry until it's added, which over-counts
but never drops anything.

View File

@@ -0,0 +1,79 @@
# 0018. Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "the model should ingest the data and normalise it to the data schema that we own and judge". Plan and decisions D1–D4: `engineering_hub_responsiblities/reports/cp1/cmdb-upload-ingestion-plan-2026-11-05.md` |
| Requirements changed | ALZ-DSC-04 (ADO 3214454): a live run reads CMDB exports uploaded with the run. ALZ-DSC-01 (ADO 3214451): an uploaded CMDB is a source, not a gap |
## Context
- **A live run never read a CMDB.** `cmdb` was always a source gap, and the reconciliation step returned nothing.
- **A second uploaded file broke the run.** The upload step joins files with `\n\n---\n\n`. The planner, inventory
and report parsed the whole text as one YAML document, failed, and carried on as if no run context had been
given.
- **Exports differ in layout.** CMDB exports differ by customer and tool, so no column can be fixed in code.
## Decision
- **Uploads are listed in the run context.** The context's `uploads:` list names the files uploaded after it, in
order, each with a `source` (e.g. `cmdb`). `kab_run_workflow.py` uploads them. The shared `app/run_context.py`
(identical in the planner, inventory, reconciliation and report agents) splits the text into the context and
each listed file. It cuts a CSV, or each XLSX sheet, into a header line and one line per row. It knows no
column.
- **The model normalises; code judges.** The `normalise_cmdb_batch` sub-agent maps each batch of rows (25 by
default, each batch with its header) to `schemas/cmdb_ci.json`. Every string is copied verbatim; `environment`
and `lifecycle` are the only judgments; any other column can be kept in `other_attributes`. Code rejects a reply
unless:
- every row has exactly one result (or a reasoned `skip`);
- every value is found in its row;
- every key is a header column;
- every judgment is an allowed value.
A batch gets at most three tries (ADR 0015's rule). One that fails all three, or an upload the platform couldn't
parse, fails the step.
- **Injections:** a row is flagged when the model or a code pattern sees an instruction. The quoted text goes to
the critic (`cmdb_injections`) and to the report.
- **Reconciliation is code** (`app/reconcile.py`), against the step's own scope listing (`app/scope.py`, the
planner's rules) and the inventory:
- **match:** by cloud resource id, otherwise by name when exactly one listed resource has it;
- **stale attributes:**
- environment, compared at enum level (D1);
- location, compared with the record's site;
- a retired CI whose resource exists (`investigate`);
- owner and business service, when the record carries them;
- **ghost:** an unmatched, not-retired CI whose id lies in a scope container the listing shows (D2). It's
`investigate` outside `context.scope.regions`, in `exclude`, or with an unknown lifecycle, and otherwise
`retire_ci`. A CI with no cloud id is `unplaced` (D3), and one in another container is `out_of_scope`. Neither
is drift.
- **orphan:** an inventory record no CI matched (`add_ci`). Folded components are never orphans.
- **The planner** adds one `ingest_cmdb` task per uploaded CMDB file, and drops the `cmdb` gap when
`include_cmdb` is `yes` and a file is uploaded.
## Consequences
- **New layouts need no code change.** A new export layout, whether a ServiceNow CSV or an asset register XLSX
with other column names, is handled by the model's mapping. Code checks that mapping on every row.
- **The model never decides drift.** A manipulated normaliser can only return values that are in the row.
- **The step makes model calls** (one per batch) on a live run with a CMDB. Without one, it makes none.
- **Subscription scope is cautious.** At subscription scope, a CI in a resource group with no listed resources is
`out_of_scope`, not a ghost. Returning the connection's scope root from the existing discovery servers would
make this exact; that is left until it's needed.
- **Planner and step listings can differ.** The reconciliation step lists the scope again, so a resource created
between the planner's listing and this one is matched, not a ghost.
## Changes after the first live run (2026-11-05)
Run `exec_a0a3c486ec9b` (case `azure-crucible-kab-dev-cmdb`) passed 13 of 19 checks. Every planted CMDB result
was found except the environment difference.
- **`env` tags were redacted** by the discovery servers' shared `_scrub` (`env` is a secret key, for a container's
environment variables). The inventory then took `env: [redacted]` over `Environment: dev`, and 8 of 10 records
had environment `unknown`, so the planted environment difference couldn't be found. A single value under
`env` is now kept; a list or map under it is still redacted (azure-discovery 0.1.6, gcp-discovery 0.1.4,
aws-discovery 0.1.2).
- **Orphans name their resource** (`field` `resource`, `discovered_value` the resource id). Before, they named only
the inventory's canonical id (reconciliation 0.5.1).
- **The critic** no longer raises "drift with no CMDB source" when `source_gaps` has no `cmdb` (critic 0.5.2).
- **The report's drift table** shows both the asset and the CMDB record (report 0.7.2).

View File

@@ -0,0 +1,69 @@
# 0019. A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "the cmdb nodes should be separate from the hyperscaler but still be built in an equivalent way with children and parents at an edge"; "it's the same graph but different edge nodes". Plan and decisions D1–D7: `engineering_hub_responsiblities/reports/cp1/live-dependency-map-plan-2026-11-05.md` |
| Requirements changed | ALZ-DSC-06 (dependency mapping): a live run's graph comes from the scan's recorded dependencies and the CMDB's relationships. ALZ-DSC-04: CMDB relationships are read. ALZ-DSC-08: the critic checks dependencies in code |
## Context
Run `exec_93742bdd7e29` (case `azure-crucible-kab-dev-cmdb`) passed all 19 checks, and the critic failed it.
- **The dependency step never received the scan's findings.** The scan recorded 7 dependencies. The step rebuilt
edges from each record's network attributes, dropped 4 of them and added 2 VNet edges the scan never recorded.
- **Out-of-scope targets were bare names.** Dependencies outside the scope root (subnets in `crucible-net-dev`, a
public IP in `crucible-kab-dev-nodes`) came through as names, so the critic reported them as unsupported.
- **The critic was inconsistent.** It passed the same estate the day before, so the verdict depended on the
model's judgment that day.
- **The CMDB's own relationships were never read.**
## Decision
- **One graph, three kinds of node** (`app/graph.py` in the dependency agent, from `pre_process`, with no model
call):
- `resource`: a scope entry, i.e. its inventory record;
- `external`: a dependency target outside the scope root or not inventoried, with its container and what depends
on it (D1);
- `ci`: a judged CMDB CI, with its reconciliation result.
- **Every edge runs parent (`source`) to child (`target`):**
- **resource → resource or external:** one edge per entry in a finding's `dependencies`. They're typed by the
target's resource type, with confidence 0.8 and `auto_approved`, since they're the provider's own record (D3).
- **ci → ci:** a CMDB relationship. Confidence 0.7 and `sme_review_recommended`.
- **ci → resource, `represents`:** the reconciliation's match. `auto_approved` when matched by cloud id;
`sme_review_recommended` when matched by name.
- **Folded components:** a dependency on a resource's own child component (a NIC, a disk) is a
`folded_dependencies` entry, not an edge (D2).
- **Relationships are judged like CIs** (ADR 0018). A relationship export (`source: cmdb_relationships` in the
context's `uploads:`, D6), or a CI row's own parent or child columns (`related`), is normalised by the model to
`schemas/cmdb_relationship.json`. It's typed with the graph's edge types (D7), and every value is checked in code
against its row. An end that resolves to no CI (by `ci_ref`, or by unique name) is dangling, never a node. This
is done in the CMDB step (D5).
- **The comparison is a walk over the one graph.** For each ci → ci edge whose two CIs both represent resources:
- the cloud has the same resource → resource edge, in the same direction: `confirmed`. The cloud edge gains
the CMDB evidence, and its confidence becomes 0.95;
- otherwise: `cmdb_only`.
A cloud edge between two represented resources that no ci → ci edge matches is `discovered_only`. A relationship
with an end that has no resource is `not_comparable`.
- **The critic counts dependencies in code** (`live_dependencies`): every recorded dependency must be an edge (to a
record, or to an external node) or a folded dependency. It also treats `external`, `represents` and ci → ci edges
as supported by their own evidence.
- **The report:**
- one diagram, with each node drawn by kind: a resource as a box, external with a dashed outline, a CI as a
hexagon, and `represents` as dotted;
- tables of dependencies outside this discovery (with where to widen the scope, D4), the CMDB comparison, and
dangling relationships.
## Consequences
- No recorded dependency is dropped and none is invented. The critic's dependency verdict is the same on every run.
- An assessment sees what a landing zone depends on outside the scope, for example the shared VNet in
`crucible-net-dev`, and which resource group to add to discover it.
- The CMDB's picture and the cloud's are in one graph, so a CI's dependents, and an asset's dependents in either
source, are one walk.
- `discovered_only` is reported in the dependency step and the report. It doesn't add a drift item, because the
CMDB step (s4) runs before the graph (s7).
- Folding is a fixed list, so a dependency on another asset's child component (another VM's disk) is still folded,
not redirected to its parent: the listing doesn't say which parent owns it.

View File

@@ -0,0 +1,75 @@
# 0020. One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "lets go ahead with the enhancements when these are not combined and then later that can be decided"; the solution architecture agents are a separate stream, and Discovery's outputs are used only by the Assessment workflow. Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phases 0 and 1 |
| Requirements changed | ALZ-F02 ASM-01 (Discovery hand-off): F02 selects one package artifact rather than one per Discovery step (F02 ADR 0002). New step `s13`, agent `cp1-discovery-package-agent` |
## Context
- **F02 reads Discovery by hand.** Its Artifacts step needs each Discovery step's artifact id typed in, and every F02
agent receives all of them as one text, joined by `---` (engineering hub review
`reports/cp1/discovery-to-assessment-requirements-review-2026-11-04.md`, section 1).
- **Nothing measured whether a run was useful.** A case either passed its checks or it didn't. Nothing showed how
much of what F02 needs a run gives, how many decisions it leaves for a person, or what it cost.
- **KAB can't write a .zip.** It can expand a selected zip, but no step type creates one. Every step's output is kept
as a JSON artifact, `<execution_id>_<step_id>`.
- **Discovery and Assessment stay separate workflows** for now; combining them is a later decision.
- **The idea is borrowed, not the code.** The original `gcp_solution_architecture_agent` writes a deliverable folder
with an `execution.json` and measures its runs with limits per case. Those ideas are used here; nothing is ported,
and no hand-over to the solution architecture stream is built.
## Decision
1. **A package step, `s13`, answered in code** (`cp1-discovery-package-agent`, `app/package.py`, no model, no tools):
- It runs after the report's review point, beside the two page steps, so a package exists only for a run the
Engagement Lead let through.
- Its output holds every step's output fields, **copied unchanged**, as named parts: `plan`, `scan`, `inventory`,
`cmdb`, `classification`, `network`, `dependencies`, `critic`, `well_architected`, `report`. Parts are named by
meaning, because KAB renames step ids on import. The report part keeps `counts` and `report_markdown`, not the
HTML.
- Its `manifest` holds the engagement, run, execution id, provider, time, critic verdict, each agent's version
(a static input set when the workflow is built), the planner's source gaps, a `sha256` and item count per part,
the parts absent, `complete`, `problems` and the scorecard.
- `complete` is true only when the required parts (plan, inventory, classification, network, dependencies,
critic, report) are present and the critic passed. The CMDB and Well-Architected parts are optional.
- Each `sha256` is over the part's canonical JSON (`sort_keys`, `separators=(",", ":")`, UTF-8), so a consumer
can recompute it before trusting the part.
2. **A value scorecard** (`app/scorecard.py`), in the manifest and in `scripts/evaluate_workflow_run.py`, from the
same code:
- `f02_fields`: each F02 field (application, owner, criticality, environment, support status, end of support,
compliance scope, annual cost, data class), per record, as `filled`, `declared_gap`, `missing` or
`not_applicable`;
- `f02_run`: the run-level facts F02's ASM-01 checks;
- `decisions`: what the run leaves for a person, by kind;
- `findings`: how many findings name an owner and an action.
`filled` and `filled_or_declared` are both reported: a declared gap is honest, but it isn't value.
3. **Cases carry a value bar and limits.** `value:` (for example `f02_filled_min`) holds the baseline so a change
can't lower it; `limits:` bound tokens, model calls and agent seconds, read from the agents' logs by
`scripts/collect_run_usage.py`. Both fail as major.
4. **F02 is unchanged by this ADR.** Its Artifacts step can select the package today. Teaching F02 to verify the
manifest in code is F02's own change, with its own ADR.
## Consequences
- **First live runs** (2026-11-05, both after the Engagement Lead continued): `exec_14956d524a8b`
(`azure-crucible-kab-dev-cmdb`, 46 of 46 checks) and `exec_d1c242609f82` (`azure-crucible-kab-dev`, 19 of 19).
Both packages are complete, and every part's `sha256` recomputes.
- **Baseline:** 24% of F02's fields filled in both cases; 27% filled or declared with a CMDB, 51% without one (the
CMDB gap declares owners and applications); 6 of 7 run checks met (no applications list); 9 decisions with a CMDB;
findings 0% with an owner, 100% with an action. Owners are in the CMDB export but never reach the inventory
records, and nothing reads cost, lifecycle or compliance. These are phases 3 and 4 of the plan.
- **Size:** about 100 KB with a CMDB and 78 KB without, against KAB's 5 MB inline limit. A large estate will need
per-application summaries (plan phase 3) or a platform zip step.
- **The review record is implied, not explicit.** The package proves the run continued past review, but not who
approved it or their comments. That still needs the platform to expose review records to later steps.
- **Every Discovery output field now has a reader:** the context report's "unread" fields reach F02 through the
package, except the run's own ids.
- **The package input schema has no enums**, so the build-time contract check can never refuse a producer's value
at this step. A new field needs a row in `INPUTS`, its wiring in `build_workflow.py`, and a test checks the two
match.
- **ADR 0016 (Proposed) needs the owner's review:** it carries one run context file through to solution
architecture, which the decision of 2026-11-05 to keep the streams independent rules out. Plan phase 2 borrows
only the scope spec's field meanings, within F01.

View File

@@ -0,0 +1,59 @@
# 0021. A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "CMDB→tag→RG, review model" (code groups by CMDB business service, then the application tag, then resource group; the model proposes groups only for what's left, code judges them, and model-proposed groups are flagged for the Engagement Lead's review). Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phase 3 |
| Requirements changed | ALZ-DSC-03 and ALZ-F02 ASM-01 ("inventory (applications and assets)"): a live run lists applications, from the dependency step rather than the inventory. ALZ-DSC-04: each CI's result carries its owner, business service and support group. ALZ-DSC-06: the dependency step groups applications and names owners |
## Context
- **F02 needs applications.** Its planner groups by the Discovery `applications` list, else by `facts.business_service`,
and live runs had neither, so every asset was a gap (review of 2026-11-04, item 5). The value scorecard's baseline
([ADR 0020](0020.md)) showed application and owner missing on all 10 records of the CMDB case.
- **The evidence was there but never reached Assessment.** The uploaded CMDB names each CI's `owned_by` and
`business_service`, but the CMDB step's per-CI results dropped them. Every resource carries an `Application` tag.
- **The dependency step is the right place:** it runs after the CMDB step, already joins records to CIs (ADR 0019),
and is answered in code.
## Decision
1. **The CMDB step's `cis`** carry each CI's `owner`, `business_service` and `support_group`, verbatim (0.7.0).
2. **The dependency step groups each record, once,** in this order (`app/applications.py`, 0.6.0):
1. the matched CI's business service (`basis: cmdb_business_service`);
2. the resource's application tag (`Application`, `app`, `application_name`, `app_name`, `business_service`,
`service`, any case; `basis: tag`). **Tag alias:** when every CMDB-grouped member of one application carries
the same application tag, a tag-only record with that tag joins that application, flagged for review. A tag
two applications claim is no alias;
3. its resource group or project (`basis: container`), flagged for review;
4. for what's left only, the `propose_application_groups` sub-agent, from names, types and environments. Code
rejects the whole reply if any group names an asset not asked about, places an asset twice, or has an empty,
over-long or instruction-like name. Every model group is flagged for review; a rejected or failed reply leaves
its assets ungrouped, which the note says.
3. **Owners:** from the matched CI (`owner`, else `support_group`), else an owner tag. An application's owner is set
only when its members' owners agree (`owners_disagree` otherwise). No owner is inferred.
4. **Outputs:** `applications` (id, name, basis, members, owner, business service, evidence, `review_required` and
reasons), `asset_owners`, `ungrouped_assets` and `applications_note`. The report shows them after the inventory
and lists the groups to review (0.9.0); the package carries them as its `applications` part, and the scorecard
reads a record's application and owner from it (package 0.2.0, scorecard version 2).
## Consequences
- **Expected on the CMDB case:** one application, "KAB Platform", of 10 assets: 8 by the CMDB, 2 (the DNS zone and
the Event Grid topic, which have no CI) by the `Application: AppName1` tag every CMDB-grouped member shares,
flagged for review. Owner "Platform Engineering" for the 8 matched assets; the 2 orphans have none, correctly.
Scorecard: F02 fields filled from 24% to about 46%; run checks 7 of 7; findings with an owner from 0% to about 33% (the stale
attributes on owned assets; orphans and ghosts have none).
- **Without a CMDB:** one application, "AppName1", by tag, needing no review, and no owners: the CMDB gap declares
them.
- **The model is rarely asked:** on Azure and Google Cloud every resource has a resource group or project, so step 4
runs only for estates with no container. Its tests use a stand-in model.
- **Applications are not graph nodes.** They're a separate list, so no edge or node enum changed and no other step's
input schema had to. Membership edges can be added later if a consumer needs them in the graph.
- **The review is the Engagement Lead's,** at the report's review point. Assessment reads `review_required` and
can decide how to treat a group the lead didn't change.
- **Live, 2026-11-05:** `exec_58ba6a71d00b` (`azure-crucible-kab-dev-cmdb`) passed 56 of 56 checks: one application,
KAB Platform, of 10 assets, flagged for review; 8 owners; 46% of F02's fields filled; 7 of 7 run checks.
`exec_925194d79c47` (`azure-crucible-kab-dev`) passed 21 of 21: AppName1 by tag, no owners, 37% filled. Neither
run asked the grouping model.

View File

@@ -21,9 +21,19 @@ Each agent's `docs/adr/README.md` lists the records that change its own story.
| [0005](0005.md) | Inventory before CMDB reconciliation; the inventory gets CMDB records from `alz-inventory` | Proposed | DSC-03, 04 |
| [0006](0006.md) | The critic's verdict is gated by human approval, not by code | Proposed | DSC-08 |
| [0007](0007.md) | Agent names | Proposed | All |
| [0008](0008.md) | Models | Proposed | All |
| [0008](0008.md) | Models: executors on `gpt-5.6-sol`, the critic on `gpt-5.4` (2026-11-03) | Accepted | All |
| [0009](0009.md) | Deploy with kustomize; keys stay in Secrets | Accepted | — |
| [0010](0010.md) | Step outputs travel through the workflow; there is no shared store | Proposed | All |
| [0011](0011.md) | Uploaded files and project artifacts reach the planner and the inventory as text | Accepted | DSC-01, 03 |
| [0012](0012.md) | The Discovery report comes from code, and the approval gate moves to it | Accepted | report (new) |
| [0013](0013.md) | A `provider` input lets one workflow read the sample estate or a live cloud | Accepted | All |
| [0014](0014.md) | A Well-Architected review step in Discovery, judged in code | Accepted | DSC-10 (new), report |
| [0015](0015.md) | A live run's inventory is built in batches, at most three tries per batch; a batch that fails all three fails the step | Accepted | DSC-03 |
| [0016](0016.md) | The run context file carries an intent, in the solution-architecture scope spec's format | Proposed | DSC-01, DSC-10, package, F02 |
| [0017](0017.md) | A live run's scope is every resource under the scope root, planned and checked in code | Accepted | DSC-01, DSC-08 |
| [0018](0018.md) | Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code | Accepted | DSC-01, DSC-03, DSC-04, DSC-08, report |
| [0019](0019.md) | A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes | Accepted | DSC-04, DSC-06, DSC-08, report |
| [0020](0020.md) | One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code | Accepted | package (new), F02 ASM-01 |
| [0021](0021.md) | A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal | Accepted | DSC-03, DSC-04, DSC-06, report, package, F02 ASM-01 |
New records copy [template.md](template.md) and take the next number.