ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,10 @@ For people who run, review or operate CP1 Discovery with the individual-agents p
|
||||
|
||||
| Document | For |
|
||||
|---|---|
|
||||
| [deployment.md](../../../docs/deployment.md) | Platform operators: building, pushing and deploying the eight agents with kustomize |
|
||||
| [deployment.md](../../../docs/deployment.md) | Platform operators: building, pushing and deploying the agents with kustomize |
|
||||
| [adr/](adr/README.md) | Product owner and reviewers: every change to the original ALZ-F01 requirements in this pattern |
|
||||
| [run-context-field-review.md](run-context-field-review.md) | Product owner: ADR 0016's 51 run context fields, one by one, with a recommendation and a decision column |
|
||||
| [discovery-value-plan.md](discovery-value-plan.md) | Product owner: what the output must give the Engagement Lead and Assessment, the scorecard baseline, and the phases to raise it |
|
||||
| `<agent>/docs/user-guide.md` | Engagement leads: what each step does and how to read its output |
|
||||
|
||||
- [MCP tools and context](../../../docs/platform-issues/agent-tools-and-context/individual-agents-report-2026-11-01.md): why each agent holds its tools, how context carries between steps, and what to watch for.
|
||||
|
||||
@@ -3,15 +3,42 @@
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-10-29 |
|
||||
| Date | 2026-10-29, changed 2026-11-01 and 2026-11-03 |
|
||||
| Decided by | Built as a default; awaiting product owner and platform confirmation |
|
||||
| Requirements changed | DSC-08's note on sharing a model with any executor |
|
||||
|
||||
## Decision
|
||||
|
||||
- **Seven agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
|
||||
- **The critic uses `claude-sonnet-5-5`.** Both models are deployed on the Azure AI resource behind the gateway.
|
||||
- **All eight agents use `gpt-5.6-sol`,** as the CP4 agents do, with eval cases waiting 330 seconds.
|
||||
- **The critic shares a model with the executors.** It was to use `claude-sonnet-5-5`, but the cluster's LiteLLM doesn't serve that (checked 2026-11-01), so the product owner moved it to `gpt-5.6-sol`.
|
||||
|
||||
## Change, 2026-11-03
|
||||
|
||||
**The critic now runs on `gpt-5.4`,** which the cluster's LiteLLM serves, and the eight other agents stay on
|
||||
`gpt-5.6-sol`. The product owner asked for this after the first live GCP run (`exec_0f8629066e13`), whose critic
|
||||
noted that it shared the executors' model. The critic states the models it and the executors use in `note`.
|
||||
|
||||
## Change, 2026-11-04
|
||||
|
||||
**The network agent also runs on `gpt-5.4`.** Two live Azure runs on an 81-record subscription
|
||||
(`exec_c57354b25f8e`, `exec_beb8879632b8`) failed at the network step: `gpt-5.6-sol` returned HTTP 429 for its token
|
||||
rate limit, with no fallback configured. The network agent sends the whole inventory on each of up to 25 model calls
|
||||
while it pages through topology, which uses the per-minute budget in one step. `gpt-5.4` has its own quota. The
|
||||
product owner approved this. As a result the critic shares a model with the network agent, and says so in `note`.
|
||||
Follow-up: send the network agent only network-relevant records, which fixes the cause.
|
||||
|
||||
## Consequences
|
||||
|
||||
Confirm both names on the cluster's LiteLLM gateway before the first run.
|
||||
- **The critic can share blind spots with the agents it judges.** It states this in its `note`, as DSC-08 requires, and the Engagement Lead's approval still gates its verdict (ADR 0006).
|
||||
- **Revisit this when another model is on the gateway.** A different model for the critic remains the better design.
|
||||
|
||||
## Change, 2026-11-04: output limit 80,000 on the `gpt-5.6-sol` agents
|
||||
|
||||
At the product owner's direction ("do not change the model; increase the token limits within the agents to 80,000"),
|
||||
`LITELLM_MAX_TOKENS` is `80000` (was `32000`) on the eight `gpt-5.6-sol` agents. Models are unchanged.
|
||||
|
||||
- The network agent and the critic stay at `16384`: `gpt-5.4` refuses a larger value ("max_tokens is too large",
|
||||
`exec_b929f06cbb0c`).
|
||||
- Not yet confirmed that `gpt-5.6-sol` accepts 80,000. A refusal would fail the planner, the first model step, at once.
|
||||
- Azure OpenAI counts a request's `max_tokens` against the deployment's tokens-per-minute quota before the reply is
|
||||
written, so a higher limit can bring the 429 sooner if `gpt-5.6-sol` is served from Azure.
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# ADR 0012: the Discovery report comes from code, and the approval gate moves to it
|
||||
|
||||
Status: Proposed. Date: 2026-11-01.
|
||||
|
||||
## Context
|
||||
|
||||
The workflow ended at the critic's verdict and produced no report for the Engagement Lead, who approved from the verdict alone.
|
||||
|
||||
## Decision
|
||||
|
||||
Add `cp1-discovery-report-agent` as the last step. Its `pre_process` hook renders the report in code, as Markdown and as a self-contained HTML page, and replies without calling a model, so it can only repeat what the steps recorded. The approval gate moves from the critic to this step, so approval is given on the whole report. This updates ADR 0006.
|
||||
|
||||
A Format for UI step after the gate takes `final_html`, so an approved run ends with an artifact of the clean page. This is on trial.
|
||||
|
||||
## Consequences
|
||||
|
||||
The report has no written narrative. The verdict still isn't enforced: a person can approve a `fail`, and the report marks one at the top. The saved workflow must be imported again, and a run that fails before the report step produces no report.
|
||||
@@ -0,0 +1,41 @@
|
||||
# 0013. A `provider` input lets one workflow read the sample estate or a live cloud
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-03 |
|
||||
| Decided by | Product owner, 2026-11-03: adapt both patterns, with a workflow value choosing the hyperscaler; GCP first |
|
||||
| Requirements changed | DSC-01 to DSC-09: the scope, scan, classification, network and critic sources in a live run |
|
||||
|
||||
## Context
|
||||
|
||||
The Vault Service is running on kab-dev, and the CP1 tenant's Default Project holds connections for
|
||||
`azure-discovery` and `gcp-discovery`. The agents called only the sample server.
|
||||
|
||||
The runtime offers every declared MCP tool to the model, and its `before_model` hook can only add instructions, so
|
||||
a code-only choice of cloud isn't possible in a declarative agent (see the repository's
|
||||
`docs/hyperscaler-interchange-review-note-2026-11-01.md`). The cloud servers stay standalone.
|
||||
|
||||
## Decision
|
||||
|
||||
- **A `provider` run input**, `sample`, `azure` or `gcp` (default `sample`), goes to every step. The runtime
|
||||
enforces the enum. With `sample`, nothing changes.
|
||||
- **Live tools.** The planner, scan, classification, network and critic agents also declare the live tools they
|
||||
need from `azure-discovery` and `gcp-discovery`. A "Live provider" section in each prompt confines a live run to
|
||||
`{provider}_*` tools. Inventory, CMDB, dependency and report agents call no live tool.
|
||||
- **Live scope.** Scope comes from the cloud's own listings: servers, workloads, clusters and data stores, every
|
||||
kind, every page. Each resource's own name or id is its scope entry id, and its region is its site.
|
||||
- **Missing sources.** CMDB, ITSM, flows and schemas are declared `source_gaps` (`no_connection`).
|
||||
- **Wrong cloud.** The critic fails a run in which any identifier belongs to the other cloud.
|
||||
- **AWS later.** Its tools take `account_id` and `region`, and no AWS connection is stored yet.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Choosing the cloud rests on the prompt.** Both connections are in one project, so a wrong-cloud call would
|
||||
succeed. The critic's identifier check is the backstop.
|
||||
- **Large estates.** A live estate's tool output goes straight into each model's context. The GCP trial project
|
||||
is about 40 KB; the Azure subscription was about 5.3 MB (most of it Advisor recommendations, which no agent
|
||||
calls), and needs testing before it's relied on.
|
||||
- **No call-log check.** The critic can't check tool calls in a live run (`call_log` and `guard_hits` go in
|
||||
`not_checked`).
|
||||
- **Every declared server must be registered and healthy** for the agents to start, the cloud servers included.
|
||||
@@ -0,0 +1,44 @@
|
||||
# 0014. A Well-Architected review step in Discovery, judged in code
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-03 |
|
||||
| Decided by | Product owner, 2026-11-03: expand Discovery to support the Well-Architected Framework (advised to place it in ALZ-F02; chose Discovery) |
|
||||
| Requirements changed | New story ALZ-DSC-10 (to be created); the Discovery report's content for live runs |
|
||||
|
||||
## Context
|
||||
|
||||
The first live GCP run produced a thin report, and the owner wants a Well-Architected review of the provider as a
|
||||
file of its own. Discovery's agents collected inventory, metadata and topology, but no policies or recommendations.
|
||||
Azure's subscription has about 1,776 Advisor recommendations (3.2 MB), more than a model step can take.
|
||||
|
||||
## Decision
|
||||
|
||||
- **A new step, `cp1-well-architected-review-agent`** (ALZ-DSC-10), after the critic and before the report. It
|
||||
answers from code in `pre_process` (`app/waf.py`), like the report agent (ADR 0012):
|
||||
- It reads the in-scope resources, firewall or NSG rules, organisation policies, Azure resource locks and every
|
||||
page of the provider's recommendations, through read-only tools on the existing servers.
|
||||
- It applies fixed checks per pillar, rolls up recommendations by pillar and impact, and adds a guidance
|
||||
reference per pillar from the provider's own server (`<provider>_search_well_architected`; changed on 2026-11-04,
|
||||
when `cloud-knowledge` was retired).
|
||||
- It renders `waf_markdown`, `waf_html` and `final_waf_html`.
|
||||
- No model is involved. A pillar with no evidence is "not assessed", never a pass.
|
||||
- With `provider` `sample`, it's "not applicable" and calls nothing.
|
||||
- **Its page is released by the existing approval gate**, through its own Format for UI step (`s12`). The report
|
||||
shows a pillar summary.
|
||||
- **The report gains live sections:** estate overview, asset details from a new `provider_attributes` field (scan,
|
||||
then inventory, which also fills Site), network topology from the network agent's `topology_summary`, the
|
||||
Well-Architected summary, and a Mermaid dependency diagram. Sections empty only because their source was missing
|
||||
collapse into one "Not examined" list.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The individual pattern has a second agent with code.** Its checks are unit-tested (`tests/test_waf.py`), and a
|
||||
check change needs a test.
|
||||
- **Judgement in Discovery.** This review judges configuration, which ALZ-F02's ALZ-ASM-05 also does for security.
|
||||
ALZ-ASM-05 should take this review as input rather than repeat it.
|
||||
- **Partial pillars.** Performance and sustainability rest on configuration and recommendations only: there's no
|
||||
utilisation telemetry.
|
||||
- **No automatic `.md` file.** The platform registers step outputs as `output.json`, so the Markdown is uploaded
|
||||
after approval with `scripts/kab_publish_report.py` until the platform can save a field as a file.
|
||||
@@ -0,0 +1,57 @@
|
||||
# 0015. A live run's inventory is built in batches, with at most three tries per batch
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-04 |
|
||||
| Decided by | Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step" |
|
||||
| Requirements changed | ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged |
|
||||
|
||||
## Context
|
||||
|
||||
The inventory agent built a live run's whole inventory in one model reply. Azure run `exec_cf87d8d3a563`
|
||||
(2026-11-04) scanned 97 resources:
|
||||
|
||||
- The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token
|
||||
output limit. Run `exec_beb8879632b8`'s 81 records had fitted.
|
||||
- The output contract's retry returned `{"records": [], "sme_review_queue": []}` in 15 seconds. That met the schema,
|
||||
so every later step ran on an empty estate and nothing failed.
|
||||
|
||||
A single reply caps the estate a run can hold, and a shortfall is not detected.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Live runs (`provider` `azure` or `gcp`) answer from code in `pre_process`** (`app/business_logic.py`,
|
||||
`app/inventory.py`), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and
|
||||
`prompt.md` unchanged.
|
||||
- **One model call per batch** of findings, through the `normalise_batch` sub-agent (`as_tool: false`, prompt
|
||||
`prompts/normalise_batch.md`), one batch at a time. The batch size is `INVENTORY_BATCH_SIZE` (20).
|
||||
- **The model judges; code copies and counts.**
|
||||
- The model judges, per finding: `hostname`, `fqdn`, `asset_class`, `environment`, `os`, `services`, `facts`,
|
||||
`fact_source` and `conflicts`.
|
||||
- Code sets: `canonical_id` (`CI-001`… in the findings' order), `scope_entry_id`, `sources`, `match_basis`,
|
||||
`provider_attributes`, `facts.site` (the region, with a GCP zone reduced to its region) and `confidence` (0.85
|
||||
for one source; 0.79 with a conflict).
|
||||
- Code also builds `sme_review_queue` (every record below 0.8).
|
||||
- **Every reply is checked before it is kept.** It must be valid JSON (a markdown fence is tolerated). It must hold
|
||||
exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's
|
||||
types and enums. Each fact value must be found in the finding or the run context file.
|
||||
- **At most three tries per batch** (`INVENTORY_MAX_TRIES`, 3 in all). A rejected reply or a failed model call (for
|
||||
example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the
|
||||
last reply was rejected.
|
||||
- **A batch that fails all three tries fails the step.** The error names the batch and its findings. Later batches
|
||||
are not asked, and no partial inventory is returned.
|
||||
- **The whole inventory is checked** before the reply: one record per (de-duplicated) finding, and the output schema
|
||||
in code, because the output contract cannot repair a `pre_process` reply.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input
|
||||
tokens each, well inside the model's per-minute token limit.
|
||||
- A shortfall is a failed step, never an empty inventory passed as complete.
|
||||
- Code-set fields can no longer drift from the finding, which also fixes `environment` being left `unknown` when the
|
||||
context file gives one (the batch prompt reads `context.assets.<name>.environment`).
|
||||
- Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's
|
||||
19-minute budget (`AGENT_MAX_DURATION_SECONDS`).
|
||||
- **Not changed:** the sample estate still builds its inventory in one reply. Batching it means merging one asset's
|
||||
scan and CMDB records across applications, which needs its own design.
|
||||
@@ -0,0 +1,88 @@
|
||||
# 0016. The run context file carries an intent, in the solution-architecture scope spec's format
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-04 |
|
||||
| Decided by | Proposed at the product owner's request, 2026-11-04 ("create an ADR locally"). The owner described the idea and pointed to the solution-architecture agent's scope spec as a model. Not yet accepted, and nothing is built |
|
||||
| Requirements changed | ALZ-DSC-01 (the planner reads the intent); ALZ-DSC-10 (the Well-Architected review is weighted and checked against it); the Discovery package and ALZ-F02 (both carry the intent forward). The run context file's layout ([run-context.template.yaml](../../workflows/run-context.template.yaml)) |
|
||||
|
||||
## Context
|
||||
|
||||
- Discovery always runs towards an intent: what the customer wants the estate to become. Today the run context file
|
||||
holds that only as loose text that no step acts on: `plan.target` ("Azure landing zone"), `plan.objectives` and
|
||||
`customer.regulatory`.
|
||||
- The owner wants an explicit intent. It would describe the current architecture against it, and feed the
|
||||
Well-Architected review with each hyperscaler's patterns and compliance policies, so that the gap to a target
|
||||
architecture is clear.
|
||||
- The solution-architecture agent (outcome pattern) already reads such a file: its **scope spec** (ADR-0008 in
|
||||
`outcome-pattern/solution_architecture/`; reader `app/core/scope.py`; field catalogue
|
||||
`skills/scoping/assets/catalogue.yaml`; example `eval/datasets/specs/landing-zone.yaml`). It has:
|
||||
- fixed-value fields: `engagement.kind`, `estate.mode` (`greenfield`, `brownfield_expansion`, `migration`),
|
||||
`compliance.data_classification`, `compliance.encryption.type`, `networking.exposure` and
|
||||
`networking.egress_control.model`;
|
||||
- a marker per unsettled field: `ask` (not known yet), `decide` (the agent may assume, and records it) or
|
||||
`not_applicable`;
|
||||
- requirements with ids (`REQ-01` …);
|
||||
- a completed copy written beside the spec, with `discovery_gaps`, `provenance` and `readiness`.
|
||||
- Two constraints apply:
|
||||
- The `*_search_well_architected` tools refuse free-text topics, so customer details never reach a vendor's
|
||||
search.
|
||||
- Under charter rule 3, scope, budgets and approvals are platform-supplied facts.
|
||||
|
||||
## Decision (proposed)
|
||||
|
||||
1. **The intent uses the scope spec's format.** The run context file gains the scope spec's sections, read against
|
||||
the same catalogue: `metadata`, `engagement`, `intent`, `estate`, `workload`, `compliance`, `networking`,
|
||||
`governance` and `operations`, each at the catalogue's `schema_version`.
|
||||
- Discovery's own sections stay as they are: `workflow_inputs`, `generated`, and `context` (`assets`, `owners`,
|
||||
`business_services`, `scope`, `sources`).
|
||||
- `customer.regulatory` and `plan.target` give way to `compliance.frameworks` and `intent`.
|
||||
- One file then runs from Discovery through the package and Assessment to solution architecture.
|
||||
2. **Discovery describes the current state against the intent; it doesn't design the target.**
|
||||
- It fills or confirms the fields its evidence can settle, such as `estate.regions`,
|
||||
`estate.existing_systems`, `networking.topology`, `networking.cidr_allocations`, `networking.exposure`,
|
||||
`compliance.encryption.type` and `operations.monitoring`.
|
||||
- It writes each settled value with its evidence and `provenance`.
|
||||
- It lists what it couldn't settle under `discovery_gaps`.
|
||||
- The target architecture is drafted downstream (solution architecture), not in Discovery.
|
||||
3. **Only fixed-value fields drive the review.**
|
||||
- The Well-Architected review weights pillars from the intent's requirements and fields.
|
||||
- It checks each `compliance.frameworks` entry against the provider's own policy sets: Azure Policy
|
||||
initiatives, GCP organisation policies, AWS Config conformance packs.
|
||||
- It cites the `REQ-` id behind each gap.
|
||||
- `intent.statement` and other free text are data. They are never sent in a tool call and never followed as
|
||||
instructions (charter rule 1).
|
||||
4. **The file never grants anything.**
|
||||
- `run.pre_approved`, `run.allow_cloud_read` and any similar field are ignored by Discovery.
|
||||
- What Discovery may read is the connection's scope root, and approval is the workflow's review gate.
|
||||
- The context may narrow the scope (`context.scope`), never widen it.
|
||||
5. **The patterns stay separate.** F01 adopts the file format and the catalogue as a shared contract. It doesn't
|
||||
import the solution-architecture agent's code. The catalogue moves to a neutral shared location, or F01 pins a
|
||||
copy with its `schema_version` checked.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **What it enables:**
|
||||
- The Discovery report and the Well-Architected review can state the gap between the current estate and the
|
||||
intent, requirement by requirement.
|
||||
- Assessment and solution architecture start from the same intent, with Discovery's evidence attached.
|
||||
- Fields marked `ask` give the engagement a concrete question list.
|
||||
- **Work if accepted:**
|
||||
- the planner reads the intent and plans evidence for it (DSC-01);
|
||||
- the review maps frameworks and pillars to the provider's checks and policy sets (DSC-10);
|
||||
- the report gains a "current vs intent" section;
|
||||
- the package carries the completed spec;
|
||||
- `check_context_doc.py` validates the new sections against the catalogue;
|
||||
- the template, the user guides, and stories DSC-01 and DSC-10 are updated, plus the F02 handover.
|
||||
- **Risks:**
|
||||
- The catalogue is owned by the solution-architecture pattern, so a change there now affects Discovery. The
|
||||
shared `schema_version` check catches a mismatch.
|
||||
- A long free-text statement adds prompt length and an injection surface. It's kept as data and summarised by
|
||||
code, not passed to every step.
|
||||
- **Open questions for the owner:**
|
||||
- Where does the shared catalogue live?
|
||||
- Must every Discovery run have an intent, or is it optional, with the review falling back to equal pillar
|
||||
weights?
|
||||
- Does Discovery write the completed copy of the spec (the scope spec's own convention), or return it only in
|
||||
the package?
|
||||
@@ -0,0 +1,61 @@
|
||||
# 0017. A live run's scope is every resource under the scope root, planned and checked in code
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-04 |
|
||||
| Decided by | Product owner, 2026-11-04: "that scope needs to be extend to all resources" |
|
||||
| Requirements changed | ALZ-DSC-01 (ADO 3214451): what a live run's scope entries are, and how the plan is made. ALZ-DSC-08 (ADO 3214458): what live coverage is measured against |
|
||||
|
||||
## Context
|
||||
|
||||
Azure run `exec_29ec31ba92ab` (2026-11-04) was scoped to the resource group `crucible-kab-dev`. That group holds 11
|
||||
resources, but the run scanned, inventoried and reported 3: the AKS cluster, a storage account and a PostgreSQL
|
||||
server.
|
||||
|
||||
- The planner built a live scope only from `list_servers`, `list_workloads`, `list_kubernetes_clusters` and
|
||||
`list_data_stores`. Two Key Vaults, two managed identities, a DNS zone, an Event Grid topic, a private endpoint and
|
||||
a network interface were never scope entries. They weren't declared as gaps either.
|
||||
- The critic measured coverage with the same four tools, so it reported "coverage complete" and passed the run.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Every resource under the connection's scope root is accounted for.** The planner's code lists the scope with
|
||||
`<provider>_list_resources`, every page at 200 a page, and puts each resource in exactly one place
|
||||
(`app/scope.py`):
|
||||
- **a scope entry:** any resource that isn't structure or a child component;
|
||||
- **folded into its parent, with the rule recorded:**
|
||||
- Azure: disks, network interfaces, VM extensions, scale-set instances;
|
||||
- GCP: disks, GKE node pools;
|
||||
- **structure:** subscriptions, resource groups, management groups, projects, folders and organisations, which
|
||||
are counted but aren't assets.
|
||||
- **A live plan is built in code, with no model call** (`app/plan.py`, `pre_process`):
|
||||
- one `scan_target` per scope entry, in batches of 10;
|
||||
- a `classify_data` per data store;
|
||||
- one `collect_network` and one `trace_dependencies`;
|
||||
- `source_gaps` set by the same rules as before.
|
||||
|
||||
The run context still only narrows (`context.scope.regions`, `context.scope.exclude`), as declared gaps. The new
|
||||
`scope_accounting` output gives `listed`, `scope_entries`, `folded` (each with its rule) and `structure`. The sample
|
||||
estate keeps its model and prompt unchanged.
|
||||
- **An incomplete listing fails the planner step.** A refused or failed page, or more than 100 pages, means no plan
|
||||
is made, rather than a plan of part of the estate.
|
||||
- **The critic counts coverage itself.** Its code lists the scope again with the same rules: `app/scope.py` is
|
||||
identical in both agents, and a test compares the two files. It compares that listing with the findings and the
|
||||
planner's declared gaps, and gives the model `live_scope`: counts, the uncovered ids (up to 200, with the exact
|
||||
count) and any listing issues. The model copies it into `coverage_gaps` and `coverage_complete`. A listing issue
|
||||
fails coverage.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Supporting services are discovered, inventoried and reported: Key Vault, identities, DNS, messaging, private
|
||||
endpoints, registries, monitoring and so on. The Well-Architected review and Assessment (ALZ-ASM-05) see the same
|
||||
estate.
|
||||
- The critic can't pass a run that skipped a resource type.
|
||||
- **Scale:**
|
||||
- At subscription scope, about 650 resources become about 600 scan tasks. The planner handles that in code.
|
||||
- The scan executor (ALZ-DSC-02) still has a model call each resource's metadata in one request. That fits a
|
||||
resource group, but not a large subscription.
|
||||
- Batching the scan in code, as ADR 0015 did for the inventory, is the next change before a subscription-wide run.
|
||||
- Folding is a fixed list. A new child-component type shows up as a scope entry until it's added, which over-counts
|
||||
but never drops anything.
|
||||
@@ -0,0 +1,79 @@
|
||||
# 0018. Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "the model should ingest the data and normalise it to the data schema that we own and judge". Plan and decisions D1–D4: `engineering_hub_responsiblities/reports/cp1/cmdb-upload-ingestion-plan-2026-11-05.md` |
|
||||
| Requirements changed | ALZ-DSC-04 (ADO 3214454): a live run reads CMDB exports uploaded with the run. ALZ-DSC-01 (ADO 3214451): an uploaded CMDB is a source, not a gap |
|
||||
|
||||
## Context
|
||||
|
||||
- **A live run never read a CMDB.** `cmdb` was always a source gap, and the reconciliation step returned nothing.
|
||||
- **A second uploaded file broke the run.** The upload step joins files with `\n\n---\n\n`. The planner, inventory
|
||||
and report parsed the whole text as one YAML document, failed, and carried on as if no run context had been
|
||||
given.
|
||||
- **Exports differ in layout.** CMDB exports differ by customer and tool, so no column can be fixed in code.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Uploads are listed in the run context.** The context's `uploads:` list names the files uploaded after it, in
|
||||
order, each with a `source` (e.g. `cmdb`). `kab_run_workflow.py` uploads them. The shared `app/run_context.py`
|
||||
(identical in the planner, inventory, reconciliation and report agents) splits the text into the context and
|
||||
each listed file. It cuts a CSV, or each XLSX sheet, into a header line and one line per row. It knows no
|
||||
column.
|
||||
- **The model normalises; code judges.** The `normalise_cmdb_batch` sub-agent maps each batch of rows (25 by
|
||||
default, each batch with its header) to `schemas/cmdb_ci.json`. Every string is copied verbatim; `environment`
|
||||
and `lifecycle` are the only judgments; any other column can be kept in `other_attributes`. Code rejects a reply
|
||||
unless:
|
||||
- every row has exactly one result (or a reasoned `skip`);
|
||||
- every value is found in its row;
|
||||
- every key is a header column;
|
||||
- every judgment is an allowed value.
|
||||
|
||||
A batch gets at most three tries (ADR 0015's rule). One that fails all three, or an upload the platform couldn't
|
||||
parse, fails the step.
|
||||
- **Injections:** a row is flagged when the model or a code pattern sees an instruction. The quoted text goes to
|
||||
the critic (`cmdb_injections`) and to the report.
|
||||
- **Reconciliation is code** (`app/reconcile.py`), against the step's own scope listing (`app/scope.py`, the
|
||||
planner's rules) and the inventory:
|
||||
- **match:** by cloud resource id, otherwise by name when exactly one listed resource has it;
|
||||
- **stale attributes:**
|
||||
- environment, compared at enum level (D1);
|
||||
- location, compared with the record's site;
|
||||
- a retired CI whose resource exists (`investigate`);
|
||||
- owner and business service, when the record carries them;
|
||||
- **ghost:** an unmatched, not-retired CI whose id lies in a scope container the listing shows (D2). It's
|
||||
`investigate` outside `context.scope.regions`, in `exclude`, or with an unknown lifecycle, and otherwise
|
||||
`retire_ci`. A CI with no cloud id is `unplaced` (D3), and one in another container is `out_of_scope`. Neither
|
||||
is drift.
|
||||
- **orphan:** an inventory record no CI matched (`add_ci`). Folded components are never orphans.
|
||||
- **The planner** adds one `ingest_cmdb` task per uploaded CMDB file, and drops the `cmdb` gap when
|
||||
`include_cmdb` is `yes` and a file is uploaded.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **New layouts need no code change.** A new export layout, whether a ServiceNow CSV or an asset register XLSX
|
||||
with other column names, is handled by the model's mapping. Code checks that mapping on every row.
|
||||
- **The model never decides drift.** A manipulated normaliser can only return values that are in the row.
|
||||
- **The step makes model calls** (one per batch) on a live run with a CMDB. Without one, it makes none.
|
||||
- **Subscription scope is cautious.** At subscription scope, a CI in a resource group with no listed resources is
|
||||
`out_of_scope`, not a ghost. Returning the connection's scope root from the existing discovery servers would
|
||||
make this exact; that is left until it's needed.
|
||||
- **Planner and step listings can differ.** The reconciliation step lists the scope again, so a resource created
|
||||
between the planner's listing and this one is matched, not a ghost.
|
||||
|
||||
## Changes after the first live run (2026-11-05)
|
||||
|
||||
Run `exec_a0a3c486ec9b` (case `azure-crucible-kab-dev-cmdb`) passed 13 of 19 checks. Every planted CMDB result
|
||||
was found except the environment difference.
|
||||
|
||||
- **`env` tags were redacted** by the discovery servers' shared `_scrub` (`env` is a secret key, for a container's
|
||||
environment variables). The inventory then took `env: [redacted]` over `Environment: dev`, and 8 of 10 records
|
||||
had environment `unknown`, so the planted environment difference couldn't be found. A single value under
|
||||
`env` is now kept; a list or map under it is still redacted (azure-discovery 0.1.6, gcp-discovery 0.1.4,
|
||||
aws-discovery 0.1.2).
|
||||
- **Orphans name their resource** (`field` `resource`, `discovered_value` the resource id). Before, they named only
|
||||
the inventory's canonical id (reconciliation 0.5.1).
|
||||
- **The critic** no longer raises "drift with no CMDB source" when `source_gaps` has no `cmdb` (critic 0.5.2).
|
||||
- **The report's drift table** shows both the asset and the CMDB record (report 0.7.2).
|
||||
@@ -0,0 +1,69 @@
|
||||
# 0019. A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "the cmdb nodes should be separate from the hyperscaler but still be built in an equivalent way with children and parents at an edge"; "it's the same graph but different edge nodes". Plan and decisions D1–D7: `engineering_hub_responsiblities/reports/cp1/live-dependency-map-plan-2026-11-05.md` |
|
||||
| Requirements changed | ALZ-DSC-06 (dependency mapping): a live run's graph comes from the scan's recorded dependencies and the CMDB's relationships. ALZ-DSC-04: CMDB relationships are read. ALZ-DSC-08: the critic checks dependencies in code |
|
||||
|
||||
## Context
|
||||
|
||||
Run `exec_93742bdd7e29` (case `azure-crucible-kab-dev-cmdb`) passed all 19 checks, and the critic failed it.
|
||||
|
||||
- **The dependency step never received the scan's findings.** The scan recorded 7 dependencies. The step rebuilt
|
||||
edges from each record's network attributes, dropped 4 of them and added 2 VNet edges the scan never recorded.
|
||||
- **Out-of-scope targets were bare names.** Dependencies outside the scope root (subnets in `crucible-net-dev`, a
|
||||
public IP in `crucible-kab-dev-nodes`) came through as names, so the critic reported them as unsupported.
|
||||
- **The critic was inconsistent.** It passed the same estate the day before, so the verdict depended on the
|
||||
model's judgment that day.
|
||||
- **The CMDB's own relationships were never read.**
|
||||
|
||||
## Decision
|
||||
|
||||
- **One graph, three kinds of node** (`app/graph.py` in the dependency agent, from `pre_process`, with no model
|
||||
call):
|
||||
- `resource`: a scope entry, i.e. its inventory record;
|
||||
- `external`: a dependency target outside the scope root or not inventoried, with its container and what depends
|
||||
on it (D1);
|
||||
- `ci`: a judged CMDB CI, with its reconciliation result.
|
||||
- **Every edge runs parent (`source`) to child (`target`):**
|
||||
- **resource → resource or external:** one edge per entry in a finding's `dependencies`. They're typed by the
|
||||
target's resource type, with confidence 0.8 and `auto_approved`, since they're the provider's own record (D3).
|
||||
- **ci → ci:** a CMDB relationship. Confidence 0.7 and `sme_review_recommended`.
|
||||
- **ci → resource, `represents`:** the reconciliation's match. `auto_approved` when matched by cloud id;
|
||||
`sme_review_recommended` when matched by name.
|
||||
- **Folded components:** a dependency on a resource's own child component (a NIC, a disk) is a
|
||||
`folded_dependencies` entry, not an edge (D2).
|
||||
- **Relationships are judged like CIs** (ADR 0018). A relationship export (`source: cmdb_relationships` in the
|
||||
context's `uploads:`, D6), or a CI row's own parent or child columns (`related`), is normalised by the model to
|
||||
`schemas/cmdb_relationship.json`. It's typed with the graph's edge types (D7), and every value is checked in code
|
||||
against its row. An end that resolves to no CI (by `ci_ref`, or by unique name) is dangling, never a node. This
|
||||
is done in the CMDB step (D5).
|
||||
- **The comparison is a walk over the one graph.** For each ci → ci edge whose two CIs both represent resources:
|
||||
- the cloud has the same resource → resource edge, in the same direction: `confirmed`. The cloud edge gains
|
||||
the CMDB evidence, and its confidence becomes 0.95;
|
||||
- otherwise: `cmdb_only`.
|
||||
|
||||
A cloud edge between two represented resources that no ci → ci edge matches is `discovered_only`. A relationship
|
||||
with an end that has no resource is `not_comparable`.
|
||||
- **The critic counts dependencies in code** (`live_dependencies`): every recorded dependency must be an edge (to a
|
||||
record, or to an external node) or a folded dependency. It also treats `external`, `represents` and ci → ci edges
|
||||
as supported by their own evidence.
|
||||
- **The report:**
|
||||
- one diagram, with each node drawn by kind: a resource as a box, external with a dashed outline, a CI as a
|
||||
hexagon, and `represents` as dotted;
|
||||
- tables of dependencies outside this discovery (with where to widen the scope, D4), the CMDB comparison, and
|
||||
dangling relationships.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No recorded dependency is dropped and none is invented. The critic's dependency verdict is the same on every run.
|
||||
- An assessment sees what a landing zone depends on outside the scope, for example the shared VNet in
|
||||
`crucible-net-dev`, and which resource group to add to discover it.
|
||||
- The CMDB's picture and the cloud's are in one graph, so a CI's dependents, and an asset's dependents in either
|
||||
source, are one walk.
|
||||
- `discovered_only` is reported in the dependency step and the report. It doesn't add a drift item, because the
|
||||
CMDB step (s4) runs before the graph (s7).
|
||||
- Folding is a fixed list, so a dependency on another asset's child component (another VM's disk) is still folded,
|
||||
not redirected to its parent: the listing doesn't say which parent owns it.
|
||||
@@ -0,0 +1,75 @@
|
||||
# 0020. One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "lets go ahead with the enhancements when these are not combined and then later that can be decided"; the solution architecture agents are a separate stream, and Discovery's outputs are used only by the Assessment workflow. Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phases 0 and 1 |
|
||||
| Requirements changed | ALZ-F02 ASM-01 (Discovery hand-off): F02 selects one package artifact rather than one per Discovery step (F02 ADR 0002). New step `s13`, agent `cp1-discovery-package-agent` |
|
||||
|
||||
## Context
|
||||
|
||||
- **F02 reads Discovery by hand.** Its Artifacts step needs each Discovery step's artifact id typed in, and every F02
|
||||
agent receives all of them as one text, joined by `---` (engineering hub review
|
||||
`reports/cp1/discovery-to-assessment-requirements-review-2026-11-04.md`, section 1).
|
||||
- **Nothing measured whether a run was useful.** A case either passed its checks or it didn't. Nothing showed how
|
||||
much of what F02 needs a run gives, how many decisions it leaves for a person, or what it cost.
|
||||
- **KAB can't write a .zip.** It can expand a selected zip, but no step type creates one. Every step's output is kept
|
||||
as a JSON artifact, `<execution_id>_<step_id>`.
|
||||
- **Discovery and Assessment stay separate workflows** for now; combining them is a later decision.
|
||||
- **The idea is borrowed, not the code.** The original `gcp_solution_architecture_agent` writes a deliverable folder
|
||||
with an `execution.json` and measures its runs with limits per case. Those ideas are used here; nothing is ported,
|
||||
and no hand-over to the solution architecture stream is built.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **A package step, `s13`, answered in code** (`cp1-discovery-package-agent`, `app/package.py`, no model, no tools):
|
||||
- It runs after the report's review point, beside the two page steps, so a package exists only for a run the
|
||||
Engagement Lead let through.
|
||||
- Its output holds every step's output fields, **copied unchanged**, as named parts: `plan`, `scan`, `inventory`,
|
||||
`cmdb`, `classification`, `network`, `dependencies`, `critic`, `well_architected`, `report`. Parts are named by
|
||||
meaning, because KAB renames step ids on import. The report part keeps `counts` and `report_markdown`, not the
|
||||
HTML.
|
||||
- Its `manifest` holds the engagement, run, execution id, provider, time, critic verdict, each agent's version
|
||||
(a static input set when the workflow is built), the planner's source gaps, a `sha256` and item count per part,
|
||||
the parts absent, `complete`, `problems` and the scorecard.
|
||||
- `complete` is true only when the required parts (plan, inventory, classification, network, dependencies,
|
||||
critic, report) are present and the critic passed. The CMDB and Well-Architected parts are optional.
|
||||
- Each `sha256` is over the part's canonical JSON (`sort_keys`, `separators=(",", ":")`, UTF-8), so a consumer
|
||||
can recompute it before trusting the part.
|
||||
2. **A value scorecard** (`app/scorecard.py`), in the manifest and in `scripts/evaluate_workflow_run.py`, from the
|
||||
same code:
|
||||
- `f02_fields`: each F02 field (application, owner, criticality, environment, support status, end of support,
|
||||
compliance scope, annual cost, data class), per record, as `filled`, `declared_gap`, `missing` or
|
||||
`not_applicable`;
|
||||
- `f02_run`: the run-level facts F02's ASM-01 checks;
|
||||
- `decisions`: what the run leaves for a person, by kind;
|
||||
- `findings`: how many findings name an owner and an action.
|
||||
`filled` and `filled_or_declared` are both reported: a declared gap is honest, but it isn't value.
|
||||
3. **Cases carry a value bar and limits.** `value:` (for example `f02_filled_min`) holds the baseline so a change
|
||||
can't lower it; `limits:` bound tokens, model calls and agent seconds, read from the agents' logs by
|
||||
`scripts/collect_run_usage.py`. Both fail as major.
|
||||
4. **F02 is unchanged by this ADR.** Its Artifacts step can select the package today. Teaching F02 to verify the
|
||||
manifest in code is F02's own change, with its own ADR.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **First live runs** (2026-11-05, both after the Engagement Lead continued): `exec_14956d524a8b`
|
||||
(`azure-crucible-kab-dev-cmdb`, 46 of 46 checks) and `exec_d1c242609f82` (`azure-crucible-kab-dev`, 19 of 19).
|
||||
Both packages are complete, and every part's `sha256` recomputes.
|
||||
- **Baseline:** 24% of F02's fields filled in both cases; 27% filled or declared with a CMDB, 51% without one (the
|
||||
CMDB gap declares owners and applications); 6 of 7 run checks met (no applications list); 9 decisions with a CMDB;
|
||||
findings 0% with an owner, 100% with an action. Owners are in the CMDB export but never reach the inventory
|
||||
records, and nothing reads cost, lifecycle or compliance. These are phases 3 and 4 of the plan.
|
||||
- **Size:** about 100 KB with a CMDB and 78 KB without, against KAB's 5 MB inline limit. A large estate will need
|
||||
per-application summaries (plan phase 3) or a platform zip step.
|
||||
- **The review record is implied, not explicit.** The package proves the run continued past review, but not who
|
||||
approved it or their comments. That still needs the platform to expose review records to later steps.
|
||||
- **Every Discovery output field now has a reader:** the context report's "unread" fields reach F02 through the
|
||||
package, except the run's own ids.
|
||||
- **The package input schema has no enums**, so the build-time contract check can never refuse a producer's value
|
||||
at this step. A new field needs a row in `INPUTS`, its wiring in `build_workflow.py`, and a test checks the two
|
||||
match.
|
||||
- **ADR 0016 (Proposed) needs the owner's review:** it carries one run context file through to solution
|
||||
architecture, which the decision of 2026-11-05 to keep the streams independent rules out. Plan phase 2 borrows
|
||||
only the scope spec's field meanings, within F01.
|
||||
@@ -0,0 +1,59 @@
|
||||
# 0021. A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "CMDB→tag→RG, review model" (code groups by CMDB business service, then the application tag, then resource group; the model proposes groups only for what's left, code judges them, and model-proposed groups are flagged for the Engagement Lead's review). Plan: [../discovery-value-plan.md](../discovery-value-plan.md), phase 3 |
|
||||
| Requirements changed | ALZ-DSC-03 and ALZ-F02 ASM-01 ("inventory (applications and assets)"): a live run lists applications, from the dependency step rather than the inventory. ALZ-DSC-04: each CI's result carries its owner, business service and support group. ALZ-DSC-06: the dependency step groups applications and names owners |
|
||||
|
||||
## Context
|
||||
|
||||
- **F02 needs applications.** Its planner groups by the Discovery `applications` list, else by `facts.business_service`,
|
||||
and live runs had neither, so every asset was a gap (review of 2026-11-04, item 5). The value scorecard's baseline
|
||||
([ADR 0020](0020.md)) showed application and owner missing on all 10 records of the CMDB case.
|
||||
- **The evidence was there but never reached Assessment.** The uploaded CMDB names each CI's `owned_by` and
|
||||
`business_service`, but the CMDB step's per-CI results dropped them. Every resource carries an `Application` tag.
|
||||
- **The dependency step is the right place:** it runs after the CMDB step, already joins records to CIs (ADR 0019),
|
||||
and is answered in code.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **The CMDB step's `cis`** carry each CI's `owner`, `business_service` and `support_group`, verbatim (0.7.0).
|
||||
2. **The dependency step groups each record, once,** in this order (`app/applications.py`, 0.6.0):
|
||||
1. the matched CI's business service (`basis: cmdb_business_service`);
|
||||
2. the resource's application tag (`Application`, `app`, `application_name`, `app_name`, `business_service`,
|
||||
`service`, any case; `basis: tag`). **Tag alias:** when every CMDB-grouped member of one application carries
|
||||
the same application tag, a tag-only record with that tag joins that application, flagged for review. A tag
|
||||
two applications claim is no alias;
|
||||
3. its resource group or project (`basis: container`), flagged for review;
|
||||
4. for what's left only, the `propose_application_groups` sub-agent, from names, types and environments. Code
|
||||
rejects the whole reply if any group names an asset not asked about, places an asset twice, or has an empty,
|
||||
over-long or instruction-like name. Every model group is flagged for review; a rejected or failed reply leaves
|
||||
its assets ungrouped, which the note says.
|
||||
3. **Owners:** from the matched CI (`owner`, else `support_group`), else an owner tag. An application's owner is set
|
||||
only when its members' owners agree (`owners_disagree` otherwise). No owner is inferred.
|
||||
4. **Outputs:** `applications` (id, name, basis, members, owner, business service, evidence, `review_required` and
|
||||
reasons), `asset_owners`, `ungrouped_assets` and `applications_note`. The report shows them after the inventory
|
||||
and lists the groups to review (0.9.0); the package carries them as its `applications` part, and the scorecard
|
||||
reads a record's application and owner from it (package 0.2.0, scorecard version 2).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Expected on the CMDB case:** one application, "KAB Platform", of 10 assets: 8 by the CMDB, 2 (the DNS zone and
|
||||
the Event Grid topic, which have no CI) by the `Application: AppName1` tag every CMDB-grouped member shares,
|
||||
flagged for review. Owner "Platform Engineering" for the 8 matched assets; the 2 orphans have none, correctly.
|
||||
Scorecard: F02 fields filled from 24% to about 46%; run checks 7 of 7; findings with an owner from 0% to about 33% (the stale
|
||||
attributes on owned assets; orphans and ghosts have none).
|
||||
- **Without a CMDB:** one application, "AppName1", by tag, needing no review, and no owners: the CMDB gap declares
|
||||
them.
|
||||
- **The model is rarely asked:** on Azure and Google Cloud every resource has a resource group or project, so step 4
|
||||
runs only for estates with no container. Its tests use a stand-in model.
|
||||
- **Applications are not graph nodes.** They're a separate list, so no edge or node enum changed and no other step's
|
||||
input schema had to. Membership edges can be added later if a consumer needs them in the graph.
|
||||
- **The review is the Engagement Lead's,** at the report's review point. Assessment reads `review_required` and
|
||||
can decide how to treat a group the lead didn't change.
|
||||
- **Live, 2026-11-05:** `exec_58ba6a71d00b` (`azure-crucible-kab-dev-cmdb`) passed 56 of 56 checks: one application,
|
||||
KAB Platform, of 10 assets, flagged for review; 8 owners; 46% of F02's fields filled; 7 of 7 run checks.
|
||||
`exec_925194d79c47` (`azure-crucible-kab-dev`) passed 21 of 21: AppName1 by tag, no owners, 37% filled. Neither
|
||||
run asked the grouping model.
|
||||
@@ -21,9 +21,19 @@ Each agent's `docs/adr/README.md` lists the records that change its own story.
|
||||
| [0005](0005.md) | Inventory before CMDB reconciliation; the inventory gets CMDB records from `alz-inventory` | Proposed | DSC-03, 04 |
|
||||
| [0006](0006.md) | The critic's verdict is gated by human approval, not by code | Proposed | DSC-08 |
|
||||
| [0007](0007.md) | Agent names | Proposed | All |
|
||||
| [0008](0008.md) | Models | Proposed | All |
|
||||
| [0008](0008.md) | Models: executors on `gpt-5.6-sol`, the critic on `gpt-5.4` (2026-11-03) | Accepted | All |
|
||||
| [0009](0009.md) | Deploy with kustomize; keys stay in Secrets | Accepted | — |
|
||||
| [0010](0010.md) | Step outputs travel through the workflow; there is no shared store | Proposed | All |
|
||||
| [0011](0011.md) | Uploaded files and project artifacts reach the planner and the inventory as text | Accepted | DSC-01, 03 |
|
||||
| [0012](0012.md) | The Discovery report comes from code, and the approval gate moves to it | Accepted | report (new) |
|
||||
| [0013](0013.md) | A `provider` input lets one workflow read the sample estate or a live cloud | Accepted | All |
|
||||
| [0014](0014.md) | A Well-Architected review step in Discovery, judged in code | Accepted | DSC-10 (new), report |
|
||||
| [0015](0015.md) | A live run's inventory is built in batches, at most three tries per batch; a batch that fails all three fails the step | Accepted | DSC-03 |
|
||||
| [0016](0016.md) | The run context file carries an intent, in the solution-architecture scope spec's format | Proposed | DSC-01, DSC-10, package, F02 |
|
||||
| [0017](0017.md) | A live run's scope is every resource under the scope root, planned and checked in code | Accepted | DSC-01, DSC-08 |
|
||||
| [0018](0018.md) | Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code | Accepted | DSC-01, DSC-03, DSC-04, DSC-08, report |
|
||||
| [0019](0019.md) | A live run's dependency graph is built in code: one graph of resource, external and CMDB CI nodes | Accepted | DSC-04, DSC-06, DSC-08, report |
|
||||
| [0020](0020.md) | One Discovery package for the Assessment workflow, with a manifest and a value scorecard, built in code | Accepted | package (new), F02 ASM-01 |
|
||||
| [0021](0021.md) | A live run's applications and asset owners are grouped in code: CMDB, then tag, then container, then a judged model proposal | Accepted | DSC-03, DSC-04, DSC-06, report, package, F02 ASM-01 |
|
||||
|
||||
New records copy [template.md](template.md) and take the next number.
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# Discovery output value: plan
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-05 |
|
||||
| Status | Phases 0 and 1 live ([ADR 0020](adr/0020.md)); the rest is planned |
|
||||
| Decided by | Product owner, 2026-11-05: go ahead, with Discovery and Assessment kept as separate workflows; combining them is decided later |
|
||||
|
||||
## 1. Purpose and boundaries
|
||||
|
||||
Make the live Discovery workflow's output useful to the people and workflow that consume it, using ideas from the
|
||||
original `gcp_solution_architecture_agent`.
|
||||
|
||||
- **Consumers:** the Engagement Lead at the review point, and the Assessment workflow (ALZ-F02), assuming F02 is
|
||||
still required. No other workflow or pattern consumes Discovery's output.
|
||||
- **Separate streams:** the solution architecture agent (`outcome-pattern/`) and the sub-agent pattern are
|
||||
independent. No hand-over is built to them; only ideas are borrowed from the original agent, never its code or
|
||||
engine.
|
||||
- **The owner's order holds:** Azure working (done 2026-11-05), then the package step, then Discovery meets every
|
||||
F02 requirement.
|
||||
- **Discovery and Assessment stay separate workflows.** Whether to also offer one combined run is decided later.
|
||||
|
||||
## 2. What "meaningful value" means
|
||||
|
||||
| Consumer | The output must answer |
|
||||
|---|---|
|
||||
| Engagement Lead | What's there, what's wrong, what's missing, and what must be decided |
|
||||
| F02 Assessment | For each application: its assets, dependencies, support status, data class, compliance, cost and owner, each with evidence |
|
||||
|
||||
The **value scorecard** (`cp1-discovery-package-agent/app/scorecard.py`) measures this for every run, in code: F02
|
||||
fields filled, declared or missing; F02's run-level checks; decisions left for a person; findings with an owner and
|
||||
an action. Cases also carry limits on tokens, model calls and agent seconds.
|
||||
|
||||
**Baseline**, case `azure-crucible-kab-dev-cmdb`, run `exec_14956d524a8b`, 2026-11-05 (46 of 46 checks):
|
||||
|
||||
| Measure | Value |
|
||||
|---|---|
|
||||
| F02 fields filled | 24% |
|
||||
| F02 fields filled or declared | 27% |
|
||||
| Missing everywhere | application, owner, support status, end of support, compliance scope, annual cost |
|
||||
| F02 run checks | 6 of 7 (no applications list) |
|
||||
| Decisions for a person | 9 |
|
||||
| Findings with an owner / an action | 0% / 100% |
|
||||
| Cost | about 106k tokens, 10 model calls, 108 agent seconds |
|
||||
| Package size | about 100 KB |
|
||||
|
||||
## 3. Borrowed from the original
|
||||
|
||||
| Idea | In the original | In Discovery |
|
||||
|---|---|---|
|
||||
| Field meanings: `ask`, `decide`, `not_applicable`, provenance, a catalogue, a `check` command | ADR-0008, `app/scope.py` | The run context (phase 2) |
|
||||
| An explicit graph, and an id chain code won't break | ADR-0005, the requirements chain | Application nodes in the ADR 0019 graph (phase 3); evidence on every finding (phase 4) |
|
||||
| A deliverable manifest; customer questions | `execution.json`, `customer-questions.md` | The package manifest (phase 1); decisions needed (phase 4) |
|
||||
| A Baseline and its changes | ADR-0006, `changes.json` | Re-discovery against the last approved run (phase 5) |
|
||||
| An evaluation harness | `solution_eval`, judge calibration, limits | Value bars and limits (phase 0); a judge (phase 6) |
|
||||
| Lessons and overlays | ADR-0004 | Phase 7 |
|
||||
|
||||
**Left behind:** its own server, database, LangGraph and `interrupt()` (they conflict with KAB); one model for every
|
||||
role (Discovery's critic stays on a different model); Jev and unscrubbed free-text search (data would leave the
|
||||
gateways).
|
||||
|
||||
## 4. Phases
|
||||
|
||||
Sizes are relative. "F02-only" marks work that only F02 justifies.
|
||||
|
||||
| # | Phase | Size | What | Done when | Status |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | Value baseline | S | The scorecard; `value:` bars and `limits:` in both Azure cases; `scripts/collect_run_usage.py` | Both cases have a baseline | **built** (ADR 0020) |
|
||||
| 1 | Discovery package | M, F02-only | Step `s13`, `cp1-discovery-package-agent`: every part unchanged, a manifest with sha256 per part, completeness and the scorecard | An F02 run starts from one artifact id | **live** (ADR 0020): both Azure cases pass with complete packages; F02's own change to verify the manifest is next |
|
||||
| 2 | Run context field meanings | M | A catalogue for `context.yaml` (value, `ask`, `decide`, `not_applicable`; essential, important, optional) covering compliance scope, criticality, owner, business services; `check_context.py`; provenance per fact; assumptions shown apart | A `decide` gives a visible assumption; an unset essential field stops the run before it scans | todo. ADR 0016 (Proposed) needs the owner's review first: it carries the file to solution architecture |
|
||||
| 3 | Close F02's data gaps | L, mostly F02-only | Applications as graph nodes (CMDB business service, then tags, then resource group or naming, then model-proposed and judged in code); CMDB owners onto records; end-of-life rules in code over `provider_attributes`; cost through fixed-shape queries on the existing discovery servers; per-application summaries for size; the repositories source | The Azure case reaches the agreed F02 bar | **applications and owners built** ([ADR 0021](adr/0021.md)): CMDB→tag→container→judged model, owners from the CMDB or an owner tag; expected 46% filled and 7 of 7 run checks. End-of-life, cost and per-application summaries still todo |
|
||||
| 4 | A report that leads to decisions | M | First page: decisions needed, top risks, open questions. Each finding has a severity, an owner (else "unowned"), an action and its consumer; code refuses a finding without evidence or an action | 100% of findings with an owner and an action | todo |
|
||||
| 5 | Re-discovery against a Baseline | M | A run may name an earlier approved execution; a code-only diff step gives added, removed and changed assets, edges and drift | Identical runs give an empty diff; a planted change shows | todo |
|
||||
| 6 | Evaluation harness | M | A report judge on a different model, calibrated on 10–20 labelled sections; runs with and without a feature | Each phase's effect is measured | todo |
|
||||
| 7 | Lessons and overlays | M | Repeated critic findings become approved rules; customer standards arrive as overlays | — | later |
|
||||
|
||||
Phases 0, 2, 4, 5 and 6 are worth doing whatever happens to F02; 1 and 3 are justified by F02.
|
||||
|
||||
## 5. Decisions for the owner
|
||||
|
||||
1. **Is F02 still required?** It gates the rest of phase 3.
|
||||
2. **ADR 0016:** the owner will go through its fields one by one before phase 2 starts: [run-context-field-review.md](run-context-field-review.md).
|
||||
3. ~~**Application grouping**~~ decided 2026-11-05: CMDB, then tag, then resource group, then a judged model
|
||||
proposal, with model groups reviewed (ADR 0021).
|
||||
4. **Cost and end-of-life sources:** cost queries on the existing servers (needs Cost Management Reader or the
|
||||
equivalent) or an uploaded file; end-of-life from a table in code or the provider's lifecycle data.
|
||||
5. **The value bar** that counts as "Discovery meets F02".
|
||||
6. **Later:** a combined Discovery and Assessment run, beside the separate workflows.
|
||||
|
||||
## 6. Risks
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| The package outgrows KAB's 5 MB inline limit | Per-application summaries (phase 3), or a platform zip step |
|
||||
| Wrong application groups mislead F02's scoring | Checks in code, the critic, and review of model-proposed groups |
|
||||
| Cost and lifecycle need permissions beyond Reader | Update `docs/customer-cloud-access.md`; a missing permission is a declared gap |
|
||||
| Borrowing turns into porting | Each borrowing gets its own ADR; ideas only |
|
||||
| Model cost grows with new steps | The limits in each case; code-only steps wherever possible |
|
||||
@@ -0,0 +1,99 @@
|
||||
# Run context: field-by-field review for ADR 0016
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-05 |
|
||||
| For | The product owner, going through [ADR 0016](adr/0016.md) (Proposed) field by field before value plan phase 2 |
|
||||
| Source | The scope spec catalogue the ADR borrows (`outcome-pattern/solution_architecture/.../skills/scoping/assets/catalogue.yaml`), 51 fields |
|
||||
| Status | Recommendations only. Nothing is built from this until the decisions column is filled in |
|
||||
|
||||
## How to read it
|
||||
|
||||
- **Boundary (decided 2026-11-05):** Discovery's output feeds only Assessment (F02). The solution architecture stream
|
||||
is separate, so nothing here is a hand-over to it, and F01 would keep its **own** copy of whatever fields it adopts,
|
||||
with its own catalogue. ADR 0016's points 1 and 5 (one file through to solution architecture; a shared catalogue)
|
||||
would be dropped.
|
||||
- **Who settles it:**
|
||||
- **Evidence:** Discovery can read the current value from the cloud or the CMDB, and records it with its
|
||||
provenance. The customer may still state a target; Discovery then reports current against stated.
|
||||
- **Customer:** only the customer can say it. Discovery reads it and uses it; it can be `ask`, `decide` or
|
||||
`not_applicable`.
|
||||
- **Not Discovery:** a design or delivery choice. It belongs to a later workflow, not to this file.
|
||||
- **Used by:** the step or consumer that would act on it today or in a planned phase.
|
||||
- **Recommendation:** **Keep** (adopt now), **Later** (worth having, no consumer yet), **Drop** (not Discovery's, or
|
||||
already covered).
|
||||
|
||||
Discovery's own sections stay as they are: `workflow_inputs`, `generated`, `uploads`, `customer`, `plan`, and
|
||||
`context` (`scope`, `sources`, `owners`, `business_services`, `assets`).
|
||||
|
||||
## Fields
|
||||
|
||||
| # | Field | What it is | Who settles it | Used by | Recommendation | Decision |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | `metadata.customer_case` | The engagement's short reference | Customer | All | **Drop**: `workflow_inputs.engagement_id` already is this | |
|
||||
| 2 | `engagement.kind` | Workload, platform foundation or migration | Customer | Planner (what's essential), F02 | **Keep** | |
|
||||
| 3 | `intent.statement` | What the customer wants, in their words | Customer | Report (shown as data) | **Keep**, as data only: never sent in a tool call, never passed to every step | |
|
||||
| 4 | `intent.requirements` | Stated requirements with ids (`REQ-01`) | Customer | Well-Architected review (cite the `REQ-` behind a gap), report | **Keep** | |
|
||||
| 5 | `intent.exclusions` | What's out of scope | Customer | Planner | **Drop**: `context.scope.exclude` already holds this for Discovery | |
|
||||
| 6 | `intent.success_criteria` | How the customer judges success | Customer | — | **Later** | |
|
||||
| 7 | `estate.cloud_provider` | The provider | Customer | — | **Drop**: `workflow_inputs.provider` is this | |
|
||||
| 8 | `estate.mode` | Greenfield, brownfield expansion or migration | Customer | Planner, F02 | **Keep** | |
|
||||
| 9 | `estate.regions.primary` | The main region | Evidence | Planner, report | **Keep**, settled from evidence; `context.scope.regions` stays the scan's filter | |
|
||||
| 10 | `estate.regions.secondary` | A second region | Evidence | Report | **Keep**, settled from evidence | |
|
||||
| 11 | `estate.regions.resiliency_tier` | Single region, warm standby or active-active | Evidence (current); customer (target) | Well-Architected reliability | **Later** | |
|
||||
| 12 | `estate.data_residency` | Where data must stay | Customer | F02 (compliance), Well-Architected | **Keep** | |
|
||||
| 13 | `estate.existing_systems` | What exists today | Evidence | — | **Drop** as a field: it is Discovery's whole output (inventory, applications) | |
|
||||
| 14 | `workload.users.who` | Who uses it | Customer | — | **Later** | |
|
||||
| 15 | `workload.users.count` | How many users | Customer | — | **Later** | |
|
||||
| 16 | `workload.users.peak_concurrent` | Peak concurrent users | Customer | — | **Later** | |
|
||||
| 17 | `workload.availability.target` | Availability, as a percentage | Customer | Well-Architected reliability, F02 risk | **Keep**, optional | |
|
||||
| 18 | `workload.availability.hours` | Business hours, extended or always | Customer | F02 risk | **Later** | |
|
||||
| 19 | `workload.availability.recovery_time` | RTO | Customer | Well-Architected reliability, F02 | **Keep**, optional | |
|
||||
| 20 | `workload.availability.recovery_point` | RPO | Customer | Well-Architected reliability (backup checks), F02 | **Keep**, optional | |
|
||||
| 21 | `compliance.industry` | The customer's industry | Customer | Report | **Drop**: `customer.industry` already holds it | |
|
||||
| 22 | `compliance.frameworks` | Regulations and standards that apply | Customer | F02 `compliance_scope` (gap 11), Well-Architected policy sets | **Keep**, essential: replaces `customer.regulatory` | |
|
||||
| 23 | `compliance.data_classification` | How sensitive the data is | Customer | Classification step (stated against found), F02 | **Keep** | |
|
||||
| 24 | `compliance.encryption.type` | Provider- or customer-managed keys | Evidence | Well-Architected security | **Keep**, settled from evidence | |
|
||||
| 25 | `compliance.encryption.key_rotation_days` | Key rotation period | Evidence (partly) | Well-Architected security | **Later** | |
|
||||
| 26 | `compliance.audit_logging.retention_days` | Audit log retention | Evidence (partly) | Well-Architected operations | **Later** | |
|
||||
| 27 | `compliance.audit_logging.immutable_retention_policy` | Undeletable audit logs | Evidence (partly) | Well-Architected security | **Later** | |
|
||||
| 28 | `networking.exposure` | Internal only or internet-facing | Evidence | Network step, Well-Architected security | **Keep**, settled from evidence | |
|
||||
| 29 | `networking.topology` | The network topology | Evidence | Network step, report | **Keep**, settled from evidence | |
|
||||
| 30 | `networking.hybrid_connectivity.enabled` | Private links to on-premises or another cloud | Evidence | Network step, F02 | **Keep**, settled from evidence | |
|
||||
| 31 | `networking.hybrid_connectivity.type` | VPN, ExpressRoute, Interconnect | Evidence | Network step | **Keep**, settled from evidence | |
|
||||
| 32 | `networking.hybrid_connectivity.bandwidth_gbps` | Bandwidth | Evidence (partly) | — | **Later** | |
|
||||
| 33 | `networking.egress_control.model` | Direct, provider-managed or a central appliance | Evidence | Network step, Well-Architected security | **Keep**, settled from evidence | |
|
||||
| 34 | `networking.egress_control.appliance_vendor` | Which appliance | Evidence (partly) | — | **Later** | |
|
||||
| 35 | `networking.cidr_allocations` | Allocated IP ranges | Evidence | Network step (address overlap) | **Keep**, settled from evidence | |
|
||||
| 36 | `governance.identity_source_of_truth` | The identity provider | Customer | — | **Later** | |
|
||||
| 37 | `governance.pipeline_authentication` | How pipelines authenticate | Not Discovery | — | **Drop** | |
|
||||
| 38 | `governance.environment_tiers` | Which environments exist | Evidence | Inventory (environment), F02 | **Keep**, settled from evidence (environment tags) | |
|
||||
| 39 | `operations.operating_model` | Who runs it day to day | Customer | Owners (ADR 0021), F02 | **Later**: owners now come from the CMDB and tags | |
|
||||
| 40 | `operations.support_hours` | Supported hours | Customer | F02 risk | **Later** | |
|
||||
| 41 | `operations.monitoring` | Monitoring and alerting | Evidence | Well-Architected operations | **Keep**, settled from evidence | |
|
||||
| 42 | `operations.change_process` | How changes are approved | Customer, or ITSM | F02 change frequency | **Later**, with an ITSM source | |
|
||||
| 43 | `integrations` | Other systems it exchanges data with | Evidence | Dependency graph (external nodes) | **Drop** as a field: the dependency graph holds it | |
|
||||
| 44 | `delivery.target_start_date` | When work starts | Not Discovery | — | **Drop** | |
|
||||
| 45 | `delivery.window_weeks` | The deadline | Not Discovery | — | **Drop** | |
|
||||
| 46 | `delivery.commercial_model` | How the work is contracted | Not Discovery | — | **Drop** | |
|
||||
| 47 | `delivery.iac_tooling` | Infrastructure-as-code tool | Not Discovery | — | **Drop** | |
|
||||
| 48 | `delivery.ci_cd` | Pipeline tool | Not Discovery | — | **Drop** | |
|
||||
| 49 | `delivery.team_size` | Engineers building it | Not Discovery | — | **Drop** | |
|
||||
| 50 | `cost.monthly_budget` | Monthly running budget | Customer | F02 TCO (indicative) | **Later**: F02 uses its rate card | |
|
||||
| 51 | `constraints.products_required`, `products_forbidden` | Mandated or ruled-out products | Not Discovery | — | **Drop** | |
|
||||
|
||||
## Totals, as recommended
|
||||
|
||||
| Recommendation | Fields |
|
||||
|---|---|
|
||||
| Keep | 21 (10 the customer states, 11 settled from evidence) |
|
||||
| Later | 16 |
|
||||
| Drop | 14 (6 already covered by Discovery's own sections or outputs, 8 delivery or design) |
|
||||
|
||||
## Also to decide
|
||||
|
||||
1. **Must every run have an intent,** or is it optional, with the Well-Architected review weighting pillars equally
|
||||
when there's none? Recommended: optional.
|
||||
2. **Where the completed values go:** a completed copy of the context file, or only in the package (a new `context`
|
||||
part). Recommended: the package only, so there's one artifact for F02.
|
||||
3. **ADR 0016's fate:** rewrite it to the fields kept here, within F01 and F02, with no shared catalogue.
|
||||
Reference in New Issue
Block a user