Files
cp1-agents/individual-agents/ALZ-F01: Discovery Workflow Agent/scripts/add_live_provider.py
Jonathan Boniface 9e503885cc ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

223 lines
15 KiB
Python

#!/usr/bin/env python3
"""Add the `provider` run input and the live cloud tools to the nine individual Discovery agents (ADR 0013).
`provider` is `sample` (the Meridian estate through the alz-* tools, unchanged) or a live cloud, `azure` or `gcp`,
read through the existing azure-discovery and gcp-discovery servers with the connection stored in KAB's vault.
For each agent this adds the input to schemas/input.json, the live tools it needs to agent.yaml `mcp:`, a
"Live provider" section to prompt.md (before the role), a minor version and a release note. Idempotent: an agent
that already has the section is left alone. Run scripts/build_workflow.py afterwards. Standard library only.
AWS isn't offered yet: its tools take account_id and region, and no AWS connection is stored.
Usage (from the Feature folder): python3 scripts/add_live_provider.py
"""
from __future__ import annotations
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
DATE = "2026-11-03"
MARK = "## Live provider"
PROVIDERS = ("azure", "gcp")
# The MCP gateway offers servers by display name, and agent.yaml `mcp:` must use it.
GATEWAY_NAMES = {"azure": "Azure Discovery", "gcp": "Google Cloud Discovery"}
PROVIDER_INPUT = {"type": "string", "enum": ["sample", *PROVIDERS], "default": "sample",
"description": "Which estate this run reads: `sample` (the Meridian sample estate, through the alz-* "
"tools) or a live cloud, `azure` or `gcp`, through that cloud's discovery server. Set "
"once for the whole workflow run."}
LIST = ("list_servers", "list_workloads", "list_kubernetes_clusters", "list_data_stores")
KINDS = """| Tool | `azure` kinds | `gcp` kinds |
|---|---|---|
| `list_accounts` | `subscriptions`, `resource_groups` | `projects` |
| `list_servers` | `virtual_machines`, `scale_sets` | `instances` |
| `list_workloads` | `app_service`, `container_apps`, `container_instances` | `cloud_run`, `cloud_functions`, `app_engine` |
| `list_kubernetes_clusters` | (takes no kind) | `clusters` |
| `list_data_stores` | `sql`, `cosmos_db`, `storage_accounts`, `postgresql`, `mysql`, `redis` | `cloud_sql`, `firestore`, `cloud_storage` |
| `get_network_topology` | `virtual_networks`, `route_tables`, `network_security_groups`, `gateways`, `load_balancers`, `private_endpoints`, `public_ips` | `networks`, `subnetworks`, `routes`, `firewall_rules`, `load_balancers`, `gateways` |
Call a tool once for **every** kind its row lists, with `page_size` 50, and follow `next_page_token` to the end. A
tool called without `kind` returns its first kind only."""
COMMON = """`provider` is set once for the whole run. `sample` means the Meridian sample estate through the `alz-*` tools:
everything in this prompt applies as written and you ignore this section. `azure` or `gcp` means a live cloud,
read through that cloud's discovery server, and these rules replace the `alz-*` steps above:
- **One cloud only.** Use only the tools whose names start with `{provider}_`. Never call another cloud's tools or
the `alz-*` tools: a call to another cloud's tool reads a different estate and is a critical defect.
- **Identifiers.** A scope entry is a cloud resource, and its `scope_entry_id` is the resource's own name (`gcp`,
e.g. `//compute.googleapis.com/projects/p/zones/z/instances/i`) or id (`azure`, e.g.
`/subscriptions/s/resourceGroups/g/providers/...`) exactly as a tool returned it. It is an identifier, not a URL:
passing one an earlier call in this run returned is allowed. Never pass anything else.
- **Missing sources.** A live run has no CMDB, ITSM, flow or schema source. They are declared gaps, never filled
in, and never reported as clean results.
- **Volume.** Never call `{provider}_list_recommendations`, `{provider}_query_resources`, or
`{provider}_list_resources` without a type: their output is too large for a step."""
AGENTS = {
"cp1-discovery-planner-agent": dict(tools=("list_accounts", *LIST), kinds=True, text="""
**Planning a live run:**
1. Call `{provider}_list_accounts` (`projects` / `subscriptions`), then each of `{provider}_list_servers`,
`{provider}_list_workloads`, `{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every
kind. Leave out disks, scale-set instances and node pools: they belong to the server or cluster they're part of.
2. The items `{provider}_list_accounts` returns (projects, folders, subscriptions, resource groups) are the
estate's structure and confirm the scope root. They are never scope entries: plan no task for one. Every item the
four other list tools return is one scope entry. Its site is its region: a `gcp` zone such as `europe-west2-a` is the
region `europe-west2`; an `azure` location is used as it is.
3. Tasks: one `scan_target` per scope entry (`assigned_agent` `ALZ-DSC-02`, `mcp_tool`
`{provider}-discovery.{provider}_get_resource_metadata`, `target_ref` the scope entry id, batches of 10,
`declared_rps` 1); one `classify_data` per data store (ALZ-DSC-05); one `collect_network` with `target_ref`
`scope` (ALZ-DSC-07); one `trace_dependencies` (ALZ-DSC-06). No ingest or reconcile task.
4. `source_gaps`: `cmdb`, `itsm`, `netflow` and `schema`, each with `reason` `no_connection` and its impact
(no CMDB: owners, cost and criticality are unknown; no ITSM: change and incident history are unknown; no
flows: traffic, bottlenecks and unknown sources can't be shown; no schemas: data can't be classified beyond
store metadata).
5. A resource the list tools return but that can't be scanned (no metadata access) is a coverage gap with
`no_read_only_credential`. Nothing is dropped silently."""),
"cp1-estate-scan-executor-agent": dict(tools=("get_resource_metadata",), kinds=False, text="""
**Scanning a live run:** for every `scan_target` task, call `{provider}_get_resource_metadata` once, with
`resource_name` (`gcp`) or `resource_id` (`azure`) set to the task's `target_ref`, and `scope_entry_id` set to the
same value. Fill the finding from that result only:
- `hostname`: the resource's display name; `os`: the image or OS fields, if the metadata has them, else `""`;
`services`: engine, runtime and version fields, if any.
- `dependencies`: other scope entries this resource references (its network, subnetwork, attached disks' owner,
the database it connects to), by their scope entry id. Never a service account or a person.
- `migration_risks`: only what the metadata states, e.g. a public address, an end-of-life version, no backups.
- `provider_attributes`: the metadata's own values, copied verbatim where present: resource type, region, zone,
machine type or size, state, OS image, disk names, network and subnetwork, public address (yes or no), labels or
tags, deletion protection, Secure Boot, encryption, backup and versioning settings. Leave out what isn't there.
A refused or failed call is a finding with empty fields and the refusal quoted in `migration_risks`."""),
"cp1-asset-inventory-normalisation-agent": dict(tools=(), kinds=False, text="""
**Normalising a live run:** there is no application list and no CMDB, so call no tool. Build one record per
finding:
- `canonical_id` `CI-001`, `CI-002`, … in the findings' order; `scope_entry_id` the finding's.
- `asset_class`: VMs and instances `server`; databases `database`; buckets and storage accounts `storage_array`;
everything else, clusters included, `cloud_resource`.
- `environment`: from a label or tag named `env` or `environment`, if present, else `unknown`.
- `facts`: only labels or tags that name an owner, cost centre, criticality or business service, copied verbatim.
Leave every other fact out; never infer one.
- `provider_attributes`: the finding's, unchanged; `facts.site` its region (a `gcp` zone such as
`europe-west2-b` is the region `europe-west2`).
- `sources`: one, `{ source_type: 'provider_api', source_ref: <scope entry id>, evidence_ref: <the finding's> }`;
`match_basis` `['cloud_resource_id']`; `confidence` 0.85 (one source)."""),
"cp1-cmdb-itsm-reconciliation-agent": dict(tools=(), kinds=False, text="""
**A live run** has `cmdb` and `itsm` in `source_gaps`: call no tool, and return empty `drift_items` and
`itsm_signals` with every summary count 0. Never report an asset as missing from a CMDB that wasn't read."""),
"cp1-data-classification-agent": dict(tools=("list_data_stores",), kinds=True, text="""
**Classifying a live run:** call `{provider}_list_data_stores` for every kind in **Kinds**. For each data store
that is a scope entry in `records`: `store_name` its name; `highest_class` `unclassified`; every class count 0;
`column_tags` empty; `reason` "Metadata only: no schema source in this run". Set `requires_disposition_controls`
true only when a label or tag marks the store as holding personal or card data, and say so in `reason`."""),
"cp1-network-discovery-assessment-agent": dict(tools=("get_network_topology",), kinds=True, text="""
**Assessing a live run's network:** call `{provider}_get_network_topology` for every kind in **Kinds**.
- `ranges`: each subnetwork's CIDR, with `site` its region and `source_ref` its resource name or id; mark overlaps.
- `findings`: `address_overlap` for overlapping CIDRs; `internet_exposure` (severity `high`) for each firewall or
NSG rule allowing `0.0.0.0/0`, `::/0` or `*` inbound to 22, 3389, 21, 1433, 3306 or 5432; `single_point_of_failure`
only where the topology shows one path and nothing else.
- `flows`: empty. `netflow` is a source gap, so never report a bottleneck or an unknown source.
- `topology_summary`: how many items of each kind you read, e.g. `{"networks": 1, "subnetworks": 1, "routes": 2,
"firewall_rules": 0}`, every kind in **Kinds** included, with 0 for none."""),
"cp1-dependency-mapping-agent": dict(tools=(), kinds=False, text="""
**Mapping a live run:** call no tool. For every finding, emit one edge per entry in its `dependencies`:
- `source`: the `canonical_id` of that finding's record.
- `target`: the dependency's record `canonical_id` when it is a scope entry; otherwise the referenced resource's
name or id exactly as the finding gives it. A network, subnetwork or disk the scope doesn't list is still a
dependency: never drop one because its target isn't in scope.
- `type`: `network_attachment` for a network or subnetwork, `hosted_on` for a disk, `unclassified` otherwise;
`evidence` `[{ kind: 'config', ref: <the finding's evidence_ref> }]`; `confidence` 0.6; `validation`
`sme_review_recommended`.
`nodes`: one per record, with `fan_in` and `fan_out` counted from these edges; a record something depends on has
`retire_constraint` `cannot_retire_without_mitigation`. `orphans`: only records with no edge in either direction.
Configured links only: say so, because traffic isn't known."""),
"cp1-discovery-critic-agent": dict(tools=LIST, kinds=True, text="""
**Validating a live run:**
- The authoritative scope is the cloud's own listing. Call `{provider}_list_servers`, `{provider}_list_workloads`,
`{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every kind, and measure coverage
against that, never against the findings.
- `alz-scope` and `alz-evidence` aren't used in a live run: leave `tool_call_issues` empty, and put `call_log` and
`guard_hits` in `not_checked`.
- Every entry in a finding's `dependencies` must appear in `edges`, from that finding's record. A missing one is a
`consistency_issues` entry and fails the run: the scan found a dependency the map dropped.
- A scope entry that is a project, folder, subscription or resource group is a `consistency_issues` entry and fails
the run: those are the estate's structure, not assets.
- A finding, record or edge whose identifier belongs to the other cloud (a `gcp` name begins `//`; an `azure` id
begins `/subscriptions/`) is a `consistency_issues` entry and fails the run."""),
"cp1-discovery-report-agent": dict(tools=(), kinds=False, text="""
**Reporting a live run:** name the provider and the scope root in the summary, and list every source gap as what
the run could not examine. A missing source is never reported as a clean result."""),
}
def bump(version: str) -> str:
major, minor, _ = (int(x) for x in version.split("."))
return f"{major}.{minor + 1}.0"
def add_tools(yaml_text: str, tools) -> str:
if not tools:
return yaml_text
block = "".join(f' "{GATEWAY_NAMES[p]}":\n' + "".join(f' - "{p}_{t}"\n' for t in tools) for p in PROVIDERS)
if re.search(r"^mcp:\n", yaml_text, re.M):
return re.sub(r"^(mcp:\n(?: .*\n| .*\n)+)", lambda m: m.group(1) + block, yaml_text, count=1, flags=re.M)
return re.sub(r"^skills:", "mcp:\n" + block + "skills:", yaml_text, count=1, flags=re.M)
def main() -> int:
for agent, spec in AGENTS.items():
d = ROOT / agent
prompt = (d / "prompt.md").read_text()
if MARK in prompt:
print(f"{agent}: already has the live provider, unchanged")
continue
manifest = (d / "agent.yaml").read_text()
old = re.search(r'^version: "([\d.]+)"', manifest, re.M).group(1)
new = bump(old)
manifest = manifest.replace(f'version: "{old}"', f'version: "{new}"', 1)
(d / "agent.yaml").write_text(add_tools(manifest, spec["tools"]))
schema = json.loads((d / "schemas" / "input.json").read_text())
schema["properties"]["provider"] = PROVIDER_INPUT
(d / "schemas" / "input.json").write_text(json.dumps(schema, indent=2) + "\n")
if agent == "cp1-network-discovery-assessment-agent":
out = json.loads((d / "schemas" / "output.json").read_text())
types = out["properties"]["findings"]["items"]["properties"]["type"]["enum"]
if "internet_exposure" not in types:
types.append("internet_exposure")
(d / "schemas" / "output.json").write_text(json.dumps(out, indent=2) + "\n")
live_tools = "".join(f"| `{p}-discovery` | `{p}_{t}` | read |\n" for p in PROVIDERS for t in spec["tools"])
if live_tools:
prompt = re.sub(r"(## MCP tools\n\n\| Server \| Tool \| Access \|\n\|---\|---\|---\|\n(?:\|.*\n)*)",
lambda m: m.group(1) + live_tools, prompt, count=1)
section = f"{MARK}\n\n{COMMON}\n" + (f"\n**Kinds:**\n\n{KINDS}\n" if spec["kinds"] else "") + spec["text"] + "\n\n"
at = prompt.find("## Role: ")
prompt = prompt[:at] + section + prompt[at:] if at >= 0 else prompt.rstrip() + "\n\n" + section
# The runtime reads {name} in a prompt as a session variable: placeholders are written <name>.
prompt = re.sub(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", r"<\1>", prompt)
(d / "prompt.md").write_text(prompt)
notes = (d / "RELEASE_NOTES.md").read_text()
entry = (f"## [{new}] - {DATE}\n\n### Added\n\n- The `provider` input: `sample` (unchanged) or a live cloud, "
f"`azure` or `gcp`, read through the existing discovery servers (ADR 0013)."
+ (f" Live tools: {', '.join(f'`{{provider}}_{t}`' for t in spec['tools'])}." if spec["tools"] else "")
+ "\n\n")
(d / "RELEASE_NOTES.md").write_text(entry + notes)
print(f"{agent}: {old} -> {new}, {len(spec['tools']) * len(PROVIDERS)} live tools")
return 0
if __name__ == "__main__":
raise SystemExit(main())