ALZ-F01 Discovery: package step, value scorecard, application grouping, cases

- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent f659564208
commit 9e503885cc
211 changed files with 18170 additions and 158 deletions

View File

@@ -0,0 +1,222 @@
#!/usr/bin/env python3
"""Add the `provider` run input and the live cloud tools to the nine individual Discovery agents (ADR 0013).
`provider` is `sample` (the Meridian estate through the alz-* tools, unchanged) or a live cloud, `azure` or `gcp`,
read through the existing azure-discovery and gcp-discovery servers with the connection stored in KAB's vault.
For each agent this adds the input to schemas/input.json, the live tools it needs to agent.yaml `mcp:`, a
"Live provider" section to prompt.md (before the role), a minor version and a release note. Idempotent: an agent
that already has the section is left alone. Run scripts/build_workflow.py afterwards. Standard library only.
AWS isn't offered yet: its tools take account_id and region, and no AWS connection is stored.
Usage (from the Feature folder): python3 scripts/add_live_provider.py
"""
from __future__ import annotations
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
DATE = "2026-11-03"
MARK = "## Live provider"
PROVIDERS = ("azure", "gcp")
# The MCP gateway offers servers by display name, and agent.yaml `mcp:` must use it.
GATEWAY_NAMES = {"azure": "Azure Discovery", "gcp": "Google Cloud Discovery"}
PROVIDER_INPUT = {"type": "string", "enum": ["sample", *PROVIDERS], "default": "sample",
"description": "Which estate this run reads: `sample` (the Meridian sample estate, through the alz-* "
"tools) or a live cloud, `azure` or `gcp`, through that cloud's discovery server. Set "
"once for the whole workflow run."}
LIST = ("list_servers", "list_workloads", "list_kubernetes_clusters", "list_data_stores")
KINDS = """| Tool | `azure` kinds | `gcp` kinds |
|---|---|---|
| `list_accounts` | `subscriptions`, `resource_groups` | `projects` |
| `list_servers` | `virtual_machines`, `scale_sets` | `instances` |
| `list_workloads` | `app_service`, `container_apps`, `container_instances` | `cloud_run`, `cloud_functions`, `app_engine` |
| `list_kubernetes_clusters` | (takes no kind) | `clusters` |
| `list_data_stores` | `sql`, `cosmos_db`, `storage_accounts`, `postgresql`, `mysql`, `redis` | `cloud_sql`, `firestore`, `cloud_storage` |
| `get_network_topology` | `virtual_networks`, `route_tables`, `network_security_groups`, `gateways`, `load_balancers`, `private_endpoints`, `public_ips` | `networks`, `subnetworks`, `routes`, `firewall_rules`, `load_balancers`, `gateways` |
Call a tool once for **every** kind its row lists, with `page_size` 50, and follow `next_page_token` to the end. A
tool called without `kind` returns its first kind only."""
COMMON = """`provider` is set once for the whole run. `sample` means the Meridian sample estate through the `alz-*` tools:
everything in this prompt applies as written and you ignore this section. `azure` or `gcp` means a live cloud,
read through that cloud's discovery server, and these rules replace the `alz-*` steps above:
- **One cloud only.** Use only the tools whose names start with `{provider}_`. Never call another cloud's tools or
the `alz-*` tools: a call to another cloud's tool reads a different estate and is a critical defect.
- **Identifiers.** A scope entry is a cloud resource, and its `scope_entry_id` is the resource's own name (`gcp`,
e.g. `//compute.googleapis.com/projects/p/zones/z/instances/i`) or id (`azure`, e.g.
`/subscriptions/s/resourceGroups/g/providers/...`) exactly as a tool returned it. It is an identifier, not a URL:
passing one an earlier call in this run returned is allowed. Never pass anything else.
- **Missing sources.** A live run has no CMDB, ITSM, flow or schema source. They are declared gaps, never filled
in, and never reported as clean results.
- **Volume.** Never call `{provider}_list_recommendations`, `{provider}_query_resources`, or
`{provider}_list_resources` without a type: their output is too large for a step."""
AGENTS = {
"cp1-discovery-planner-agent": dict(tools=("list_accounts", *LIST), kinds=True, text="""
**Planning a live run:**
1. Call `{provider}_list_accounts` (`projects` / `subscriptions`), then each of `{provider}_list_servers`,
`{provider}_list_workloads`, `{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every
kind. Leave out disks, scale-set instances and node pools: they belong to the server or cluster they're part of.
2. The items `{provider}_list_accounts` returns (projects, folders, subscriptions, resource groups) are the
estate's structure and confirm the scope root. They are never scope entries: plan no task for one. Every item the
four other list tools return is one scope entry. Its site is its region: a `gcp` zone such as `europe-west2-a` is the
region `europe-west2`; an `azure` location is used as it is.
3. Tasks: one `scan_target` per scope entry (`assigned_agent` `ALZ-DSC-02`, `mcp_tool`
`{provider}-discovery.{provider}_get_resource_metadata`, `target_ref` the scope entry id, batches of 10,
`declared_rps` 1); one `classify_data` per data store (ALZ-DSC-05); one `collect_network` with `target_ref`
`scope` (ALZ-DSC-07); one `trace_dependencies` (ALZ-DSC-06). No ingest or reconcile task.
4. `source_gaps`: `cmdb`, `itsm`, `netflow` and `schema`, each with `reason` `no_connection` and its impact
(no CMDB: owners, cost and criticality are unknown; no ITSM: change and incident history are unknown; no
flows: traffic, bottlenecks and unknown sources can't be shown; no schemas: data can't be classified beyond
store metadata).
5. A resource the list tools return but that can't be scanned (no metadata access) is a coverage gap with
`no_read_only_credential`. Nothing is dropped silently."""),
"cp1-estate-scan-executor-agent": dict(tools=("get_resource_metadata",), kinds=False, text="""
**Scanning a live run:** for every `scan_target` task, call `{provider}_get_resource_metadata` once, with
`resource_name` (`gcp`) or `resource_id` (`azure`) set to the task's `target_ref`, and `scope_entry_id` set to the
same value. Fill the finding from that result only:
- `hostname`: the resource's display name; `os`: the image or OS fields, if the metadata has them, else `""`;
`services`: engine, runtime and version fields, if any.
- `dependencies`: other scope entries this resource references (its network, subnetwork, attached disks' owner,
the database it connects to), by their scope entry id. Never a service account or a person.
- `migration_risks`: only what the metadata states, e.g. a public address, an end-of-life version, no backups.
- `provider_attributes`: the metadata's own values, copied verbatim where present: resource type, region, zone,
machine type or size, state, OS image, disk names, network and subnetwork, public address (yes or no), labels or
tags, deletion protection, Secure Boot, encryption, backup and versioning settings. Leave out what isn't there.
A refused or failed call is a finding with empty fields and the refusal quoted in `migration_risks`."""),
"cp1-asset-inventory-normalisation-agent": dict(tools=(), kinds=False, text="""
**Normalising a live run:** there is no application list and no CMDB, so call no tool. Build one record per
finding:
- `canonical_id` `CI-001`, `CI-002`, … in the findings' order; `scope_entry_id` the finding's.
- `asset_class`: VMs and instances `server`; databases `database`; buckets and storage accounts `storage_array`;
everything else, clusters included, `cloud_resource`.
- `environment`: from a label or tag named `env` or `environment`, if present, else `unknown`.
- `facts`: only labels or tags that name an owner, cost centre, criticality or business service, copied verbatim.
Leave every other fact out; never infer one.
- `provider_attributes`: the finding's, unchanged; `facts.site` its region (a `gcp` zone such as
`europe-west2-b` is the region `europe-west2`).
- `sources`: one, `{ source_type: 'provider_api', source_ref: <scope entry id>, evidence_ref: <the finding's> }`;
`match_basis` `['cloud_resource_id']`; `confidence` 0.85 (one source)."""),
"cp1-cmdb-itsm-reconciliation-agent": dict(tools=(), kinds=False, text="""
**A live run** has `cmdb` and `itsm` in `source_gaps`: call no tool, and return empty `drift_items` and
`itsm_signals` with every summary count 0. Never report an asset as missing from a CMDB that wasn't read."""),
"cp1-data-classification-agent": dict(tools=("list_data_stores",), kinds=True, text="""
**Classifying a live run:** call `{provider}_list_data_stores` for every kind in **Kinds**. For each data store
that is a scope entry in `records`: `store_name` its name; `highest_class` `unclassified`; every class count 0;
`column_tags` empty; `reason` "Metadata only: no schema source in this run". Set `requires_disposition_controls`
true only when a label or tag marks the store as holding personal or card data, and say so in `reason`."""),
"cp1-network-discovery-assessment-agent": dict(tools=("get_network_topology",), kinds=True, text="""
**Assessing a live run's network:** call `{provider}_get_network_topology` for every kind in **Kinds**.
- `ranges`: each subnetwork's CIDR, with `site` its region and `source_ref` its resource name or id; mark overlaps.
- `findings`: `address_overlap` for overlapping CIDRs; `internet_exposure` (severity `high`) for each firewall or
NSG rule allowing `0.0.0.0/0`, `::/0` or `*` inbound to 22, 3389, 21, 1433, 3306 or 5432; `single_point_of_failure`
only where the topology shows one path and nothing else.
- `flows`: empty. `netflow` is a source gap, so never report a bottleneck or an unknown source.
- `topology_summary`: how many items of each kind you read, e.g. `{"networks": 1, "subnetworks": 1, "routes": 2,
"firewall_rules": 0}`, every kind in **Kinds** included, with 0 for none."""),
"cp1-dependency-mapping-agent": dict(tools=(), kinds=False, text="""
**Mapping a live run:** call no tool. For every finding, emit one edge per entry in its `dependencies`:
- `source`: the `canonical_id` of that finding's record.
- `target`: the dependency's record `canonical_id` when it is a scope entry; otherwise the referenced resource's
name or id exactly as the finding gives it. A network, subnetwork or disk the scope doesn't list is still a
dependency: never drop one because its target isn't in scope.
- `type`: `network_attachment` for a network or subnetwork, `hosted_on` for a disk, `unclassified` otherwise;
`evidence` `[{ kind: 'config', ref: <the finding's evidence_ref> }]`; `confidence` 0.6; `validation`
`sme_review_recommended`.
`nodes`: one per record, with `fan_in` and `fan_out` counted from these edges; a record something depends on has
`retire_constraint` `cannot_retire_without_mitigation`. `orphans`: only records with no edge in either direction.
Configured links only: say so, because traffic isn't known."""),
"cp1-discovery-critic-agent": dict(tools=LIST, kinds=True, text="""
**Validating a live run:**
- The authoritative scope is the cloud's own listing. Call `{provider}_list_servers`, `{provider}_list_workloads`,
`{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every kind, and measure coverage
against that, never against the findings.
- `alz-scope` and `alz-evidence` aren't used in a live run: leave `tool_call_issues` empty, and put `call_log` and
`guard_hits` in `not_checked`.
- Every entry in a finding's `dependencies` must appear in `edges`, from that finding's record. A missing one is a
`consistency_issues` entry and fails the run: the scan found a dependency the map dropped.
- A scope entry that is a project, folder, subscription or resource group is a `consistency_issues` entry and fails
the run: those are the estate's structure, not assets.
- A finding, record or edge whose identifier belongs to the other cloud (a `gcp` name begins `//`; an `azure` id
begins `/subscriptions/`) is a `consistency_issues` entry and fails the run."""),
"cp1-discovery-report-agent": dict(tools=(), kinds=False, text="""
**Reporting a live run:** name the provider and the scope root in the summary, and list every source gap as what
the run could not examine. A missing source is never reported as a clean result."""),
}
def bump(version: str) -> str:
major, minor, _ = (int(x) for x in version.split("."))
return f"{major}.{minor + 1}.0"
def add_tools(yaml_text: str, tools) -> str:
if not tools:
return yaml_text
block = "".join(f' "{GATEWAY_NAMES[p]}":\n' + "".join(f' - "{p}_{t}"\n' for t in tools) for p in PROVIDERS)
if re.search(r"^mcp:\n", yaml_text, re.M):
return re.sub(r"^(mcp:\n(?: .*\n| .*\n)+)", lambda m: m.group(1) + block, yaml_text, count=1, flags=re.M)
return re.sub(r"^skills:", "mcp:\n" + block + "skills:", yaml_text, count=1, flags=re.M)
def main() -> int:
for agent, spec in AGENTS.items():
d = ROOT / agent
prompt = (d / "prompt.md").read_text()
if MARK in prompt:
print(f"{agent}: already has the live provider, unchanged")
continue
manifest = (d / "agent.yaml").read_text()
old = re.search(r'^version: "([\d.]+)"', manifest, re.M).group(1)
new = bump(old)
manifest = manifest.replace(f'version: "{old}"', f'version: "{new}"', 1)
(d / "agent.yaml").write_text(add_tools(manifest, spec["tools"]))
schema = json.loads((d / "schemas" / "input.json").read_text())
schema["properties"]["provider"] = PROVIDER_INPUT
(d / "schemas" / "input.json").write_text(json.dumps(schema, indent=2) + "\n")
if agent == "cp1-network-discovery-assessment-agent":
out = json.loads((d / "schemas" / "output.json").read_text())
types = out["properties"]["findings"]["items"]["properties"]["type"]["enum"]
if "internet_exposure" not in types:
types.append("internet_exposure")
(d / "schemas" / "output.json").write_text(json.dumps(out, indent=2) + "\n")
live_tools = "".join(f"| `{p}-discovery` | `{p}_{t}` | read |\n" for p in PROVIDERS for t in spec["tools"])
if live_tools:
prompt = re.sub(r"(## MCP tools\n\n\| Server \| Tool \| Access \|\n\|---\|---\|---\|\n(?:\|.*\n)*)",
lambda m: m.group(1) + live_tools, prompt, count=1)
section = f"{MARK}\n\n{COMMON}\n" + (f"\n**Kinds:**\n\n{KINDS}\n" if spec["kinds"] else "") + spec["text"] + "\n\n"
at = prompt.find("## Role: ")
prompt = prompt[:at] + section + prompt[at:] if at >= 0 else prompt.rstrip() + "\n\n" + section
# The runtime reads {name} in a prompt as a session variable: placeholders are written <name>.
prompt = re.sub(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", r"<\1>", prompt)
(d / "prompt.md").write_text(prompt)
notes = (d / "RELEASE_NOTES.md").read_text()
entry = (f"## [{new}] - {DATE}\n\n### Added\n\n- The `provider` input: `sample` (unchanged) or a live cloud, "
f"`azure` or `gcp`, read through the existing discovery servers (ADR 0013)."
+ (f" Live tools: {', '.join(f'`{{provider}}_{t}`' for t in spec['tools'])}." if spec["tools"] else "")
+ "\n\n")
(d / "RELEASE_NOTES.md").write_text(entry + notes)
print(f"{agent}: {old} -> {new}, {len(spec['tools']) * len(PROVIDERS)} live tools")
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -1,13 +1,14 @@
#!/usr/bin/env python3
"""Write workflows/cp1-alz-discovery-individual.json: the KAB workflow export chaining the eight agents.
"""Write workflows/cp1-alz-discovery-individual.json: the KAB workflow export chaining the eleven agents.
The same export format as cp4-agents' workflow (export_version 2.0.0). There is no orchestrator: each step is one
agent, and each step's inputs are workflow inputs or named fields of earlier steps' outputs (`from_steps`). The
classification and network steps run in parallel; the critic's step stops for the Engagement Lead's approval. Each
step's schemas are read from its agent's schemas/ folder, so run this again after changing them. Standard library
only.
classification and network steps run in parallel; the report's step stops for the Engagement Lead's approval, and
only after it do the two pages (s10, s12) and the Discovery package for the Assessment workflow (s13, ADR 0020) run.
Each step's schemas are read from its agent's schemas/ folder, so run this again after changing them. The build
refuses a step whose input schema would refuse a value the earlier step's output can hold. Standard library only.
Front steps (ADR 0011, the repository's scripts/kab_front_steps.py): the planner (s1) and the inventory (s3) take the text the
Front steps (ADR 0011, the repository's scripts/kab_front_steps.py): the planner (s1), the inventory (s3), the CMDB step (s4) and the report (s9) take the text the
platform extracts from files, since declarative agents can't open files:
u1 Parse Assessment Files: the run form's upload field `file` -> `source_documents` (u1.parsed_content)
a1 Artifacts: project artifacts fixed by id -> `reference_documents` (a1.parsed_content)
@@ -26,56 +27,129 @@ from pathlib import Path
# kab_front_steps is shared by every Feature, in the repository's scripts/ folder.
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
from kab_front_steps import AGENT_STEP_TIMEOUT_MINUTES, ARTIFACTS_ID, UPLOAD_ID, front_steps, shift # noqa: E402
from kab_front_steps import AGENT_STEP_TIMEOUT_MINUTES, ARTIFACTS_ID, UPLOAD_ID, _activity_step, front_steps, shift # noqa: E402
ROOT = Path(__file__).resolve().parents[1]
NAME = "CP1 - Agentic Landing Zone - Discovery (individual agents)"
STAMP = "2026-10-29T12:00:00"
WF = ["engagement_id", "run_id"]
WF = ["engagement_id", "run_id", "provider"]
# (step id, agent folder, parallel group, inputs from earlier steps)
STEPS = [
("s1", "cp1-discovery-planner-agent", 0, {}),
("s2", "cp1-estate-scan-executor-agent", 1, {"tasks": "s1.tasks"}),
("s3", "cp1-asset-inventory-normalisation-agent", 2, {"findings": "s2.findings"}),
("s4", "cp1-cmdb-itsm-reconciliation-agent", 3, {"tasks": "s1.tasks", "records": "s3.records"}),
("s5", "cp1-data-classification-agent", 4, {"records": "s3.records"}),
("s4", "cp1-cmdb-itsm-reconciliation-agent", 3, {"tasks": "s1.tasks", "records": "s3.records",
"source_gaps": "s1.source_gaps"}),
("s5", "cp1-data-classification-agent", 4, {"records": "s3.records", "source_gaps": "s1.source_gaps"}),
("s6", "cp1-network-discovery-assessment-agent", 4, {"tasks": "s1.tasks", "records": "s3.records"}),
("s7", "cp1-dependency-mapping-agent", 5, {"records": "s3.records", "drift_items": "s4.drift_items",
"flows": "s6.flows"}),
"flows": "s6.flows", "findings": "s2.findings",
"cmdb_cis": "s4.cis", "cmdb_relationships": "s4.relationships",
"cmdb_dangling": "s4.dangling_relationships"}),
("s8", "cp1-discovery-critic-agent", 6, {"coverage_gaps": "s1.coverage_gaps", "findings": "s2.findings",
"records": "s3.records", "drift_items": "s4.drift_items",
"stores": "s5.stores", "network_findings": "s6.findings",
"edges": "s7.edges"}),
"edges": "s7.edges", "source_gaps": "s1.source_gaps",
"cmdb_injections": "s4.injections_found",
"nodes": "s7.nodes", "folded_dependencies": "s7.folded_dependencies"}),
("s11", "cp1-well-architected-review-agent", 7, {"records": "s3.records", "source_gaps": "s1.source_gaps", "verdict": "s8.verdict"}),
("s9", "cp1-discovery-report-agent", 8, {"topology_summary": "s6.topology_summary", "waf_status": "s11.status",
"waf_pillars": "s11.pillars", "waf_note": "s11.note","plan_summary": "s1.summary", "coverage_gaps": "s1.coverage_gaps", "source_gaps": "s1.source_gaps",
"scan_findings": "s2.findings", "records": "s3.records",
"sme_review_queue": "s3.sme_review_queue", "drift_items": "s4.drift_items",
"itsm_signals": "s4.itsm_signals", "stores": "s5.stores",
"network_findings": "s6.findings", "ranges": "s6.ranges", "edges": "s7.edges",
"nodes": "s7.nodes", "orphans": "s7.orphans",
"validation_queue": "s7.validation_queue", "verdict": "s8.verdict",
"critic_note": "s8.note", "obeyed_injection": "s8.obeyed_injection",
"unsupported_claims": "s8.unsupported_claims",
"consistency_issues": "s8.consistency_issues",
"parse_failures": "s8.parse_failures",
"tool_call_issues": "s8.tool_call_issues",
"cmdb_sources": "s4.cmdb_sources", "cmdb_injections": "s4.injections_found",
"graph_comparison": "s7.graph_comparison",
"folded_dependencies": "s7.folded_dependencies", "dangling": "s7.dangling",
"applications": "s7.applications", "asset_owners": "s7.asset_owners",
"ungrouped_assets": "s7.ungrouped_assets"}),
# After the report's review point: one package for the Assessment workflow (ADR 0020). Every input is copied
# unchanged; app/package.py INPUTS names the part each one belongs to.
("s13", "cp1-discovery-package-agent", 9, {
"plan_id": "s1.plan_id", "plan_summary": "s1.summary", "coverage_gaps": "s1.coverage_gaps",
"source_gaps": "s1.source_gaps", "scope_accounting": "s1.scope_accounting",
"scan_findings": "s2.findings",
"records": "s3.records", "sme_review_queue": "s3.sme_review_queue",
"drift_items": "s4.drift_items", "itsm_signals": "s4.itsm_signals", "cmdb_summary": "s4.summary",
"cmdb_cis": "s4.cis", "cmdb_relationships": "s4.relationships", "cmdb_dangling": "s4.dangling_relationships",
"cmdb_sources": "s4.cmdb_sources", "cmdb_injections": "s4.injections_found",
"stores": "s5.stores",
"ranges": "s6.ranges", "flows": "s6.flows", "network_findings": "s6.findings",
"topology_summary": "s6.topology_summary",
"nodes": "s7.nodes", "edges": "s7.edges", "orphans": "s7.orphans", "validation_queue": "s7.validation_queue",
"folded_dependencies": "s7.folded_dependencies", "dangling": "s7.dangling",
"graph_comparison": "s7.graph_comparison",
"applications": "s7.applications", "asset_owners": "s7.asset_owners",
"ungrouped_assets": "s7.ungrouped_assets", "applications_note": "s7.applications_note",
"verdict": "s8.verdict", "critic_note": "s8.note", "coverage_complete": "s8.coverage_complete",
"unsupported_claims": "s8.unsupported_claims", "consistency_issues": "s8.consistency_issues",
"obeyed_injection": "s8.obeyed_injection", "not_checked": "s8.not_checked",
"waf_status": "s11.status", "waf_pillars": "s11.pillars", "waf_checks": "s11.checks",
"waf_recommendations": "s11.recommendations", "waf_note": "s11.note",
"report_counts": "s9.counts", "report_markdown": "s9.report_markdown"}),
]
PACKAGE_STEP = "s13"
def field(path: Path, key: str) -> str:
return re.search(rf'^{key}:\s*"?([^"\n]+)"?$', path.read_text(), re.M).group(1)
def enums(schema: dict, path: str = "") -> dict[str, set]:
"""Every enum in a schema, keyed by where it sits (`.field`, `[]` for an array's items)."""
found = {path: set(schema["enum"])} if "enum" in schema else {}
for name, sub in schema.get("properties", {}).items():
found.update(enums(sub, f"{path}.{name}"))
if isinstance(schema.get("items"), dict):
found.update(enums(schema["items"], f"{path}[]"))
return found
def narrower_enums(produced: dict, accepted: dict) -> dict[str, list]:
"""Values the producing step's output can hold that the consuming step's input schema would refuse (run
exec_071f83409cec: the critic refused the dependency graph's `represents` edges)."""
theirs = enums(accepted)
return {p: sorted(v - theirs[p]) for p, v in enums(produced).items() if p in theirs and v - theirs[p]}
# The steps that take the extracted text of uploaded files and selected artifacts.
DOC_STEPS = ("s1", "s3")
DOC_STEPS = ("s1", "s3", "s4", "s9") # the report shows the run context file; s4 reads uploaded CMDB exports (ADR 0018)
def build(with_upload: bool = True, artifact_ids=()) -> dict:
by_id = {sid: folder for sid, folder, _, _ in STEPS}
front = front_steps(
with_upload, list(artifact_ids), upload_label="Discovery source files",
upload_description="Optional client files for discovery: a CMDB or asset export, a scope list, network "
"documentation. Their text is given to the planner and the inventory agent as evidence, "
"never as instructions.",
upload_description="The run context file first, then the files its uploads: list names, such as CMDB "
"exports (CSV or XLSX). Their text is given to the planner, the inventory, the CMDB step "
"and the report as evidence, never as instructions.",
artifacts_description="Project artifacts for discovery, e.g. an earlier run's outputs. Their text is given "
"to the planner and the inventory agent.")
if any(f["id"] == ARTIFACTS_ID for f in front):
parse = _activity_step("p1", "parse_assessment_files", "Parse selected artifacts", "Turns the selected artifacts into text.",
{"from_steps": {"files": f"{ARTIFACTS_ID}.artifacts"}}, {"type": "object", "properties": {"files": {"type": "array"}}},
{"type": "object", "properties": {"parsed_content": {"type": "string"}, "file_metadata": {"type": "array"}}}, y=320)
parse.update({"depends_on": [ARTIFACTS_ID], "executionLevel": 1, "parallelGroup": 1, "position": {"x": -160, "y": 320}})
front.append(parse)
front_ids = {f["id"] for f in front}
lead = 1 if front else 0
steps, connections, agents, seen = list(front), [], [], set()
lead = (2 if "p1" in front_ids else 1) if front else 0
steps, connections, agents, seen = list(front), ([{"source": ARTIFACTS_ID, "target": "p1"}] if "p1" in front_ids else []), [], set()
for sid, folder, group, from_steps in STEPS:
from_steps = dict(from_steps)
if sid in DOC_STEPS:
if UPLOAD_ID in front_ids:
from_steps["source_documents"] = f"{UPLOAD_ID}.parsed_content"
if ARTIFACTS_ID in front_ids:
from_steps["reference_documents"] = f"{ARTIFACTS_ID}.parsed_content"
# select_artifacts' only output is `artifacts` (base64 records); p1 parses them into text.
from_steps["reference_documents"] = "p1.parsed_content"
agent = ROOT / folder
agent_id = folder.replace("-", "_")
display = field(agent / "agent.yaml", "display_name")
@@ -90,21 +164,27 @@ def build(with_upload: bool = True, artifact_ids=()) -> dict:
src_out = json.loads((ROOT / by_id[src_step] / "schemas" / "output.json").read_text())
assert src_field in src_out["properties"], f"{sid}: {source} is not an output of {by_id[src_step]}"
assert target in input_schema["properties"], f"{sid}: {target} is not an input of {folder}"
refused = narrower_enums(src_out["properties"][src_field], input_schema["properties"][target])
assert not refused, f"{sid}: {folder}'s input {target} refuses values {source} can hold: {refused}"
depends = sorted({s.split(".")[0] for s in from_steps.values()})
connections += [{"source": d, "target": sid} for d in depends]
inputs = {"from_workflow": WF}
inputs = {"from_workflow": WF + (["include_cmdb", "include_itsm"] if sid == "s1" else [])}
if from_steps:
inputs["from_steps"] = from_steps
if sid == PACKAGE_STEP:
# Which agent versions produced the package, as built (ADR 0020).
inputs["static"] = {"agent_versions": {f: field(ROOT / f / "agent.yaml", "version")
for _, f, _, _ in STEPS if f != folder}}
hitl = {"enabled": False}
if folder == "cp1-discovery-critic-agent":
if folder == "cp1-discovery-report-agent":
hitl = {"enabled": True, "checkpoint_type": "approval",
"review_focus": "The Engagement Lead reviews the critic's verdict and every reason for a fail "
"review_focus": "The Engagement Lead reads the report, with the critic's verdict and every reason for a fail, "
"before Assessment starts. A fail verdict is never auto-approved.",
"timeout_hours": 24, "allowed_actions": ["continue", "rerun", "abort"], "max_reruns": 2}
steps.append(shift({
"id": sid, "agent": agent_id, "agent_type": agent_id, "name": f"{display} Step", "timeout_minutes": AGENT_STEP_TIMEOUT_MINUTES,
"type": "sequential", "executionLevel": group, "parallelGroup": group, "depends_on": depends,
"position": {"x": 320 * group, "y": 160 if sid == "s6" else 0}, "inputs": inputs, "hitl": hitl,
"position": {"x": 320 * group, "y": {"s6": 160, PACKAGE_STEP: 320}.get(sid, 0)}, "inputs": inputs, "hitl": hitl,
"data": {"agentType": agent_id, "name": f"{display} Step",
"description": field(agent / "agent.yaml", "description"), "category": "agent",
"temporalActivityName": "generic_agent_call",
@@ -116,6 +196,24 @@ def build(with_upload: bool = True, artifact_ids=()) -> dict:
if agent_id not in seen:
seen.add(agent_id)
agents.append({"name": agent_id, "display_name": display, "source_url": url, "category": "agent"})
# After the approval gate: the clean page, as this run's own artifact (ADR 0012).
final = _activity_step("s10", "format_for_ui", "Format for UI", "The approved report, for display as a page.",
{"from_steps": {"input_data": "s9.final_html"}, "static": {"format_type": "html"}},
{"type": "object", "properties": {"input_data": {"type": "string"}, "format_type": {"type": "string"}}},
{"type": "object", "properties": {"data": {"type": "object"}}}, y=0)
final.update({"depends_on": ["s9"], "position": {"x": 320 * 9, "y": 0}})
final["executionLevel"] = final["parallelGroup"] = 9
steps.append(shift(final, lead))
connections.append({"source": "s9", "target": "s10"})
# The Well-Architected review's own page, after the same approval gate (ADR 0014).
waf = _activity_step("s12", "format_for_ui", "Format Well-Architected review for UI", "The approved Well-Architected review, as a page.",
{"from_steps": {"input_data": "s11.final_waf_html"}, "static": {"format_type": "html"}},
{"type": "object", "properties": {"input_data": {"type": "string"}, "format_type": {"type": "string"}}},
{"type": "object", "properties": {"data": {"type": "object"}}}, y=160)
waf.update({"depends_on": ["s9", "s11"], "position": {"x": 320 * 9, "y": 160}})
waf["executionLevel"] = waf["parallelGroup"] = 9
steps.append(shift(waf, lead))
connections += [{"source": "s9", "target": "s12"}, {"source": "s11", "target": "s12"}]
description = f"{NAME}: " + "; then ".join(field(ROOT / f / "agent.yaml", "display_name") for _, f, _, _ in STEPS) + "."
tags = ["cp1", "public-cloud", "agentic-landing-zone"]
return {

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""Fail if docs/platform-issues/agent-tools-and-context/individual-agents-report-2026-11-01.md no longer matches the agents' tools, contracts or the workflow's wiring.
Usage (from this Feature folder): python3 scripts/check_context_doc.py
"""
import glob
import json
import sys
from pathlib import Path
import yaml
ROOT = Path(__file__).resolve().parents[1]
REPORT = ROOT.parents[1] / "docs" / "platform-issues" / "agent-tools-and-context" / "individual-agents-report-2026-11-01.md"
DOC = REPORT.read_text()
def main() -> int:
export = json.loads(next((ROOT / "workflows").glob("*.json")).read_text())
template = export.get("template") or export["workflow"]["template"]
problems, read = [], set()
for step in template["steps"]:
read.update((step.get("inputs") or {}).get("from_steps", {}).values())
for step in template["steps"]:
if step["id"] == "u1":
continue
agent = step["agent"].replace("_", "-")
if not (ROOT / agent).exists():
continue # a built-in activity, not an agent of this pattern
spec = yaml.safe_load((ROOT / agent / "agent.yaml").read_text())
for server, tools in spec.get("mcp", {}).items():
problems += [f"{agent}: {server} {tool}" for tool in tools if f"`{tool}`" not in DOC]
problems += [f"{step['id']}: receives {source}" for source in step["inputs"].get("from_steps", {}).values()
if f"`{source}`" not in DOC]
outputs = json.loads((ROOT / agent / "schemas" / "output.json").read_text()).get("properties", {})
problems += [f"{agent}: returns {field}" for field in outputs if f"`{field}`" not in DOC]
if step["id"] != "s9":
lost = [f for f in outputs if f"{step['id']}.{f}" not in read]
row = f"| {step['id']} | " + ", ".join(f"`{f}`" for f in lost) + " |"
if row not in DOC:
problems.append(f"{step['id']}: the unread fields are now {lost}")
print("\n".join(problems) or "The page matches.")
return 1 if problems else 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -0,0 +1,66 @@
#!/usr/bin/env python3
"""Write one workflow run's model usage, per agent and in total, for a case's `limits:` (ADR 0020).
Reads each agent's own `request_completed` log lines for the execution id, read-only, with kubectl. Nothing on the
cluster is changed. `seconds` is agent time: the sum of each step's request duration, not the run's wall clock, which
includes the review point's wait.
Usage: collect_run_usage.py <execution id> --out usage.json [--context crucible-kab-dev-aks] [--since 6h]
(KUBECONFIG must point at the cluster)
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import build_workflow # noqa: E402
def agent_lines(agent: str, execution_id: str, context: str, since: str) -> list[dict]:
logs = subprocess.run(["kubectl", "--context", context, "-n", "agents", "logs", f"deploy/{agent}", f"--since={since}"],
capture_output=True, text=True, check=False).stdout
found = []
for line in logs.splitlines():
if execution_id not in line or '"request_completed"' not in line:
continue
try:
found.append(json.loads(line))
except ValueError:
continue
return found
def collect(execution_id: str, context: str, since: str) -> dict:
agents = {}
for _, folder, _, _ in build_workflow.STEPS:
lines = agent_lines(folder, execution_id, context, since)
if lines:
agents[folder] = {"requests": len(lines),
"outcomes": sorted({str(l.get("outcome")) for l in lines}),
"tokens": sum(int(l.get("tokens") or 0) for l in lines),
"model_calls": sum(int(l.get("model_calls") or 0) for l in lines),
"seconds": round(sum(int(l.get("duration_ms") or 0) for l in lines) / 1000, 1)}
totals = {k: (round(sum(a[k] for a in agents.values()), 1) if k == "seconds" else sum(a[k] for a in agents.values()))
for k in ("tokens", "model_calls", "seconds")}
return {"execution_id": execution_id, "agents": agents, "totals": totals}
def main(argv=None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("execution_id")
parser.add_argument("--out", type=Path, required=True)
parser.add_argument("--context", default="crucible-kab-dev-aks")
parser.add_argument("--since", default="6h")
args = parser.parse_args(argv)
usage = collect(args.execution_id, args.context, args.since)
args.out.write_text(json.dumps(usage, indent=2) + "\n")
print(f"wrote {args.out}: {len(usage['agents'])} agents, {usage['totals']}")
return 0 if usage["agents"] else 1
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""Judge a Discovery workflow run against a case in workflows/cases/. No model or agent is involved.
Usage: evaluate_workflow_run.py <case.yaml> <steps.json> [--usage usage.json] exits 1 when a critical check fails
Besides the case's checks it prints the run's value scorecard (ADR 0020): the package step's own, or, for a run
without one, the same scorecard computed from the step outputs by the package agent's app/scorecard.py. A case's
`value:` bar and `limits:` (checked against scripts/collect_run_usage.py's file) fail as major.
"""
import json
import re
import sys
from pathlib import Path
import yaml
FEATURE = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(FEATURE / "cp1-discovery-package-agent"))
sys.path.insert(0, str(FEATURE / "scripts"))
from app import package # noqa: E402
import build_workflow # noqa: E402
AGENTS = {"s1": "planner", "s2": "scan", "s3": "inventory", "s4": "cmdb", "s5": "classification", "s6": "network",
"s7": "dependency", "s8": "critic", "s11": "well_architected", "s9": "report", "s13": "package"}
# A case's value bar: the scorecard field it reads, and whether the run's value must be at least or at most the bar.
VALUE = {"f02_filled_min": (("f02_fields", "filled"), "min"),
"f02_filled_or_declared_min": (("f02_fields", "filled_or_declared"), "min"),
"f02_missing_max": (("f02_fields", "missing"), "max"),
"f02_run_met_min": (("f02_run", "met"), "min"),
"findings_with_owner_min": (("findings", "with_owner_share"), "min"),
"findings_with_action_min": (("findings", "with_action_share"), "min")}
def _parsed(value):
while isinstance(value, str):
try:
value = json.loads(value)
except ValueError:
return value
return value
def numbered(steps: dict) -> list:
"""`kab_run_workflow.py outputs` keys each output by its step's position in the workflow (1 is the upload step),
with no agent name: name them from the committed workflow's step order."""
export = json.loads(next((Path(__file__).resolve().parents[1] / "workflows").glob("*.json")).read_text())
order = (export.get("template") or export["workflow"]["template"])["steps"]
return [{"id": order[int(k) - 1]["id"], "agent": order[int(k) - 1]["agent"], "output": v}
for k, v in sorted(steps.items(), key=lambda kv: int(kv[0])) if int(k) <= len(order)]
def outputs_by_step(steps) -> dict:
"""Each step's output, found by the agent's name: import renames the step ids."""
if isinstance(steps, dict) and isinstance(steps.get("steps"), dict) and all(k.isdigit() for k in steps["steps"]):
steps = numbered(steps["steps"])
if isinstance(steps, dict):
steps = steps.get("steps") or steps.get("outputs") or [{"id": k, **(v if isinstance(v, dict) else {"output": v})}
for k, v in steps.items()]
found = {}
for step in steps:
label = json.dumps({k: step.get(k) for k in ("id", "step_id", "agent", "agent_name", "name", "step_name")}).lower()
for short, word in AGENTS.items():
if word in label and short not in found:
output = _parsed(step.get("output", step.get("result", step)))
if isinstance(output, dict):
output = _parsed(output.get("response", output.get("output", output)))
found[short] = output
return found
def _text(value) -> str:
return json.dumps(value, ensure_ascii=False, default=str)
def _matches(value, check: dict) -> tuple[bool, str]:
if "equals" in check:
return value == check["equals"], f"is {_text(value)[:120]}"
if "contains" in check:
return str(check["contains"]) in _text(value), "text not found"
if "contains_all" in check:
missing = [str(item) for item in check["contains_all"] if str(item) not in _text(value)]
return not missing, f"missing {missing}"
if "not_contains" in check:
return str(check["not_contains"]) not in _text(value), "text found"
if "min_items" in check:
return isinstance(value, list) and len(value) >= check["min_items"], "too few items"
rule = check["item"]
items = value if isinstance(value, list) else [value]
hits = [item for item in items if re.search(rule["match"], _text(item))]
if rule.get("absent"):
return not hits, f"{len(hits)} item(s) match"
if not hits:
return False, "no item matches"
rest = {k: v for k, v in rule.items() if k != "match"}
if not rest: # `match` alone: at least one item matches
return True, ""
bad = [item for item in hits if not _matches(item, rest)[0]]
return not bad, f"{len(bad)} of {len(hits)} item(s) wrong"
def _field(output: dict, path: str):
"""A field, or a nested one named with dots (`manifest.complete`)."""
value = output
for key in path.split("."):
value = value.get(key) if isinstance(value, dict) else None
return value
def scorecard_of(found: dict) -> tuple[dict, str]:
"""The package's own scorecard, else the same one computed from the step outputs."""
pack = found.get("s13")
if isinstance(pack, dict) and isinstance((pack.get("manifest") or {}).get("scorecard"), dict):
return pack["manifest"]["scorecard"], "from the package step"
wiring = dict(next(s for s in build_workflow.STEPS if s[0] == build_workflow.PACKAGE_STEP)[3])
payload = {}
for name, source in wiring.items():
step, field = source.split(".")
if isinstance(found.get(step), dict) and field in found[step]:
payload[name] = found[step][field]
return package.scorecard.score(package.parts_from(payload)), "computed from the step outputs (no package step)"
def value_checks(case: dict, card: dict, usage: dict | None) -> list[tuple[str, bool, str]]:
results = []
for name, bar in (case.get("value") or {}).items():
(section, key), kind = VALUE[name]
got = (card.get(section) or {}).get(key)
ok = got is not None and (got >= bar if kind == "min" else got <= bar)
results.append((f"value {name}", ok, f"is {got}, bar {bar}"))
for name, limit in (case.get("limits") or {}).items():
got = (usage or {}).get("totals", {}).get(name)
if usage is None:
results.append((f"limit {name}", True, "not measured (no --usage file)"))
else:
results.append((f"limit {name}", got is not None and got <= limit, f"is {got}, limit {limit}"))
return results
def print_scorecard(card: dict, where: str) -> None:
fields, run, decisions, findings = card["f02_fields"], card["f02_run"], card["decisions"], card["findings"]
print(f"\nValue scorecard ({where}):")
print(f"- F02 fields over {fields['records']} records: {fields['filled']:.0%} filled, {fields['declared_gap']:.0%} "
f"declared gap, {fields['missing']:.0%} missing")
gaps = [f for f, c in fields["per_field"].items() if c["missing"]]
print(f" missing in: {', '.join(gaps) or 'none'}")
print(f"- F02 run checks met: {run['met']} of {run['of']}"
+ "".join(f"; not met: {k}" for k, v in run["checks"].items() if not v))
print(f"- Decisions for a person: {decisions['total']} {decisions['by_kind']}")
print(f"- Findings: {findings['total']}, {findings['with_owner_share']:.0%} with an owner, "
f"{findings['with_action_share']:.0%} with an action")
def main(argv) -> int:
usage_path = None
if "--usage" in argv:
usage_path = argv[argv.index("--usage") + 1]
argv = [a for i, a in enumerate(argv) if a != "--usage" and argv[i - 1] != "--usage"]
case = yaml.safe_load(Path(argv[0]).read_text())
found = outputs_by_step(json.loads(Path(argv[1]).read_text()))
usage = json.loads(Path(usage_path).read_text()) if usage_path else None
print(f"Case {case['id']}: {len(found)} of {len(AGENTS)} steps found\n")
failed = {"critical": 0, "major": 0}
for check in case["checks"]:
output = found.get(check["step"])
if not isinstance(output, dict):
passed, detail = False, "no output for this step"
else:
value = _field(output, check["field"])
passed, detail = (False, "field missing") if value is None else _matches(value, check)
if not passed:
failed[check["severity"]] += 1
print(f"{'pass' if passed else 'FAIL'} {check['severity']:8} {check['step']} {check['id']}" +
("" if passed else f": {detail}. {check['why']}"))
card, where = scorecard_of(found)
print_scorecard(card, where)
extra = value_checks(case, card, usage)
for name, passed, detail in extra:
failed["major"] += not passed
print(f"{'pass' if passed else 'FAIL'} major {name}" + ("" if passed else f": {detail}") +
(f" ({detail})" if passed and "not measured" in detail else ""))
total = len(case["checks"]) + len(extra)
print(f"\n{total - sum(failed.values())} of {total} passed. "
f"Failed: {failed['critical']} critical, {failed['major']} major.\n\nFor a person to review:")
print("\n".join(f"- {item}" for item in case.get("review") or ["Nothing listed for this case."]))
return 1 if failed["critical"] else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))

View File

@@ -0,0 +1,47 @@
#!/usr/bin/env python3
"""Write the Discovery report for a finished run, without running the report step.
Usage: render_discovery_report.py <steps.json> <report.md> [run id] [notice]
Also writes <report>.html beside it. Refuses to overwrite a report.
"""
import importlib.util
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "scripts"))
from evaluate_workflow_run import outputs_by_step # noqa: E402
spec = importlib.util.spec_from_file_location("report", ROOT / "cp1-discovery-report-agent/app/report.py")
report = importlib.util.module_from_spec(spec)
spec.loader.exec_module(report)
def main() -> int:
source, target = Path(sys.argv[1]), Path(sys.argv[2])
page = target.with_suffix(".html")
if target.exists() or page.exists():
raise SystemExit(f"{target} or {page} exists; reports are never overwritten")
notice = sys.argv[4] if len(sys.argv) > 4 else None
s = outputs_by_step(json.loads(source.read_text()))
# The same fields as the report step's from_steps in build_workflow.py.
data = {"engagement_id": s["s1"].get("engagement_id"), "run_id": sys.argv[3] if len(sys.argv) > 3 else None,
"plan_summary": s["s1"].get("summary"), "coverage_gaps": s["s1"].get("coverage_gaps"), "source_gaps": s["s1"].get("source_gaps"),
"scan_findings": s["s2"].get("findings"), "records": s["s3"].get("records"),
"sme_review_queue": s["s3"].get("sme_review_queue"), "drift_items": s["s4"].get("drift_items"),
"itsm_signals": s["s4"].get("itsm_signals"), "stores": s["s5"].get("stores"),
"network_findings": s["s6"].get("findings"), "ranges": s["s6"].get("ranges"), "edges": s["s7"].get("edges"),
"nodes": s["s7"].get("nodes"), "orphans": s["s7"].get("orphans"), "validation_queue": s["s7"].get("validation_queue"),
"verdict": s["s8"].get("verdict"), "critic_note": s["s8"].get("note"), "obeyed_injection": s["s8"].get("obeyed_injection"),
"unsupported_claims": s["s8"].get("unsupported_claims"), "consistency_issues": s["s8"].get("consistency_issues"),
"parse_failures": s["s8"].get("parse_failures"), "tool_call_issues": s["s8"].get("tool_call_issues")}
target.write_text(report.render(data, notice))
page.write_text(report.render_html(data, notice))
print(f"wrote {target} and {page}")
return 0
if __name__ == "__main__":
sys.exit(main())

View File

@@ -90,7 +90,7 @@ NODE = obj({"canonical_id": S, "fan_in": I, "fan_out": I, "depended_on": B,
RANGE = obj({"cidr": S, "site": S, "segment": S, "source_ref": S, "overlap_with": arr(S)}, ["cidr", "site", "source_ref"])
FLOW = obj({"source": S, "destination": S, "port": S, "bytes_total": I, "days_observed": I, "source_ref": S})
NFINDING = obj({"finding_id": S, "type": enum("bottleneck", "single_point_of_failure", "unsupported_device",
"legacy_protocol", "address_overlap", "unowned_rule"),
"legacy_protocol", "address_overlap", "unowned_rule", "unknown_source"),
"subject": S, "measurement": SN, "window": SN, "evidence_refs": arr(S),
"severity": enum("high", "medium", "low")})
ENG = {"type": "string", "description": "The engagement being discovered, e.g. ENG-MERIDIAN-001."}
@@ -230,7 +230,7 @@ AGENTS = [
"from_steps": {"coverage_gaps": "s1.coverage_gaps", "findings": "s2.findings", "records": "s3.records",
"drift_items": "s4.drift_items", "stores": "s5.stores", "network_findings": "s6.findings",
"edges": "s7.edges"}},
model="claude-sonnet-5-5"),
model="gpt-5.6-sol"),
]
# Workflow order: s1 plan, s2 scan, s3 inventory, s4 reconcile, s5 classify + s6 network (parallel), s7 deps, s8 critic
ORDER = ["01", "02", "03", "04", "05", "07", "06", "08"]