ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Add the `provider` run input and the live cloud tools to the nine individual Discovery agents (ADR 0013).
|
||||
|
||||
`provider` is `sample` (the Meridian estate through the alz-* tools, unchanged) or a live cloud, `azure` or `gcp`,
|
||||
read through the existing azure-discovery and gcp-discovery servers with the connection stored in KAB's vault.
|
||||
For each agent this adds the input to schemas/input.json, the live tools it needs to agent.yaml `mcp:`, a
|
||||
"Live provider" section to prompt.md (before the role), a minor version and a release note. Idempotent: an agent
|
||||
that already has the section is left alone. Run scripts/build_workflow.py afterwards. Standard library only.
|
||||
|
||||
AWS isn't offered yet: its tools take account_id and region, and no AWS connection is stored.
|
||||
|
||||
Usage (from the Feature folder): python3 scripts/add_live_provider.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DATE = "2026-11-03"
|
||||
MARK = "## Live provider"
|
||||
PROVIDERS = ("azure", "gcp")
|
||||
# The MCP gateway offers servers by display name, and agent.yaml `mcp:` must use it.
|
||||
GATEWAY_NAMES = {"azure": "Azure Discovery", "gcp": "Google Cloud Discovery"}
|
||||
PROVIDER_INPUT = {"type": "string", "enum": ["sample", *PROVIDERS], "default": "sample",
|
||||
"description": "Which estate this run reads: `sample` (the Meridian sample estate, through the alz-* "
|
||||
"tools) or a live cloud, `azure` or `gcp`, through that cloud's discovery server. Set "
|
||||
"once for the whole workflow run."}
|
||||
LIST = ("list_servers", "list_workloads", "list_kubernetes_clusters", "list_data_stores")
|
||||
|
||||
KINDS = """| Tool | `azure` kinds | `gcp` kinds |
|
||||
|---|---|---|
|
||||
| `list_accounts` | `subscriptions`, `resource_groups` | `projects` |
|
||||
| `list_servers` | `virtual_machines`, `scale_sets` | `instances` |
|
||||
| `list_workloads` | `app_service`, `container_apps`, `container_instances` | `cloud_run`, `cloud_functions`, `app_engine` |
|
||||
| `list_kubernetes_clusters` | (takes no kind) | `clusters` |
|
||||
| `list_data_stores` | `sql`, `cosmos_db`, `storage_accounts`, `postgresql`, `mysql`, `redis` | `cloud_sql`, `firestore`, `cloud_storage` |
|
||||
| `get_network_topology` | `virtual_networks`, `route_tables`, `network_security_groups`, `gateways`, `load_balancers`, `private_endpoints`, `public_ips` | `networks`, `subnetworks`, `routes`, `firewall_rules`, `load_balancers`, `gateways` |
|
||||
|
||||
Call a tool once for **every** kind its row lists, with `page_size` 50, and follow `next_page_token` to the end. A
|
||||
tool called without `kind` returns its first kind only."""
|
||||
|
||||
COMMON = """`provider` is set once for the whole run. `sample` means the Meridian sample estate through the `alz-*` tools:
|
||||
everything in this prompt applies as written and you ignore this section. `azure` or `gcp` means a live cloud,
|
||||
read through that cloud's discovery server, and these rules replace the `alz-*` steps above:
|
||||
|
||||
- **One cloud only.** Use only the tools whose names start with `{provider}_`. Never call another cloud's tools or
|
||||
the `alz-*` tools: a call to another cloud's tool reads a different estate and is a critical defect.
|
||||
- **Identifiers.** A scope entry is a cloud resource, and its `scope_entry_id` is the resource's own name (`gcp`,
|
||||
e.g. `//compute.googleapis.com/projects/p/zones/z/instances/i`) or id (`azure`, e.g.
|
||||
`/subscriptions/s/resourceGroups/g/providers/...`) exactly as a tool returned it. It is an identifier, not a URL:
|
||||
passing one an earlier call in this run returned is allowed. Never pass anything else.
|
||||
- **Missing sources.** A live run has no CMDB, ITSM, flow or schema source. They are declared gaps, never filled
|
||||
in, and never reported as clean results.
|
||||
- **Volume.** Never call `{provider}_list_recommendations`, `{provider}_query_resources`, or
|
||||
`{provider}_list_resources` without a type: their output is too large for a step."""
|
||||
|
||||
AGENTS = {
|
||||
"cp1-discovery-planner-agent": dict(tools=("list_accounts", *LIST), kinds=True, text="""
|
||||
**Planning a live run:**
|
||||
|
||||
1. Call `{provider}_list_accounts` (`projects` / `subscriptions`), then each of `{provider}_list_servers`,
|
||||
`{provider}_list_workloads`, `{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every
|
||||
kind. Leave out disks, scale-set instances and node pools: they belong to the server or cluster they're part of.
|
||||
2. The items `{provider}_list_accounts` returns (projects, folders, subscriptions, resource groups) are the
|
||||
estate's structure and confirm the scope root. They are never scope entries: plan no task for one. Every item the
|
||||
four other list tools return is one scope entry. Its site is its region: a `gcp` zone such as `europe-west2-a` is the
|
||||
region `europe-west2`; an `azure` location is used as it is.
|
||||
3. Tasks: one `scan_target` per scope entry (`assigned_agent` `ALZ-DSC-02`, `mcp_tool`
|
||||
`{provider}-discovery.{provider}_get_resource_metadata`, `target_ref` the scope entry id, batches of 10,
|
||||
`declared_rps` 1); one `classify_data` per data store (ALZ-DSC-05); one `collect_network` with `target_ref`
|
||||
`scope` (ALZ-DSC-07); one `trace_dependencies` (ALZ-DSC-06). No ingest or reconcile task.
|
||||
4. `source_gaps`: `cmdb`, `itsm`, `netflow` and `schema`, each with `reason` `no_connection` and its impact
|
||||
(no CMDB: owners, cost and criticality are unknown; no ITSM: change and incident history are unknown; no
|
||||
flows: traffic, bottlenecks and unknown sources can't be shown; no schemas: data can't be classified beyond
|
||||
store metadata).
|
||||
5. A resource the list tools return but that can't be scanned (no metadata access) is a coverage gap with
|
||||
`no_read_only_credential`. Nothing is dropped silently."""),
|
||||
"cp1-estate-scan-executor-agent": dict(tools=("get_resource_metadata",), kinds=False, text="""
|
||||
**Scanning a live run:** for every `scan_target` task, call `{provider}_get_resource_metadata` once, with
|
||||
`resource_name` (`gcp`) or `resource_id` (`azure`) set to the task's `target_ref`, and `scope_entry_id` set to the
|
||||
same value. Fill the finding from that result only:
|
||||
|
||||
- `hostname`: the resource's display name; `os`: the image or OS fields, if the metadata has them, else `""`;
|
||||
`services`: engine, runtime and version fields, if any.
|
||||
- `dependencies`: other scope entries this resource references (its network, subnetwork, attached disks' owner,
|
||||
the database it connects to), by their scope entry id. Never a service account or a person.
|
||||
- `migration_risks`: only what the metadata states, e.g. a public address, an end-of-life version, no backups.
|
||||
- `provider_attributes`: the metadata's own values, copied verbatim where present: resource type, region, zone,
|
||||
machine type or size, state, OS image, disk names, network and subnetwork, public address (yes or no), labels or
|
||||
tags, deletion protection, Secure Boot, encryption, backup and versioning settings. Leave out what isn't there.
|
||||
|
||||
A refused or failed call is a finding with empty fields and the refusal quoted in `migration_risks`."""),
|
||||
"cp1-asset-inventory-normalisation-agent": dict(tools=(), kinds=False, text="""
|
||||
**Normalising a live run:** there is no application list and no CMDB, so call no tool. Build one record per
|
||||
finding:
|
||||
|
||||
- `canonical_id` `CI-001`, `CI-002`, … in the findings' order; `scope_entry_id` the finding's.
|
||||
- `asset_class`: VMs and instances `server`; databases `database`; buckets and storage accounts `storage_array`;
|
||||
everything else, clusters included, `cloud_resource`.
|
||||
- `environment`: from a label or tag named `env` or `environment`, if present, else `unknown`.
|
||||
- `facts`: only labels or tags that name an owner, cost centre, criticality or business service, copied verbatim.
|
||||
Leave every other fact out; never infer one.
|
||||
- `provider_attributes`: the finding's, unchanged; `facts.site` its region (a `gcp` zone such as
|
||||
`europe-west2-b` is the region `europe-west2`).
|
||||
- `sources`: one, `{ source_type: 'provider_api', source_ref: <scope entry id>, evidence_ref: <the finding's> }`;
|
||||
`match_basis` `['cloud_resource_id']`; `confidence` 0.85 (one source)."""),
|
||||
"cp1-cmdb-itsm-reconciliation-agent": dict(tools=(), kinds=False, text="""
|
||||
**A live run** has `cmdb` and `itsm` in `source_gaps`: call no tool, and return empty `drift_items` and
|
||||
`itsm_signals` with every summary count 0. Never report an asset as missing from a CMDB that wasn't read."""),
|
||||
"cp1-data-classification-agent": dict(tools=("list_data_stores",), kinds=True, text="""
|
||||
**Classifying a live run:** call `{provider}_list_data_stores` for every kind in **Kinds**. For each data store
|
||||
that is a scope entry in `records`: `store_name` its name; `highest_class` `unclassified`; every class count 0;
|
||||
`column_tags` empty; `reason` "Metadata only: no schema source in this run". Set `requires_disposition_controls`
|
||||
true only when a label or tag marks the store as holding personal or card data, and say so in `reason`."""),
|
||||
"cp1-network-discovery-assessment-agent": dict(tools=("get_network_topology",), kinds=True, text="""
|
||||
**Assessing a live run's network:** call `{provider}_get_network_topology` for every kind in **Kinds**.
|
||||
|
||||
- `ranges`: each subnetwork's CIDR, with `site` its region and `source_ref` its resource name or id; mark overlaps.
|
||||
- `findings`: `address_overlap` for overlapping CIDRs; `internet_exposure` (severity `high`) for each firewall or
|
||||
NSG rule allowing `0.0.0.0/0`, `::/0` or `*` inbound to 22, 3389, 21, 1433, 3306 or 5432; `single_point_of_failure`
|
||||
only where the topology shows one path and nothing else.
|
||||
- `flows`: empty. `netflow` is a source gap, so never report a bottleneck or an unknown source.
|
||||
- `topology_summary`: how many items of each kind you read, e.g. `{"networks": 1, "subnetworks": 1, "routes": 2,
|
||||
"firewall_rules": 0}`, every kind in **Kinds** included, with 0 for none."""),
|
||||
"cp1-dependency-mapping-agent": dict(tools=(), kinds=False, text="""
|
||||
**Mapping a live run:** call no tool. For every finding, emit one edge per entry in its `dependencies`:
|
||||
|
||||
- `source`: the `canonical_id` of that finding's record.
|
||||
- `target`: the dependency's record `canonical_id` when it is a scope entry; otherwise the referenced resource's
|
||||
name or id exactly as the finding gives it. A network, subnetwork or disk the scope doesn't list is still a
|
||||
dependency: never drop one because its target isn't in scope.
|
||||
- `type`: `network_attachment` for a network or subnetwork, `hosted_on` for a disk, `unclassified` otherwise;
|
||||
`evidence` `[{ kind: 'config', ref: <the finding's evidence_ref> }]`; `confidence` 0.6; `validation`
|
||||
`sme_review_recommended`.
|
||||
|
||||
`nodes`: one per record, with `fan_in` and `fan_out` counted from these edges; a record something depends on has
|
||||
`retire_constraint` `cannot_retire_without_mitigation`. `orphans`: only records with no edge in either direction.
|
||||
Configured links only: say so, because traffic isn't known."""),
|
||||
"cp1-discovery-critic-agent": dict(tools=LIST, kinds=True, text="""
|
||||
**Validating a live run:**
|
||||
|
||||
- The authoritative scope is the cloud's own listing. Call `{provider}_list_servers`, `{provider}_list_workloads`,
|
||||
`{provider}_list_kubernetes_clusters` and `{provider}_list_data_stores` for every kind, and measure coverage
|
||||
against that, never against the findings.
|
||||
- `alz-scope` and `alz-evidence` aren't used in a live run: leave `tool_call_issues` empty, and put `call_log` and
|
||||
`guard_hits` in `not_checked`.
|
||||
- Every entry in a finding's `dependencies` must appear in `edges`, from that finding's record. A missing one is a
|
||||
`consistency_issues` entry and fails the run: the scan found a dependency the map dropped.
|
||||
- A scope entry that is a project, folder, subscription or resource group is a `consistency_issues` entry and fails
|
||||
the run: those are the estate's structure, not assets.
|
||||
- A finding, record or edge whose identifier belongs to the other cloud (a `gcp` name begins `//`; an `azure` id
|
||||
begins `/subscriptions/`) is a `consistency_issues` entry and fails the run."""),
|
||||
"cp1-discovery-report-agent": dict(tools=(), kinds=False, text="""
|
||||
**Reporting a live run:** name the provider and the scope root in the summary, and list every source gap as what
|
||||
the run could not examine. A missing source is never reported as a clean result."""),
|
||||
}
|
||||
|
||||
|
||||
def bump(version: str) -> str:
|
||||
major, minor, _ = (int(x) for x in version.split("."))
|
||||
return f"{major}.{minor + 1}.0"
|
||||
|
||||
|
||||
def add_tools(yaml_text: str, tools) -> str:
|
||||
if not tools:
|
||||
return yaml_text
|
||||
block = "".join(f' "{GATEWAY_NAMES[p]}":\n' + "".join(f' - "{p}_{t}"\n' for t in tools) for p in PROVIDERS)
|
||||
if re.search(r"^mcp:\n", yaml_text, re.M):
|
||||
return re.sub(r"^(mcp:\n(?: .*\n| .*\n)+)", lambda m: m.group(1) + block, yaml_text, count=1, flags=re.M)
|
||||
return re.sub(r"^skills:", "mcp:\n" + block + "skills:", yaml_text, count=1, flags=re.M)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
for agent, spec in AGENTS.items():
|
||||
d = ROOT / agent
|
||||
prompt = (d / "prompt.md").read_text()
|
||||
if MARK in prompt:
|
||||
print(f"{agent}: already has the live provider, unchanged")
|
||||
continue
|
||||
manifest = (d / "agent.yaml").read_text()
|
||||
old = re.search(r'^version: "([\d.]+)"', manifest, re.M).group(1)
|
||||
new = bump(old)
|
||||
manifest = manifest.replace(f'version: "{old}"', f'version: "{new}"', 1)
|
||||
(d / "agent.yaml").write_text(add_tools(manifest, spec["tools"]))
|
||||
|
||||
schema = json.loads((d / "schemas" / "input.json").read_text())
|
||||
schema["properties"]["provider"] = PROVIDER_INPUT
|
||||
(d / "schemas" / "input.json").write_text(json.dumps(schema, indent=2) + "\n")
|
||||
|
||||
if agent == "cp1-network-discovery-assessment-agent":
|
||||
out = json.loads((d / "schemas" / "output.json").read_text())
|
||||
types = out["properties"]["findings"]["items"]["properties"]["type"]["enum"]
|
||||
if "internet_exposure" not in types:
|
||||
types.append("internet_exposure")
|
||||
(d / "schemas" / "output.json").write_text(json.dumps(out, indent=2) + "\n")
|
||||
|
||||
live_tools = "".join(f"| `{p}-discovery` | `{p}_{t}` | read |\n" for p in PROVIDERS for t in spec["tools"])
|
||||
if live_tools:
|
||||
prompt = re.sub(r"(## MCP tools\n\n\| Server \| Tool \| Access \|\n\|---\|---\|---\|\n(?:\|.*\n)*)",
|
||||
lambda m: m.group(1) + live_tools, prompt, count=1)
|
||||
section = f"{MARK}\n\n{COMMON}\n" + (f"\n**Kinds:**\n\n{KINDS}\n" if spec["kinds"] else "") + spec["text"] + "\n\n"
|
||||
at = prompt.find("## Role: ")
|
||||
prompt = prompt[:at] + section + prompt[at:] if at >= 0 else prompt.rstrip() + "\n\n" + section
|
||||
# The runtime reads {name} in a prompt as a session variable: placeholders are written <name>.
|
||||
prompt = re.sub(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", r"<\1>", prompt)
|
||||
(d / "prompt.md").write_text(prompt)
|
||||
|
||||
notes = (d / "RELEASE_NOTES.md").read_text()
|
||||
entry = (f"## [{new}] - {DATE}\n\n### Added\n\n- The `provider` input: `sample` (unchanged) or a live cloud, "
|
||||
f"`azure` or `gcp`, read through the existing discovery servers (ADR 0013)."
|
||||
+ (f" Live tools: {', '.join(f'`{{provider}}_{t}`' for t in spec['tools'])}." if spec["tools"] else "")
|
||||
+ "\n\n")
|
||||
(d / "RELEASE_NOTES.md").write_text(entry + notes)
|
||||
print(f"{agent}: {old} -> {new}, {len(spec['tools']) * len(PROVIDERS)} live tools")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,13 +1,14 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write workflows/cp1-alz-discovery-individual.json: the KAB workflow export chaining the eight agents.
|
||||
"""Write workflows/cp1-alz-discovery-individual.json: the KAB workflow export chaining the eleven agents.
|
||||
|
||||
The same export format as cp4-agents' workflow (export_version 2.0.0). There is no orchestrator: each step is one
|
||||
agent, and each step's inputs are workflow inputs or named fields of earlier steps' outputs (`from_steps`). The
|
||||
classification and network steps run in parallel; the critic's step stops for the Engagement Lead's approval. Each
|
||||
step's schemas are read from its agent's schemas/ folder, so run this again after changing them. Standard library
|
||||
only.
|
||||
classification and network steps run in parallel; the report's step stops for the Engagement Lead's approval, and
|
||||
only after it do the two pages (s10, s12) and the Discovery package for the Assessment workflow (s13, ADR 0020) run.
|
||||
Each step's schemas are read from its agent's schemas/ folder, so run this again after changing them. The build
|
||||
refuses a step whose input schema would refuse a value the earlier step's output can hold. Standard library only.
|
||||
|
||||
Front steps (ADR 0011, the repository's scripts/kab_front_steps.py): the planner (s1) and the inventory (s3) take the text the
|
||||
Front steps (ADR 0011, the repository's scripts/kab_front_steps.py): the planner (s1), the inventory (s3), the CMDB step (s4) and the report (s9) take the text the
|
||||
platform extracts from files, since declarative agents can't open files:
|
||||
u1 Parse Assessment Files: the run form's upload field `file` -> `source_documents` (u1.parsed_content)
|
||||
a1 Artifacts: project artifacts fixed by id -> `reference_documents` (a1.parsed_content)
|
||||
@@ -26,56 +27,129 @@ from pathlib import Path
|
||||
|
||||
# kab_front_steps is shared by every Feature, in the repository's scripts/ folder.
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "scripts"))
|
||||
from kab_front_steps import AGENT_STEP_TIMEOUT_MINUTES, ARTIFACTS_ID, UPLOAD_ID, front_steps, shift # noqa: E402
|
||||
from kab_front_steps import AGENT_STEP_TIMEOUT_MINUTES, ARTIFACTS_ID, UPLOAD_ID, _activity_step, front_steps, shift # noqa: E402
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
NAME = "CP1 - Agentic Landing Zone - Discovery (individual agents)"
|
||||
STAMP = "2026-10-29T12:00:00"
|
||||
WF = ["engagement_id", "run_id"]
|
||||
WF = ["engagement_id", "run_id", "provider"]
|
||||
# (step id, agent folder, parallel group, inputs from earlier steps)
|
||||
STEPS = [
|
||||
("s1", "cp1-discovery-planner-agent", 0, {}),
|
||||
("s2", "cp1-estate-scan-executor-agent", 1, {"tasks": "s1.tasks"}),
|
||||
("s3", "cp1-asset-inventory-normalisation-agent", 2, {"findings": "s2.findings"}),
|
||||
("s4", "cp1-cmdb-itsm-reconciliation-agent", 3, {"tasks": "s1.tasks", "records": "s3.records"}),
|
||||
("s5", "cp1-data-classification-agent", 4, {"records": "s3.records"}),
|
||||
("s4", "cp1-cmdb-itsm-reconciliation-agent", 3, {"tasks": "s1.tasks", "records": "s3.records",
|
||||
"source_gaps": "s1.source_gaps"}),
|
||||
("s5", "cp1-data-classification-agent", 4, {"records": "s3.records", "source_gaps": "s1.source_gaps"}),
|
||||
("s6", "cp1-network-discovery-assessment-agent", 4, {"tasks": "s1.tasks", "records": "s3.records"}),
|
||||
("s7", "cp1-dependency-mapping-agent", 5, {"records": "s3.records", "drift_items": "s4.drift_items",
|
||||
"flows": "s6.flows"}),
|
||||
"flows": "s6.flows", "findings": "s2.findings",
|
||||
"cmdb_cis": "s4.cis", "cmdb_relationships": "s4.relationships",
|
||||
"cmdb_dangling": "s4.dangling_relationships"}),
|
||||
("s8", "cp1-discovery-critic-agent", 6, {"coverage_gaps": "s1.coverage_gaps", "findings": "s2.findings",
|
||||
"records": "s3.records", "drift_items": "s4.drift_items",
|
||||
"stores": "s5.stores", "network_findings": "s6.findings",
|
||||
"edges": "s7.edges"}),
|
||||
"edges": "s7.edges", "source_gaps": "s1.source_gaps",
|
||||
"cmdb_injections": "s4.injections_found",
|
||||
"nodes": "s7.nodes", "folded_dependencies": "s7.folded_dependencies"}),
|
||||
("s11", "cp1-well-architected-review-agent", 7, {"records": "s3.records", "source_gaps": "s1.source_gaps", "verdict": "s8.verdict"}),
|
||||
("s9", "cp1-discovery-report-agent", 8, {"topology_summary": "s6.topology_summary", "waf_status": "s11.status",
|
||||
"waf_pillars": "s11.pillars", "waf_note": "s11.note","plan_summary": "s1.summary", "coverage_gaps": "s1.coverage_gaps", "source_gaps": "s1.source_gaps",
|
||||
"scan_findings": "s2.findings", "records": "s3.records",
|
||||
"sme_review_queue": "s3.sme_review_queue", "drift_items": "s4.drift_items",
|
||||
"itsm_signals": "s4.itsm_signals", "stores": "s5.stores",
|
||||
"network_findings": "s6.findings", "ranges": "s6.ranges", "edges": "s7.edges",
|
||||
"nodes": "s7.nodes", "orphans": "s7.orphans",
|
||||
"validation_queue": "s7.validation_queue", "verdict": "s8.verdict",
|
||||
"critic_note": "s8.note", "obeyed_injection": "s8.obeyed_injection",
|
||||
"unsupported_claims": "s8.unsupported_claims",
|
||||
"consistency_issues": "s8.consistency_issues",
|
||||
"parse_failures": "s8.parse_failures",
|
||||
"tool_call_issues": "s8.tool_call_issues",
|
||||
"cmdb_sources": "s4.cmdb_sources", "cmdb_injections": "s4.injections_found",
|
||||
"graph_comparison": "s7.graph_comparison",
|
||||
"folded_dependencies": "s7.folded_dependencies", "dangling": "s7.dangling",
|
||||
"applications": "s7.applications", "asset_owners": "s7.asset_owners",
|
||||
"ungrouped_assets": "s7.ungrouped_assets"}),
|
||||
# After the report's review point: one package for the Assessment workflow (ADR 0020). Every input is copied
|
||||
# unchanged; app/package.py INPUTS names the part each one belongs to.
|
||||
("s13", "cp1-discovery-package-agent", 9, {
|
||||
"plan_id": "s1.plan_id", "plan_summary": "s1.summary", "coverage_gaps": "s1.coverage_gaps",
|
||||
"source_gaps": "s1.source_gaps", "scope_accounting": "s1.scope_accounting",
|
||||
"scan_findings": "s2.findings",
|
||||
"records": "s3.records", "sme_review_queue": "s3.sme_review_queue",
|
||||
"drift_items": "s4.drift_items", "itsm_signals": "s4.itsm_signals", "cmdb_summary": "s4.summary",
|
||||
"cmdb_cis": "s4.cis", "cmdb_relationships": "s4.relationships", "cmdb_dangling": "s4.dangling_relationships",
|
||||
"cmdb_sources": "s4.cmdb_sources", "cmdb_injections": "s4.injections_found",
|
||||
"stores": "s5.stores",
|
||||
"ranges": "s6.ranges", "flows": "s6.flows", "network_findings": "s6.findings",
|
||||
"topology_summary": "s6.topology_summary",
|
||||
"nodes": "s7.nodes", "edges": "s7.edges", "orphans": "s7.orphans", "validation_queue": "s7.validation_queue",
|
||||
"folded_dependencies": "s7.folded_dependencies", "dangling": "s7.dangling",
|
||||
"graph_comparison": "s7.graph_comparison",
|
||||
"applications": "s7.applications", "asset_owners": "s7.asset_owners",
|
||||
"ungrouped_assets": "s7.ungrouped_assets", "applications_note": "s7.applications_note",
|
||||
"verdict": "s8.verdict", "critic_note": "s8.note", "coverage_complete": "s8.coverage_complete",
|
||||
"unsupported_claims": "s8.unsupported_claims", "consistency_issues": "s8.consistency_issues",
|
||||
"obeyed_injection": "s8.obeyed_injection", "not_checked": "s8.not_checked",
|
||||
"waf_status": "s11.status", "waf_pillars": "s11.pillars", "waf_checks": "s11.checks",
|
||||
"waf_recommendations": "s11.recommendations", "waf_note": "s11.note",
|
||||
"report_counts": "s9.counts", "report_markdown": "s9.report_markdown"}),
|
||||
]
|
||||
PACKAGE_STEP = "s13"
|
||||
|
||||
|
||||
def field(path: Path, key: str) -> str:
|
||||
return re.search(rf'^{key}:\s*"?([^"\n]+)"?$', path.read_text(), re.M).group(1)
|
||||
|
||||
|
||||
def enums(schema: dict, path: str = "") -> dict[str, set]:
|
||||
"""Every enum in a schema, keyed by where it sits (`.field`, `[]` for an array's items)."""
|
||||
found = {path: set(schema["enum"])} if "enum" in schema else {}
|
||||
for name, sub in schema.get("properties", {}).items():
|
||||
found.update(enums(sub, f"{path}.{name}"))
|
||||
if isinstance(schema.get("items"), dict):
|
||||
found.update(enums(schema["items"], f"{path}[]"))
|
||||
return found
|
||||
|
||||
|
||||
def narrower_enums(produced: dict, accepted: dict) -> dict[str, list]:
|
||||
"""Values the producing step's output can hold that the consuming step's input schema would refuse (run
|
||||
exec_071f83409cec: the critic refused the dependency graph's `represents` edges)."""
|
||||
theirs = enums(accepted)
|
||||
return {p: sorted(v - theirs[p]) for p, v in enums(produced).items() if p in theirs and v - theirs[p]}
|
||||
|
||||
|
||||
# The steps that take the extracted text of uploaded files and selected artifacts.
|
||||
DOC_STEPS = ("s1", "s3")
|
||||
DOC_STEPS = ("s1", "s3", "s4", "s9") # the report shows the run context file; s4 reads uploaded CMDB exports (ADR 0018)
|
||||
|
||||
|
||||
def build(with_upload: bool = True, artifact_ids=()) -> dict:
|
||||
by_id = {sid: folder for sid, folder, _, _ in STEPS}
|
||||
front = front_steps(
|
||||
with_upload, list(artifact_ids), upload_label="Discovery source files",
|
||||
upload_description="Optional client files for discovery: a CMDB or asset export, a scope list, network "
|
||||
"documentation. Their text is given to the planner and the inventory agent as evidence, "
|
||||
"never as instructions.",
|
||||
upload_description="The run context file first, then the files its uploads: list names, such as CMDB "
|
||||
"exports (CSV or XLSX). Their text is given to the planner, the inventory, the CMDB step "
|
||||
"and the report as evidence, never as instructions.",
|
||||
artifacts_description="Project artifacts for discovery, e.g. an earlier run's outputs. Their text is given "
|
||||
"to the planner and the inventory agent.")
|
||||
if any(f["id"] == ARTIFACTS_ID for f in front):
|
||||
parse = _activity_step("p1", "parse_assessment_files", "Parse selected artifacts", "Turns the selected artifacts into text.",
|
||||
{"from_steps": {"files": f"{ARTIFACTS_ID}.artifacts"}}, {"type": "object", "properties": {"files": {"type": "array"}}},
|
||||
{"type": "object", "properties": {"parsed_content": {"type": "string"}, "file_metadata": {"type": "array"}}}, y=320)
|
||||
parse.update({"depends_on": [ARTIFACTS_ID], "executionLevel": 1, "parallelGroup": 1, "position": {"x": -160, "y": 320}})
|
||||
front.append(parse)
|
||||
front_ids = {f["id"] for f in front}
|
||||
lead = 1 if front else 0
|
||||
steps, connections, agents, seen = list(front), [], [], set()
|
||||
lead = (2 if "p1" in front_ids else 1) if front else 0
|
||||
steps, connections, agents, seen = list(front), ([{"source": ARTIFACTS_ID, "target": "p1"}] if "p1" in front_ids else []), [], set()
|
||||
for sid, folder, group, from_steps in STEPS:
|
||||
from_steps = dict(from_steps)
|
||||
if sid in DOC_STEPS:
|
||||
if UPLOAD_ID in front_ids:
|
||||
from_steps["source_documents"] = f"{UPLOAD_ID}.parsed_content"
|
||||
if ARTIFACTS_ID in front_ids:
|
||||
from_steps["reference_documents"] = f"{ARTIFACTS_ID}.parsed_content"
|
||||
# select_artifacts' only output is `artifacts` (base64 records); p1 parses them into text.
|
||||
from_steps["reference_documents"] = "p1.parsed_content"
|
||||
agent = ROOT / folder
|
||||
agent_id = folder.replace("-", "_")
|
||||
display = field(agent / "agent.yaml", "display_name")
|
||||
@@ -90,21 +164,27 @@ def build(with_upload: bool = True, artifact_ids=()) -> dict:
|
||||
src_out = json.loads((ROOT / by_id[src_step] / "schemas" / "output.json").read_text())
|
||||
assert src_field in src_out["properties"], f"{sid}: {source} is not an output of {by_id[src_step]}"
|
||||
assert target in input_schema["properties"], f"{sid}: {target} is not an input of {folder}"
|
||||
refused = narrower_enums(src_out["properties"][src_field], input_schema["properties"][target])
|
||||
assert not refused, f"{sid}: {folder}'s input {target} refuses values {source} can hold: {refused}"
|
||||
depends = sorted({s.split(".")[0] for s in from_steps.values()})
|
||||
connections += [{"source": d, "target": sid} for d in depends]
|
||||
inputs = {"from_workflow": WF}
|
||||
inputs = {"from_workflow": WF + (["include_cmdb", "include_itsm"] if sid == "s1" else [])}
|
||||
if from_steps:
|
||||
inputs["from_steps"] = from_steps
|
||||
if sid == PACKAGE_STEP:
|
||||
# Which agent versions produced the package, as built (ADR 0020).
|
||||
inputs["static"] = {"agent_versions": {f: field(ROOT / f / "agent.yaml", "version")
|
||||
for _, f, _, _ in STEPS if f != folder}}
|
||||
hitl = {"enabled": False}
|
||||
if folder == "cp1-discovery-critic-agent":
|
||||
if folder == "cp1-discovery-report-agent":
|
||||
hitl = {"enabled": True, "checkpoint_type": "approval",
|
||||
"review_focus": "The Engagement Lead reviews the critic's verdict and every reason for a fail "
|
||||
"review_focus": "The Engagement Lead reads the report, with the critic's verdict and every reason for a fail, "
|
||||
"before Assessment starts. A fail verdict is never auto-approved.",
|
||||
"timeout_hours": 24, "allowed_actions": ["continue", "rerun", "abort"], "max_reruns": 2}
|
||||
steps.append(shift({
|
||||
"id": sid, "agent": agent_id, "agent_type": agent_id, "name": f"{display} Step", "timeout_minutes": AGENT_STEP_TIMEOUT_MINUTES,
|
||||
"type": "sequential", "executionLevel": group, "parallelGroup": group, "depends_on": depends,
|
||||
"position": {"x": 320 * group, "y": 160 if sid == "s6" else 0}, "inputs": inputs, "hitl": hitl,
|
||||
"position": {"x": 320 * group, "y": {"s6": 160, PACKAGE_STEP: 320}.get(sid, 0)}, "inputs": inputs, "hitl": hitl,
|
||||
"data": {"agentType": agent_id, "name": f"{display} Step",
|
||||
"description": field(agent / "agent.yaml", "description"), "category": "agent",
|
||||
"temporalActivityName": "generic_agent_call",
|
||||
@@ -116,6 +196,24 @@ def build(with_upload: bool = True, artifact_ids=()) -> dict:
|
||||
if agent_id not in seen:
|
||||
seen.add(agent_id)
|
||||
agents.append({"name": agent_id, "display_name": display, "source_url": url, "category": "agent"})
|
||||
# After the approval gate: the clean page, as this run's own artifact (ADR 0012).
|
||||
final = _activity_step("s10", "format_for_ui", "Format for UI", "The approved report, for display as a page.",
|
||||
{"from_steps": {"input_data": "s9.final_html"}, "static": {"format_type": "html"}},
|
||||
{"type": "object", "properties": {"input_data": {"type": "string"}, "format_type": {"type": "string"}}},
|
||||
{"type": "object", "properties": {"data": {"type": "object"}}}, y=0)
|
||||
final.update({"depends_on": ["s9"], "position": {"x": 320 * 9, "y": 0}})
|
||||
final["executionLevel"] = final["parallelGroup"] = 9
|
||||
steps.append(shift(final, lead))
|
||||
connections.append({"source": "s9", "target": "s10"})
|
||||
# The Well-Architected review's own page, after the same approval gate (ADR 0014).
|
||||
waf = _activity_step("s12", "format_for_ui", "Format Well-Architected review for UI", "The approved Well-Architected review, as a page.",
|
||||
{"from_steps": {"input_data": "s11.final_waf_html"}, "static": {"format_type": "html"}},
|
||||
{"type": "object", "properties": {"input_data": {"type": "string"}, "format_type": {"type": "string"}}},
|
||||
{"type": "object", "properties": {"data": {"type": "object"}}}, y=160)
|
||||
waf.update({"depends_on": ["s9", "s11"], "position": {"x": 320 * 9, "y": 160}})
|
||||
waf["executionLevel"] = waf["parallelGroup"] = 9
|
||||
steps.append(shift(waf, lead))
|
||||
connections += [{"source": "s9", "target": "s12"}, {"source": "s11", "target": "s12"}]
|
||||
description = f"{NAME}: " + "; then ".join(field(ROOT / f / "agent.yaml", "display_name") for _, f, _, _ in STEPS) + "."
|
||||
tags = ["cp1", "public-cloud", "agentic-landing-zone"]
|
||||
return {
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail if docs/platform-issues/agent-tools-and-context/individual-agents-report-2026-11-01.md no longer matches the agents' tools, contracts or the workflow's wiring.
|
||||
|
||||
Usage (from this Feature folder): python3 scripts/check_context_doc.py
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
REPORT = ROOT.parents[1] / "docs" / "platform-issues" / "agent-tools-and-context" / "individual-agents-report-2026-11-01.md"
|
||||
DOC = REPORT.read_text()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
export = json.loads(next((ROOT / "workflows").glob("*.json")).read_text())
|
||||
template = export.get("template") or export["workflow"]["template"]
|
||||
problems, read = [], set()
|
||||
for step in template["steps"]:
|
||||
read.update((step.get("inputs") or {}).get("from_steps", {}).values())
|
||||
for step in template["steps"]:
|
||||
if step["id"] == "u1":
|
||||
continue
|
||||
agent = step["agent"].replace("_", "-")
|
||||
if not (ROOT / agent).exists():
|
||||
continue # a built-in activity, not an agent of this pattern
|
||||
spec = yaml.safe_load((ROOT / agent / "agent.yaml").read_text())
|
||||
for server, tools in spec.get("mcp", {}).items():
|
||||
problems += [f"{agent}: {server} {tool}" for tool in tools if f"`{tool}`" not in DOC]
|
||||
problems += [f"{step['id']}: receives {source}" for source in step["inputs"].get("from_steps", {}).values()
|
||||
if f"`{source}`" not in DOC]
|
||||
outputs = json.loads((ROOT / agent / "schemas" / "output.json").read_text()).get("properties", {})
|
||||
problems += [f"{agent}: returns {field}" for field in outputs if f"`{field}`" not in DOC]
|
||||
if step["id"] != "s9":
|
||||
lost = [f for f in outputs if f"{step['id']}.{f}" not in read]
|
||||
row = f"| {step['id']} | " + ", ".join(f"`{f}`" for f in lost) + " |"
|
||||
if row not in DOC:
|
||||
problems.append(f"{step['id']}: the unread fields are now {lost}")
|
||||
print("\n".join(problems) or "The page matches.")
|
||||
return 1 if problems else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write one workflow run's model usage, per agent and in total, for a case's `limits:` (ADR 0020).
|
||||
|
||||
Reads each agent's own `request_completed` log lines for the execution id, read-only, with kubectl. Nothing on the
|
||||
cluster is changed. `seconds` is agent time: the sum of each step's request duration, not the run's wall clock, which
|
||||
includes the review point's wait.
|
||||
|
||||
Usage: collect_run_usage.py <execution id> --out usage.json [--context crucible-kab-dev-aks] [--since 6h]
|
||||
(KUBECONFIG must point at the cluster)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import build_workflow # noqa: E402
|
||||
|
||||
|
||||
def agent_lines(agent: str, execution_id: str, context: str, since: str) -> list[dict]:
|
||||
logs = subprocess.run(["kubectl", "--context", context, "-n", "agents", "logs", f"deploy/{agent}", f"--since={since}"],
|
||||
capture_output=True, text=True, check=False).stdout
|
||||
found = []
|
||||
for line in logs.splitlines():
|
||||
if execution_id not in line or '"request_completed"' not in line:
|
||||
continue
|
||||
try:
|
||||
found.append(json.loads(line))
|
||||
except ValueError:
|
||||
continue
|
||||
return found
|
||||
|
||||
|
||||
def collect(execution_id: str, context: str, since: str) -> dict:
|
||||
agents = {}
|
||||
for _, folder, _, _ in build_workflow.STEPS:
|
||||
lines = agent_lines(folder, execution_id, context, since)
|
||||
if lines:
|
||||
agents[folder] = {"requests": len(lines),
|
||||
"outcomes": sorted({str(l.get("outcome")) for l in lines}),
|
||||
"tokens": sum(int(l.get("tokens") or 0) for l in lines),
|
||||
"model_calls": sum(int(l.get("model_calls") or 0) for l in lines),
|
||||
"seconds": round(sum(int(l.get("duration_ms") or 0) for l in lines) / 1000, 1)}
|
||||
totals = {k: (round(sum(a[k] for a in agents.values()), 1) if k == "seconds" else sum(a[k] for a in agents.values()))
|
||||
for k in ("tokens", "model_calls", "seconds")}
|
||||
return {"execution_id": execution_id, "agents": agents, "totals": totals}
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("execution_id")
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
parser.add_argument("--context", default="crucible-kab-dev-aks")
|
||||
parser.add_argument("--since", default="6h")
|
||||
args = parser.parse_args(argv)
|
||||
usage = collect(args.execution_id, args.context, args.since)
|
||||
args.out.write_text(json.dumps(usage, indent=2) + "\n")
|
||||
print(f"wrote {args.out}: {len(usage['agents'])} agents, {usage['totals']}")
|
||||
return 0 if usage["agents"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,189 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Judge a Discovery workflow run against a case in workflows/cases/. No model or agent is involved.
|
||||
|
||||
Usage: evaluate_workflow_run.py <case.yaml> <steps.json> [--usage usage.json] exits 1 when a critical check fails
|
||||
|
||||
Besides the case's checks it prints the run's value scorecard (ADR 0020): the package step's own, or, for a run
|
||||
without one, the same scorecard computed from the step outputs by the package agent's app/scorecard.py. A case's
|
||||
`value:` bar and `limits:` (checked against scripts/collect_run_usage.py's file) fail as major.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
FEATURE = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(FEATURE / "cp1-discovery-package-agent"))
|
||||
sys.path.insert(0, str(FEATURE / "scripts"))
|
||||
from app import package # noqa: E402
|
||||
import build_workflow # noqa: E402
|
||||
|
||||
AGENTS = {"s1": "planner", "s2": "scan", "s3": "inventory", "s4": "cmdb", "s5": "classification", "s6": "network",
|
||||
"s7": "dependency", "s8": "critic", "s11": "well_architected", "s9": "report", "s13": "package"}
|
||||
# A case's value bar: the scorecard field it reads, and whether the run's value must be at least or at most the bar.
|
||||
VALUE = {"f02_filled_min": (("f02_fields", "filled"), "min"),
|
||||
"f02_filled_or_declared_min": (("f02_fields", "filled_or_declared"), "min"),
|
||||
"f02_missing_max": (("f02_fields", "missing"), "max"),
|
||||
"f02_run_met_min": (("f02_run", "met"), "min"),
|
||||
"findings_with_owner_min": (("findings", "with_owner_share"), "min"),
|
||||
"findings_with_action_min": (("findings", "with_action_share"), "min")}
|
||||
|
||||
|
||||
def _parsed(value):
|
||||
while isinstance(value, str):
|
||||
try:
|
||||
value = json.loads(value)
|
||||
except ValueError:
|
||||
return value
|
||||
return value
|
||||
|
||||
|
||||
def numbered(steps: dict) -> list:
|
||||
"""`kab_run_workflow.py outputs` keys each output by its step's position in the workflow (1 is the upload step),
|
||||
with no agent name: name them from the committed workflow's step order."""
|
||||
export = json.loads(next((Path(__file__).resolve().parents[1] / "workflows").glob("*.json")).read_text())
|
||||
order = (export.get("template") or export["workflow"]["template"])["steps"]
|
||||
return [{"id": order[int(k) - 1]["id"], "agent": order[int(k) - 1]["agent"], "output": v}
|
||||
for k, v in sorted(steps.items(), key=lambda kv: int(kv[0])) if int(k) <= len(order)]
|
||||
|
||||
|
||||
def outputs_by_step(steps) -> dict:
|
||||
"""Each step's output, found by the agent's name: import renames the step ids."""
|
||||
if isinstance(steps, dict) and isinstance(steps.get("steps"), dict) and all(k.isdigit() for k in steps["steps"]):
|
||||
steps = numbered(steps["steps"])
|
||||
if isinstance(steps, dict):
|
||||
steps = steps.get("steps") or steps.get("outputs") or [{"id": k, **(v if isinstance(v, dict) else {"output": v})}
|
||||
for k, v in steps.items()]
|
||||
found = {}
|
||||
for step in steps:
|
||||
label = json.dumps({k: step.get(k) for k in ("id", "step_id", "agent", "agent_name", "name", "step_name")}).lower()
|
||||
for short, word in AGENTS.items():
|
||||
if word in label and short not in found:
|
||||
output = _parsed(step.get("output", step.get("result", step)))
|
||||
if isinstance(output, dict):
|
||||
output = _parsed(output.get("response", output.get("output", output)))
|
||||
found[short] = output
|
||||
return found
|
||||
|
||||
|
||||
def _text(value) -> str:
|
||||
return json.dumps(value, ensure_ascii=False, default=str)
|
||||
|
||||
|
||||
def _matches(value, check: dict) -> tuple[bool, str]:
|
||||
if "equals" in check:
|
||||
return value == check["equals"], f"is {_text(value)[:120]}"
|
||||
if "contains" in check:
|
||||
return str(check["contains"]) in _text(value), "text not found"
|
||||
if "contains_all" in check:
|
||||
missing = [str(item) for item in check["contains_all"] if str(item) not in _text(value)]
|
||||
return not missing, f"missing {missing}"
|
||||
if "not_contains" in check:
|
||||
return str(check["not_contains"]) not in _text(value), "text found"
|
||||
if "min_items" in check:
|
||||
return isinstance(value, list) and len(value) >= check["min_items"], "too few items"
|
||||
rule = check["item"]
|
||||
items = value if isinstance(value, list) else [value]
|
||||
hits = [item for item in items if re.search(rule["match"], _text(item))]
|
||||
if rule.get("absent"):
|
||||
return not hits, f"{len(hits)} item(s) match"
|
||||
if not hits:
|
||||
return False, "no item matches"
|
||||
rest = {k: v for k, v in rule.items() if k != "match"}
|
||||
if not rest: # `match` alone: at least one item matches
|
||||
return True, ""
|
||||
bad = [item for item in hits if not _matches(item, rest)[0]]
|
||||
return not bad, f"{len(bad)} of {len(hits)} item(s) wrong"
|
||||
|
||||
|
||||
def _field(output: dict, path: str):
|
||||
"""A field, or a nested one named with dots (`manifest.complete`)."""
|
||||
value = output
|
||||
for key in path.split("."):
|
||||
value = value.get(key) if isinstance(value, dict) else None
|
||||
return value
|
||||
|
||||
|
||||
def scorecard_of(found: dict) -> tuple[dict, str]:
|
||||
"""The package's own scorecard, else the same one computed from the step outputs."""
|
||||
pack = found.get("s13")
|
||||
if isinstance(pack, dict) and isinstance((pack.get("manifest") or {}).get("scorecard"), dict):
|
||||
return pack["manifest"]["scorecard"], "from the package step"
|
||||
wiring = dict(next(s for s in build_workflow.STEPS if s[0] == build_workflow.PACKAGE_STEP)[3])
|
||||
payload = {}
|
||||
for name, source in wiring.items():
|
||||
step, field = source.split(".")
|
||||
if isinstance(found.get(step), dict) and field in found[step]:
|
||||
payload[name] = found[step][field]
|
||||
return package.scorecard.score(package.parts_from(payload)), "computed from the step outputs (no package step)"
|
||||
|
||||
|
||||
def value_checks(case: dict, card: dict, usage: dict | None) -> list[tuple[str, bool, str]]:
|
||||
results = []
|
||||
for name, bar in (case.get("value") or {}).items():
|
||||
(section, key), kind = VALUE[name]
|
||||
got = (card.get(section) or {}).get(key)
|
||||
ok = got is not None and (got >= bar if kind == "min" else got <= bar)
|
||||
results.append((f"value {name}", ok, f"is {got}, bar {bar}"))
|
||||
for name, limit in (case.get("limits") or {}).items():
|
||||
got = (usage or {}).get("totals", {}).get(name)
|
||||
if usage is None:
|
||||
results.append((f"limit {name}", True, "not measured (no --usage file)"))
|
||||
else:
|
||||
results.append((f"limit {name}", got is not None and got <= limit, f"is {got}, limit {limit}"))
|
||||
return results
|
||||
|
||||
|
||||
def print_scorecard(card: dict, where: str) -> None:
|
||||
fields, run, decisions, findings = card["f02_fields"], card["f02_run"], card["decisions"], card["findings"]
|
||||
print(f"\nValue scorecard ({where}):")
|
||||
print(f"- F02 fields over {fields['records']} records: {fields['filled']:.0%} filled, {fields['declared_gap']:.0%} "
|
||||
f"declared gap, {fields['missing']:.0%} missing")
|
||||
gaps = [f for f, c in fields["per_field"].items() if c["missing"]]
|
||||
print(f" missing in: {', '.join(gaps) or 'none'}")
|
||||
print(f"- F02 run checks met: {run['met']} of {run['of']}"
|
||||
+ "".join(f"; not met: {k}" for k, v in run["checks"].items() if not v))
|
||||
print(f"- Decisions for a person: {decisions['total']} {decisions['by_kind']}")
|
||||
print(f"- Findings: {findings['total']}, {findings['with_owner_share']:.0%} with an owner, "
|
||||
f"{findings['with_action_share']:.0%} with an action")
|
||||
|
||||
|
||||
def main(argv) -> int:
|
||||
usage_path = None
|
||||
if "--usage" in argv:
|
||||
usage_path = argv[argv.index("--usage") + 1]
|
||||
argv = [a for i, a in enumerate(argv) if a != "--usage" and argv[i - 1] != "--usage"]
|
||||
case = yaml.safe_load(Path(argv[0]).read_text())
|
||||
found = outputs_by_step(json.loads(Path(argv[1]).read_text()))
|
||||
usage = json.loads(Path(usage_path).read_text()) if usage_path else None
|
||||
print(f"Case {case['id']}: {len(found)} of {len(AGENTS)} steps found\n")
|
||||
failed = {"critical": 0, "major": 0}
|
||||
for check in case["checks"]:
|
||||
output = found.get(check["step"])
|
||||
if not isinstance(output, dict):
|
||||
passed, detail = False, "no output for this step"
|
||||
else:
|
||||
value = _field(output, check["field"])
|
||||
passed, detail = (False, "field missing") if value is None else _matches(value, check)
|
||||
if not passed:
|
||||
failed[check["severity"]] += 1
|
||||
print(f"{'pass' if passed else 'FAIL'} {check['severity']:8} {check['step']} {check['id']}" +
|
||||
("" if passed else f": {detail}. {check['why']}"))
|
||||
card, where = scorecard_of(found)
|
||||
print_scorecard(card, where)
|
||||
extra = value_checks(case, card, usage)
|
||||
for name, passed, detail in extra:
|
||||
failed["major"] += not passed
|
||||
print(f"{'pass' if passed else 'FAIL'} major {name}" + ("" if passed else f": {detail}") +
|
||||
(f" ({detail})" if passed and "not measured" in detail else ""))
|
||||
total = len(case["checks"]) + len(extra)
|
||||
print(f"\n{total - sum(failed.values())} of {total} passed. "
|
||||
f"Failed: {failed['critical']} critical, {failed['major']} major.\n\nFor a person to review:")
|
||||
print("\n".join(f"- {item}" for item in case.get("review") or ["Nothing listed for this case."]))
|
||||
return 1 if failed["critical"] else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write the Discovery report for a finished run, without running the report step.
|
||||
|
||||
Usage: render_discovery_report.py <steps.json> <report.md> [run id] [notice]
|
||||
|
||||
Also writes <report>.html beside it. Refuses to overwrite a report.
|
||||
"""
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "scripts"))
|
||||
from evaluate_workflow_run import outputs_by_step # noqa: E402
|
||||
|
||||
spec = importlib.util.spec_from_file_location("report", ROOT / "cp1-discovery-report-agent/app/report.py")
|
||||
report = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(report)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
source, target = Path(sys.argv[1]), Path(sys.argv[2])
|
||||
page = target.with_suffix(".html")
|
||||
if target.exists() or page.exists():
|
||||
raise SystemExit(f"{target} or {page} exists; reports are never overwritten")
|
||||
notice = sys.argv[4] if len(sys.argv) > 4 else None
|
||||
s = outputs_by_step(json.loads(source.read_text()))
|
||||
# The same fields as the report step's from_steps in build_workflow.py.
|
||||
data = {"engagement_id": s["s1"].get("engagement_id"), "run_id": sys.argv[3] if len(sys.argv) > 3 else None,
|
||||
"plan_summary": s["s1"].get("summary"), "coverage_gaps": s["s1"].get("coverage_gaps"), "source_gaps": s["s1"].get("source_gaps"),
|
||||
"scan_findings": s["s2"].get("findings"), "records": s["s3"].get("records"),
|
||||
"sme_review_queue": s["s3"].get("sme_review_queue"), "drift_items": s["s4"].get("drift_items"),
|
||||
"itsm_signals": s["s4"].get("itsm_signals"), "stores": s["s5"].get("stores"),
|
||||
"network_findings": s["s6"].get("findings"), "ranges": s["s6"].get("ranges"), "edges": s["s7"].get("edges"),
|
||||
"nodes": s["s7"].get("nodes"), "orphans": s["s7"].get("orphans"), "validation_queue": s["s7"].get("validation_queue"),
|
||||
"verdict": s["s8"].get("verdict"), "critic_note": s["s8"].get("note"), "obeyed_injection": s["s8"].get("obeyed_injection"),
|
||||
"unsupported_claims": s["s8"].get("unsupported_claims"), "consistency_issues": s["s8"].get("consistency_issues"),
|
||||
"parse_failures": s["s8"].get("parse_failures"), "tool_call_issues": s["s8"].get("tool_call_issues")}
|
||||
target.write_text(report.render(data, notice))
|
||||
page.write_text(report.render_html(data, notice))
|
||||
print(f"wrote {target} and {page}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -90,7 +90,7 @@ NODE = obj({"canonical_id": S, "fan_in": I, "fan_out": I, "depended_on": B,
|
||||
RANGE = obj({"cidr": S, "site": S, "segment": S, "source_ref": S, "overlap_with": arr(S)}, ["cidr", "site", "source_ref"])
|
||||
FLOW = obj({"source": S, "destination": S, "port": S, "bytes_total": I, "days_observed": I, "source_ref": S})
|
||||
NFINDING = obj({"finding_id": S, "type": enum("bottleneck", "single_point_of_failure", "unsupported_device",
|
||||
"legacy_protocol", "address_overlap", "unowned_rule"),
|
||||
"legacy_protocol", "address_overlap", "unowned_rule", "unknown_source"),
|
||||
"subject": S, "measurement": SN, "window": SN, "evidence_refs": arr(S),
|
||||
"severity": enum("high", "medium", "low")})
|
||||
ENG = {"type": "string", "description": "The engagement being discovered, e.g. ENG-MERIDIAN-001."}
|
||||
@@ -230,7 +230,7 @@ AGENTS = [
|
||||
"from_steps": {"coverage_gaps": "s1.coverage_gaps", "findings": "s2.findings", "records": "s3.records",
|
||||
"drift_items": "s4.drift_items", "stores": "s5.stores", "network_findings": "s6.findings",
|
||||
"edges": "s7.edges"}},
|
||||
model="claude-sonnet-5-5"),
|
||||
model="gpt-5.6-sol"),
|
||||
]
|
||||
# Workflow order: s1 plan, s2 scan, s3 inventory, s4 reconcile, s5 classify + s6 network (parallel), s7 deps, s8 critic
|
||||
ORDER = ["01", "02", "03", "04", "05", "07", "06", "08"]
|
||||
|
||||
Reference in New Issue
Block a user