Files
cp1-agents/docs/hyperscaler-interchange-review-note-2026-11-01.md
Jonathan Boniface f659564208 Platform scripts, MCP servers, deployment and docs for the KAB runs
- Scripts to run workflows from a YAML context or run inputs (kab_run_workflow.py, with
  --input name=@file), store per-project discovery connections, bootstrap agent MCP keys,
  refresh the KAB token, publish reports and build estate snapshots.
- Hyperscaler discovery servers extended in place; cloud-knowledge folded into them.
- deploy/crucible-kab-dev.yaml: the Discovery package agent and the Assessment agents.
- Platform issue write-ups and current-wiring notes under docs/platform-issues/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

14 KiB

Using real cloud servers in the Discovery agents: review note

Date 2026-11-01
Status Under review for a real discovery of an Azure and a Google Cloud environment. Nothing here is built, and the plan below isn't approved
Scope ALZ-F01 Discovery, both patterns

Requirements

  • azure-discovery, aws-discovery and gcp-discovery stay standalone. A single server for all clouds is ruled out.
  • One workflow and one set of agents must work with any of the three. No duplicate workflows or per-cloud agents.

Where things stand

  • All three servers are deployed, and each has the same 13 tools once its prefix is removed.
  • Two things stop them being interchangeable: tool names carry a prefix, and the arguments differ. AWS tools take account_id and region, Azure tools don't, and GCP varies. The runtime refuses the same tool name from two servers.
  • Results share one envelope (provider, items, count, next_page_token, collected_at, evidence_ref), but items are the provider's own records, with no normalising. A Resource Graph row and a Cloud Asset record have different fields.

Options for interchange

A: provider workflow input (recommended) B: unprefixed tools behind a logical name
How Agents declare all three servers' tools, and the run says which cloud to use All three drop their prefix, and alz-cloud points at one server
Choose the cloud Per run Per KAB tenant, by re-registering
Agents Need provider rules in their prompts Know nothing about clouds
Main risk 39 tools per agent, and the model could pick the wrong cloud The tenant serves one cloud at a time; duplicate tool names are untested on the gateway

A fits better, because engagements on different clouds run in the same tenant. With a credential stored only for the engagement's cloud, a wrong-cloud call fails visibly. Only agents that read the cloud need these tools.

Credentials without Vault

Kubernetes Secrets can work as a stopgap, but the fallback doesn't exist yet.

  • Each server's pod already loads an optional <server>-secrets Secret, and with no Vault the gateway sends no credential.
  • REQUIRE_VAULT_CONNECTION=false is meant to use the server's environment, but _common/azure_support.py reads only the request's connection, so calls would fail. _common/gcp_support.py has the same gap. The change is to read the environment when no connection arrives. AWS hasn't been checked.
  • One credential then serves every call to that server. That suits a trial on our own subscription, not real customers. When Vault arrives, turn the setting back on and delete the Secret.

Data the cloud servers can't supply

Scope and sites, CMDB and ITSM, database schemas, flows and link measurements, and the critic's call log and guard hits. Each engagement needs a real source for these, or they become declared gaps. Mixing a real cloud with the sample server's fictional CMDB would be misleading, and meridian-baseline can't score a real estate.

Data adapters

A data adapter parses a file type into Neo4j and PostgreSQL. It holds no credentials and reaches no other system, so it doesn't help with Vault or interchange.

Gap Fit
CMDB and ITSM exports Best fit
Flow and firewall exports Partial: a snapshot only
Schema exports Partial: only if the customer supplies them
Scope list Partial: a file would then decide scope
Call log and guard hits None: these record the run itself

Costs: one adapter per format, with fixtures that pass conformance. Product docs list only COBOL and JCL as read today, and only kyn-cobol-ingestion-adapter is deployed. No way for a workflow step to start ingestion was confirmed. Agents would need knowledge-graph access, and its free-form query_graph sits awkwardly with the fixed-tool rule. No server of that name was found on the cluster.

For small estates, the workflow's upload step (u1) already passes file text to the planner and inventory agents, so a CMDB export could use that with no adapter.

This came from reading the platform code and docs. No adapter was tested, and the live catalog wasn't checked.

Recommendation to review

  1. Option A for interchange.
  2. The Secret fallback in each server until Vault arrives.
  3. The upload step for CMDB and ITSM exports first, with adapters only if real exports are too large or varied.

Review of the provider input

Reviewed against runtime 0.35.7 on 2026-11-01. This replaces the recommendation for option A above. The owner has confirmed a metadata-only first discovery, but not this design.

Option A as first described isn't safe enough:

  • Nothing enforces the choice. The runtime offers every declared MCP tool to the model. The before_model hook adds instructions only, so a prompt rule is the only guard.
  • One unhealthy server stops every agent. resolve fails if any declared server isn't offered by the gateway, and the gateway leaves out unhealthy servers. An aws-discovery fault would stop an Azure run.
  • Argument and volume differences stay with the model. Each prompt would carry per-cloud rules, and raw output (5.3 MB on Azure) would reach the model.

Recommended instead: A with neutral function tools. The three servers stay standalone, and the choice moves into the agent's code:

  • Agents declare the servers under mcp:, and neutral tools under functions:, such as list_servers(kind).
  • The function validates provider, calls {provider}_{tool} with call_tool, walks every kind and page, and returns a normalised summary.
  • before_model withholds the raw tools from the model. The Knowledge Graph Analyst already works this way (agent.yaml mcp:, app/graph.py).
  • provider is an enum in each input schema, which the runtime enforces, as the unknown_source failure showed.
Gives Costs
The model can't pick the wrong cloud The individual agents gain app/ code and stop being purely declarative
One normalised shape, with kinds and paging handled Shared module across the agents, tested per cloud
Volume is cut before the model sees it Declared servers must still all be healthy at startup

Open points: confirm before_model can withhold tools under the ADK runtime these agents use, and decide how to handle the startup rule. Declaring only azure-discovery and gcp-discovery at first limits the risk but leaves AWS for later. The sub-agent pattern needs less change, because its orchestrator already makes the MCP calls.

Secret fallback ruled out

Tried on 2026-11-01. The MCP Server Builder rejects any server code that reads a credential from the environment (_credential_env_read in mcp_server_service.py), so a Kubernetes Secret can't feed these servers through rendered code. The check is a deliberate control, and editing server.py by hand is forbidden, so the fallback was removed. Without Vault, the gateway has no supported way to send a connection. The section above on credentials without Vault is out of date.

azure_list_data_stores now covers PostgreSQL, MySQL and Redis (azure-discovery 0.1.3), and gcp-discovery 0.1.2 takes an access_token. Both pass their tests and ran locally against the real environments. Both are deployed to mcp-servers since 2026-11-02, on the public Python base like the others.

Snapshot trial

Built on 2026-11-01 so a real estate can go through the unchanged workflow without the Vault Service. scripts/build_estate_snapshot.py turns locally collected Google Cloud results into a snapshot. alz-discovery-sample 0.2.0 serves it in place of Meridian when ESTATE_SNAPSHOT is set, through the server's optional -dotenv-secrets Secret. Deleting that Secret and restarting brings Meridian back.

The snapshot holds resource metadata and no credential. It's kept out of this repository. While it's loaded, the sample isn't available. CMDB, ITSM, schemas, flows and link measurements are empty, so a run shows those gaps. The agents and the workflow don't change. This doesn't test the live path to the cloud, and the Secret's roughly 1 MB limit rules out the Azure estate as it stands.

Trial result, RUN-GCP-001

Run exec_4d5bd23e38c3 on 2026-11-01: 3 scope entries from one Google Cloud project. All ten steps completed, and the critic's verdict was pass. The Secret was deleted afterwards, and the sample is back.

  • The report overstates what was examined. No source gaps are declared for CMDB, ITSM, schemas or flows, so "No network findings" and "No data stores were classified" read as clean results, and the critic passes.
  • False drift. With no CMDB at all, all three assets are reported as missing from the CMDB, with add_ci.
  • Wrong edge. A VM on a network is typed app_to_app.
  • Bucket not classified. The snapshot marks it as a data store, but it has no schema, so it's left out silently.
  • Slow model call. The CMDB step's first model call took about 10 minutes, and the cause isn't known.

Fixes for all of these are made and checked offline, but not deployed: planner 0.5.0, inventory, CMDB, dependency and critic 0.3.0, classification and report 0.2.0, orchestrator 0.5.0 and sample server 0.2.1. The snapshot now holds 8 entries. The slow call wasn't diagnosed: with no CMDB, that step now replies without calling a tool, and the next run will show whether that helps.

Trial result, RUN-GCP-002

Run exec_f41bcfa4a04c on 2026-11-01, on 8 entries: 8 of 8 checks, verdict pass, in about 90 seconds. The four missing sources were declared, no false drift was reported, and the critic listed them in not_checked. The CMDB step took 1.4 seconds, so the 10-minute call didn't recur. Its cause remains unknown. The Secret was deleted afterwards.

Plan: first real discovery

Plan: first real discovery

The agents' tools are all sample tools today, so a real run changes all eight agents.

Step Agent Real source Limit
s1 Planner list_accounts, list_resources No engagement scope exists, so scope becomes what scope_root shows, and regions stand in for sites
s2 Scan list_servers, list_workloads, list_kubernetes_clusters, list_data_stores, get_resource_metadata Plans by resource type, because per-resource tasks won't scale. No OS or service detail inside a VM
s3 Inventory Normalises s2 output across providers list_apps has no cloud equivalent
s4 CMDB None Declared gap, unless an export is uploaded
s5 Classification list_data_stores Store metadata only. No schemas, so no PII or card data findings
s6 Network get_network_topology No flows or link measurements, so no bottleneck or unknown-source findings
s7 Dependencies Peerings, private endpoints, resource references Configured links only, so confidence is lower
s8 Critic Re-reads scope from the cloud No call log or guard hits. Its tool-call check needs a replacement, or it fails every run

Phases:

  1. Real data, no agents. Run each server locally, with the connection sent in the call. No cluster change, no model, and nothing leaves the laptop. Proves access and shows real output.
  2. Credentials on the cluster. Add the Secret fallback to both servers, and create the Secrets by hand.
  3. Agents. Add the provider input, and change the eight agents and the workflow.
  4. Real case. Write expected facts from phase 1's output, then run each cloud and score it.

The sample rerun comes first, because phase 3 builds on those fixes.

Phase 1 results, 2026-11-01

Owner's decision: use a short-lived personal token for each environment, not a key or service account.

Azure: run locally against the agreed subscription with an az token, through the server's own tools. Sign-in and the scope check passed. It found 644 resources of 51 types in 7 locations, including 16 VMs, 16 scale sets and 15 AKS clusters. No model was involved, and the results stayed on the laptop.

  • Data store gap: azure_list_data_stores covers SQL, Cosmos DB and storage accounts only. This subscription's 13 PostgreSQL flexible servers and 7 Redis caches aren't in any of its kinds.
  • Default kinds: a tool with no kind returns only its first kind, so agents must ask for every kind. list_data_stores returned 0 this way, because its default is sql.
  • Volume: the tools returned about 5.3 MB, of which 3.2 MB is 1,776 Advisor recommendations. The sample run was 27 KB in total, so this is the real test of ADR 0010, and agents can't be handed this much as it stands.

Google Cloud: run locally against the trial project with a gcloud token. Sign-in and the scope search passed, and the tools work, but the project is close to empty: 2 assets (the project and its billing record) and 2 enabled services. Compute isn't enabled. It proves access, not discovery. list_resource_policies returned 403, and list_recommendations wasn't tested, because it needs project_id and location, which the probe didn't pass. The account can see two other projects with 87 and 124 enabled services, which weren't scanned.

At the owner's request, cloudasset, orgpolicy, recommender and cloudresourcemanager were then enabled on that project. Compute wasn't, because the server doesn't call it. The rerun found 3 assets, and list_resource_policies still returned 403.

After the owner added resources, the same check found 20 assets in 4 locations, about 40 KB in total: 1 VM, 1 disk, 1 network, 1 subnetwork, 2 routes, 1 storage bucket and 1 service account, plus service and billing records. This is enough for a small end-to-end test, but far smaller than the Azure estate. list_data_stores showed 0 only because the probe used its default kind, cloud_sql; the bucket appeared in list_resources. Firewall rules returned 0.

Personal tokens on the cluster: an az token lasts about an hour, so a Secret would need refreshing around each run. It carries the person's own rights, not a Reader role, and anyone who can read the Secret can use it. gcp-discovery has no token field yet, so it would need one. Phase 1's local run replaced that server's token step in a throwaway script.

Questions for the review

  • How does each engagement supply scope and sites?
  • Which data gaps are acceptable as declared gaps?
  • Does a run ever need to cover two clouds?
  • Is a metadata-only discovery, with these gaps declared, useful enough for a first real run?
  • Can the gateway's records replace the call log for the critic? Not yet reviewed.