# Using real cloud servers in the Discovery agents: review note | | | |---|---| | Date | 2026-11-01 | | Status | Under review for a real discovery of an Azure and a Google Cloud environment. Nothing here is built, and the plan below isn't approved | | Scope | ALZ-F01 Discovery, both patterns | ## Requirements - `azure-discovery`, `aws-discovery` and `gcp-discovery` stay standalone. A single server for all clouds is ruled out. - One workflow and one set of agents must work with any of the three. No duplicate workflows or per-cloud agents. ## Where things stand - All three servers are deployed, and each has the same 13 tools once its prefix is removed. - Two things stop them being interchangeable: tool names carry a prefix, and the arguments differ. AWS tools take `account_id` and `region`, Azure tools don't, and GCP varies. The runtime refuses the same tool name from two servers. - Results share one envelope (`provider`, `items`, `count`, `next_page_token`, `collected_at`, `evidence_ref`), but `items` are the provider's own records, with no normalising. A Resource Graph row and a Cloud Asset record have different fields. ## Options for interchange | | A: `provider` workflow input (recommended) | B: unprefixed tools behind a logical name | |---|---|---| | How | Agents declare all three servers' tools, and the run says which cloud to use | All three drop their prefix, and `alz-cloud` points at one server | | Choose the cloud | Per run | Per KAB tenant, by re-registering | | Agents | Need provider rules in their prompts | Know nothing about clouds | | Main risk | 39 tools per agent, and the model could pick the wrong cloud | The tenant serves one cloud at a time; duplicate tool names are untested on the gateway | A fits better, because engagements on different clouds run in the same tenant. With a credential stored only for the engagement's cloud, a wrong-cloud call fails visibly. Only agents that read the cloud need these tools. ## Credentials without Vault Kubernetes Secrets can work as a stopgap, but the fallback doesn't exist yet. - Each server's pod already loads an optional `-secrets` Secret, and with no Vault the gateway sends no credential. - `REQUIRE_VAULT_CONNECTION=false` is meant to use the server's environment, but `_common/azure_support.py` reads only the request's connection, so calls would fail. `_common/gcp_support.py` has the same gap. The change is to read the environment when no connection arrives. AWS hasn't been checked. - One credential then serves every call to that server. That suits a trial on our own subscription, not real customers. When Vault arrives, turn the setting back on and delete the Secret. ## Data the cloud servers can't supply Scope and sites, CMDB and ITSM, database schemas, flows and link measurements, and the critic's call log and guard hits. Each engagement needs a real source for these, or they become declared gaps. Mixing a real cloud with the sample server's fictional CMDB would be misleading, and `meridian-baseline` can't score a real estate. ## Data adapters A data adapter parses a file type into Neo4j and PostgreSQL. It holds no credentials and reaches no other system, so it doesn't help with Vault or interchange. | Gap | Fit | |---|---| | CMDB and ITSM exports | Best fit | | Flow and firewall exports | Partial: a snapshot only | | Schema exports | Partial: only if the customer supplies them | | Scope list | Partial: a file would then decide scope | | Call log and guard hits | None: these record the run itself | Costs: one adapter per format, with fixtures that pass conformance. Product docs list only COBOL and JCL as read today, and only `kyn-cobol-ingestion-adapter` is deployed. No way for a workflow step to start ingestion was confirmed. Agents would need `knowledge-graph` access, and its free-form `query_graph` sits awkwardly with the fixed-tool rule. No server of that name was found on the cluster. For small estates, the workflow's upload step (`u1`) already passes file text to the planner and inventory agents, so a CMDB export could use that with no adapter. This came from reading the platform code and docs. No adapter was tested, and the live catalog wasn't checked. ## Recommendation to review 1. Option A for interchange. 2. The Secret fallback in each server until Vault arrives. 3. The upload step for CMDB and ITSM exports first, with adapters only if real exports are too large or varied. ## Review of the `provider` input Reviewed against runtime 0.35.7 on 2026-11-01. This replaces the recommendation for option A above. The owner has confirmed a metadata-only first discovery, but not this design. Option A as first described isn't safe enough: - **Nothing enforces the choice.** The runtime offers every declared MCP tool to the model. The `before_model` hook adds instructions only, so a prompt rule is the only guard. - **One unhealthy server stops every agent.** `resolve` fails if any declared server isn't offered by the gateway, and the gateway leaves out unhealthy servers. An `aws-discovery` fault would stop an Azure run. - **Argument and volume differences stay with the model.** Each prompt would carry per-cloud rules, and raw output (5.3 MB on Azure) would reach the model. **Recommended instead: A with neutral function tools.** The three servers stay standalone, and the choice moves into the agent's code: - Agents declare the servers under `mcp:`, and neutral tools under `functions:`, such as `list_servers(kind)`. - The function validates `provider`, calls `{provider}_{tool}` with `call_tool`, walks every `kind` and page, and returns a normalised summary. - `before_model` withholds the raw tools from the model. The Knowledge Graph Analyst already works this way (`agent.yaml` `mcp:`, `app/graph.py`). - `provider` is an enum in each input schema, which the runtime enforces, as the `unknown_source` failure showed. | Gives | Costs | |---|---| | The model can't pick the wrong cloud | The individual agents gain `app/` code and stop being purely declarative | | One normalised shape, with kinds and paging handled | Shared module across the agents, tested per cloud | | Volume is cut before the model sees it | Declared servers must still all be healthy at startup | Open points: confirm `before_model` can withhold tools under the ADK runtime these agents use, and decide how to handle the startup rule. Declaring only `azure-discovery` and `gcp-discovery` at first limits the risk but leaves AWS for later. The sub-agent pattern needs less change, because its orchestrator already makes the MCP calls. ## Secret fallback ruled out Tried on 2026-11-01. The MCP Server Builder rejects any server code that reads a credential from the environment (`_credential_env_read` in `mcp_server_service.py`), so a Kubernetes Secret can't feed these servers through rendered code. The check is a deliberate control, and editing `server.py` by hand is forbidden, so the fallback was removed. Without Vault, the gateway has no supported way to send a connection. The section above on credentials without Vault is out of date. `azure_list_data_stores` now covers PostgreSQL, MySQL and Redis (azure-discovery 0.1.3), and `gcp-discovery` 0.1.2 takes an `access_token`. Both pass their tests and ran locally against the real environments. Both are deployed to `mcp-servers` since 2026-11-02, on the public Python base like the others. ## Snapshot trial Built on 2026-11-01 so a real estate can go through the unchanged workflow without the Vault Service. `scripts/build_estate_snapshot.py` turns locally collected Google Cloud results into a snapshot. `alz-discovery-sample` 0.2.0 serves it in place of Meridian when `ESTATE_SNAPSHOT` is set, through the server's optional `-dotenv-secrets` Secret. Deleting that Secret and restarting brings Meridian back. The snapshot holds resource metadata and no credential. It's kept out of this repository. While it's loaded, the sample isn't available. CMDB, ITSM, schemas, flows and link measurements are empty, so a run shows those gaps. The agents and the workflow don't change. This doesn't test the live path to the cloud, and the Secret's roughly 1 MB limit rules out the Azure estate as it stands. ### Trial result, RUN-GCP-001 Run `exec_4d5bd23e38c3` on 2026-11-01: 3 scope entries from one Google Cloud project. All ten steps completed, and the critic's verdict was `pass`. The Secret was deleted afterwards, and the sample is back. - **The report overstates what was examined.** No source gaps are declared for CMDB, ITSM, schemas or flows, so "No network findings" and "No data stores were classified" read as clean results, and the critic passes. - **False drift.** With no CMDB at all, all three assets are reported as missing from the CMDB, with `add_ci`. - **Wrong edge.** A VM on a network is typed `app_to_app`. - **Bucket not classified.** The snapshot marks it as a data store, but it has no schema, so it's left out silently. - **Slow model call.** The CMDB step's first model call took about 10 minutes, and the cause isn't known. Fixes for all of these are made and checked offline, but not deployed: planner 0.5.0, inventory, CMDB, dependency and critic 0.3.0, classification and report 0.2.0, orchestrator 0.5.0 and sample server 0.2.1. The snapshot now holds 8 entries. The slow call wasn't diagnosed: with no CMDB, that step now replies without calling a tool, and the next run will show whether that helps. ### Trial result, RUN-GCP-002 Run `exec_f41bcfa4a04c` on 2026-11-01, on 8 entries: 8 of 8 checks, verdict `pass`, in about 90 seconds. The four missing sources were declared, no false drift was reported, and the critic listed them in `not_checked`. The CMDB step took 1.4 seconds, so the 10-minute call didn't recur. Its cause remains unknown. The Secret was deleted afterwards. ## Plan: first real discovery ## Plan: first real discovery The agents' tools are all sample tools today, so a real run changes all eight agents. | Step | Agent | Real source | Limit | |---|---|---|---| | s1 | Planner | `list_accounts`, `list_resources` | No engagement scope exists, so scope becomes what `scope_root` shows, and regions stand in for sites | | s2 | Scan | `list_servers`, `list_workloads`, `list_kubernetes_clusters`, `list_data_stores`, `get_resource_metadata` | Plans by resource type, because per-resource tasks won't scale. No OS or service detail inside a VM | | s3 | Inventory | Normalises `s2` output across providers | `list_apps` has no cloud equivalent | | s4 | CMDB | None | Declared gap, unless an export is uploaded | | s5 | Classification | `list_data_stores` | Store metadata only. No schemas, so no PII or card data findings | | s6 | Network | `get_network_topology` | No flows or link measurements, so no bottleneck or unknown-source findings | | s7 | Dependencies | Peerings, private endpoints, resource references | Configured links only, so confidence is lower | | s8 | Critic | Re-reads scope from the cloud | No call log or guard hits. Its tool-call check needs a replacement, or it fails every run | Phases: 1. **Real data, no agents.** Run each server locally, with the connection sent in the call. No cluster change, no model, and nothing leaves the laptop. Proves access and shows real output. 2. **Credentials on the cluster.** Add the Secret fallback to both servers, and create the Secrets by hand. 3. **Agents.** Add the `provider` input, and change the eight agents and the workflow. 4. **Real case.** Write expected facts from phase 1's output, then run each cloud and score it. The sample rerun comes first, because phase 3 builds on those fixes. ## Phase 1 results, 2026-11-01 Owner's decision: use a short-lived personal token for each environment, not a key or service account. **Azure:** run locally against the agreed subscription with an `az` token, through the server's own tools. Sign-in and the scope check passed. It found 644 resources of 51 types in 7 locations, including 16 VMs, 16 scale sets and 15 AKS clusters. No model was involved, and the results stayed on the laptop. - **Data store gap:** `azure_list_data_stores` covers SQL, Cosmos DB and storage accounts only. This subscription's 13 PostgreSQL flexible servers and 7 Redis caches aren't in any of its kinds. - **Default kinds:** a tool with no `kind` returns only its first kind, so agents must ask for every kind. `list_data_stores` returned 0 this way, because its default is `sql`. - **Volume:** the tools returned about 5.3 MB, of which 3.2 MB is 1,776 Advisor recommendations. The sample run was 27 KB in total, so this is the real test of ADR 0010, and agents can't be handed this much as it stands. **Google Cloud:** run locally against the trial project with a `gcloud` token. Sign-in and the scope search passed, and the tools work, but the project is close to empty: 2 assets (the project and its billing record) and 2 enabled services. Compute isn't enabled. It proves access, not discovery. `list_resource_policies` returned 403, and `list_recommendations` wasn't tested, because it needs `project_id` and `location`, which the probe didn't pass. The account can see two other projects with 87 and 124 enabled services, which weren't scanned. At the owner's request, `cloudasset`, `orgpolicy`, `recommender` and `cloudresourcemanager` were then enabled on that project. Compute wasn't, because the server doesn't call it. The rerun found 3 assets, and `list_resource_policies` still returned 403. After the owner added resources, the same check found 20 assets in 4 locations, about 40 KB in total: 1 VM, 1 disk, 1 network, 1 subnetwork, 2 routes, 1 storage bucket and 1 service account, plus service and billing records. This is enough for a small end-to-end test, but far smaller than the Azure estate. `list_data_stores` showed 0 only because the probe used its default kind, `cloud_sql`; the bucket appeared in `list_resources`. Firewall rules returned 0. **Personal tokens on the cluster:** an `az` token lasts about an hour, so a Secret would need refreshing around each run. It carries the person's own rights, not a Reader role, and anyone who can read the Secret can use it. `gcp-discovery` has no token field yet, so it would need one. Phase 1's local run replaced that server's token step in a throwaway script. ## Questions for the review - How does each engagement supply scope and sites? - Which data gaps are acceptable as declared gaps? - Does a run ever need to cover two clouds? - Is a metadata-only discovery, with these gaps declared, useful enough for a first real run? - Can the gateway's records replace the call log for the critic? Not yet reviewed.