Platform scripts, MCP servers, deployment and docs for the KAB runs
- Scripts to run workflows from a YAML context or run inputs (kab_run_workflow.py, with --input name=@file), store per-project discovery connections, bootstrap agent MCP keys, refresh the KAB token, publish reports and build estate snapshots. - Hyperscaler discovery servers extended in place; cloud-knowledge folded into them. - deploy/crucible-kab-dev.yaml: the Discovery package agent and the Assessment agents. - Platform issue write-ups and current-wiring notes under docs/platform-issues/. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ Repository-wide documents for people. The docs for each agent are in its own `do
|
||||
| [deployment.md](deployment.md) | Platform operators: building, pushing and deploying every agent and MCP server with kustomize |
|
||||
| [artifact-integration-plan-2026-10-29.md](artifact-integration-plan-2026-10-29.md) | Reviewers: how uploaded files and project artifacts reach the agents in both patterns |
|
||||
| [hyperscaler-mcp-servers-plan-2026-10-30.md](hyperscaler-mcp-servers-plan-2026-10-30.md) | Engineers and reviewers: the plan to build the Azure, AWS and Google Cloud discovery MCP servers |
|
||||
| [hyperscaler-interchange-review-note-2026-11-01.md](hyperscaler-interchange-review-note-2026-11-01.md) | Reviewers: using the real cloud servers in one set of Discovery agents, parked for future review |
|
||||
| [customer-cloud-access.md](customer-cloud-access.md) | Customer cloud administrators: granting read-only access for discovery |
|
||||
| [platform-issues/](platform-issues/) | Platform team: KAB problems found while building these agents |
|
||||
| [source-material/](source-material/) | Everyone: the inputs these agents were built from (read-only) |
|
||||
@@ -19,6 +20,9 @@ Repository-wide documents for people. The docs for each agent are in its own `do
|
||||
| [MCP server registration fails with a 500 (default tenant not mapped in the MCP gateway)](platform-issues/mcp-gateway-default-tenant/report-2026-10-29.md) | Resolved 2026-10-30: registration returns 201, and the gateway now runs with `MCP_GATEWAY_TENANCY_MODE=pooled`, which binds each caller's tenant. The manifests alongside the report weren't applied from this repository. |
|
||||
| [The Vault Service isn't deployed on the dev cluster](platform-issues/vault-service-not-deployed/report-2026-10-30.md) | Open. It blocks every stored MCP credential. |
|
||||
| [The Artifacts step can't select an earlier step's output within the same run](platform-issues/artifacts-step-within-a-run/report-2026-10-29.md) | Open, with questions for the platform team. It doesn't block F01, where hand-offs use `from_steps`. It affects F01 → F02. |
|
||||
| [A workflow can't run or chain another workflow](platform-issues/workflow-composition/report-2026-11-02.md) | Open. Each Feature is started by hand, with the previous Feature's artifact ids. It affects F01 → F02 and every later hand-off. |
|
||||
| [The Artifacts step's lookup isn't scoped to the run's tenant or project](platform-issues/select-artifacts-tenant-scope/report-2026-11-02.md) | Open, needs triage: a possible tenant-isolation defect, found by reading the code and not tested. CP1 selects by exact id until it's resolved. |
|
||||
| [MCP tools and context in the individual-agents pattern](platform-issues/agent-tools-and-context/individual-agents-report-2026-11-01.md) | Open. Some step outputs reach no later step, and large hand-offs are unconfirmed. The sub-agent pattern's review will follow |
|
||||
|
||||
## Source material
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Enter the values under **Credentials** in KAB, for the engagement's project and
|
||||
| `tenant_id` | The Entra tenant id |
|
||||
| `client_id` | The app registration's id |
|
||||
| `client_secret` | The client secret |
|
||||
| `scope_root` | `managementGroups/<id>` or `subscriptions/<id>` |
|
||||
| `scope_root` | `managementGroups/<id>`, `subscriptions/<id>`, or one resource group, `subscriptions/<id>/resourceGroups/<name>` (azure-discovery 0.1.5 or later) |
|
||||
| `cloud` | Optional: `public`, `usgov` or `china` |
|
||||
|
||||
**For a short trial on your own estate only:** leave those three fields blank and put a token in `access_token`, from `az account get-access-token --resource https://management.azure.com --query accessToken -o tsv`. It expires within about an hour, after which calls fail until you store a new one. It carries your own permissions, not Reader, so never use this for a customer.
|
||||
|
||||
@@ -113,11 +113,46 @@ python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Wor
|
||||
|
||||
## Deploying the hyperscaler MCP servers
|
||||
|
||||
`azure-discovery`, `aws-discovery`, `gcp-discovery` and `cloud-knowledge` deploy as in step 3, each with its own `<server>-secrets` Secret and registration. Their rendered Dockerfile needs a login to `kyndryl.jfrog.io`, or a local base passed with `--build-arg BASE_IMAGE=<image>`. Until that's available, build on the public Python image: `podman build -f mcp-servers/_common/Dockerfile.python -t <image> mcp-servers/<server>`. This image isn't hardened, so don't use it for customer data.
|
||||
`azure-discovery`, `aws-discovery` and `gcp-discovery` deploy as in step 3, each with its own `<server>-secrets` Secret and registration. Their rendered Dockerfile needs a login to `kyndryl.jfrog.io`, or a local base passed with `--build-arg BASE_IMAGE=<image>`. Until that's available, build on the public Python image: `podman build -f mcp-servers/_common/Dockerfile.python -t <image> mcp-servers/<server>`. This image isn't hardened, so don't use it for customer data.
|
||||
|
||||
Then store each connection in KAB under **Credentials**:
|
||||
- **Discovery servers:** per project, as described in [customer-cloud-access.md](customer-cloud-access.md). Never tenant-wide.
|
||||
- **`cloud-knowledge`:** tenant-wide. Azure needs no credential, but the server refuses a call that has no stored connection.
|
||||
Then store each connection in KAB under **Credentials**, per project, as described in
|
||||
[customer-cloud-access.md](customer-cloud-access.md). Never tenant-wide. Well-Architected guidance uses the same connection.
|
||||
|
||||
### Starting a run from a context file
|
||||
|
||||
Every run starts from a YAML context file. Its `workflow_inputs:` become the run's inputs (validated before the run
|
||||
starts). The script generates `run_id` for every run (UTC time plus a random suffix) and `plan_id` = `PLAN-<run_id>`,
|
||||
and adds them as a `generated:` section; a file that sets `run_id` itself is refused. The file also declares the
|
||||
`customer:` (business details only, no personal data) and the engagement `plan:`. The whole file is uploaded to the workflow's upload step, so the planner, the inventory agent and the
|
||||
report also read its `context:` section. The template is
|
||||
`individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/run-context.template.yaml`; the live trials have their
|
||||
own in `workflows/cases/`. Never put a credential in one.
|
||||
|
||||
```sh
|
||||
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
|
||||
--context "individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/gcp-live/context.yaml" --dry-run
|
||||
```
|
||||
|
||||
### Storing the connections from a script
|
||||
|
||||
`scripts/kab_store_connections.py` stores the discovery servers' connections, per project, through the platform's
|
||||
Credentials API, without the UI. It finds each server in the MCP gateway, checks the connection against the
|
||||
server's declared fields, and tests it with `connections:verify`. Only with `--apply` does it store the connection
|
||||
(`PUT /api/v1/secrets`, `secretType` `mcp`, format `connection`, create-only unless `--replace`).
|
||||
|
||||
Credential values never appear in its output, its arguments or a file. They come from the environment
|
||||
(`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`), from a service account key file kept outside the
|
||||
repository, or from `az` / `gcloud` access tokens. A token lasts about an hour and carries your own rights.
|
||||
|
||||
```sh
|
||||
python3 scripts/kab_store_connections.py --api https://<platform host> --tenant <tenant id> --project-id <project id> \
|
||||
--azure-scope-root subscriptions/<id> --azure-credential service-principal \
|
||||
--gcp-scope-root projects/<id> --gcp-credential key-file:$HOME/keys/<read-only-sa>.json # test only
|
||||
# ... then the same command with --apply
|
||||
```
|
||||
|
||||
It needs a KAB token (`$KAB_TOKEN` or `~/.kab_token`, `--token-file` to change) and the Administrator role. API keys work only on `/api/external/` routes, so they can't be used here. The
|
||||
servers must already be registered in that platform's MCP gateway, at an address its gateway can reach.
|
||||
|
||||
## Rolling back and removing
|
||||
|
||||
|
||||
169
docs/hyperscaler-interchange-review-note-2026-11-01.md
Normal file
169
docs/hyperscaler-interchange-review-note-2026-11-01.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# Using real cloud servers in the Discovery agents: review note
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-01 |
|
||||
| Status | Under review for a real discovery of an Azure and a Google Cloud environment. Nothing here is built, and the plan below isn't approved |
|
||||
| Scope | ALZ-F01 Discovery, both patterns |
|
||||
|
||||
## Requirements
|
||||
|
||||
- `azure-discovery`, `aws-discovery` and `gcp-discovery` stay standalone. A single server for all clouds is ruled out.
|
||||
- One workflow and one set of agents must work with any of the three. No duplicate workflows or per-cloud agents.
|
||||
|
||||
## Where things stand
|
||||
|
||||
- All three servers are deployed, and each has the same 13 tools once its prefix is removed.
|
||||
- Two things stop them being interchangeable: tool names carry a prefix, and the arguments differ. AWS tools take `account_id` and `region`, Azure tools don't, and GCP varies. The runtime refuses the same tool name from two servers.
|
||||
- Results share one envelope (`provider`, `items`, `count`, `next_page_token`, `collected_at`, `evidence_ref`), but `items` are the provider's own records, with no normalising. A Resource Graph row and a Cloud Asset record have different fields.
|
||||
|
||||
## Options for interchange
|
||||
|
||||
| | A: `provider` workflow input (recommended) | B: unprefixed tools behind a logical name |
|
||||
|---|---|---|
|
||||
| How | Agents declare all three servers' tools, and the run says which cloud to use | All three drop their prefix, and `alz-cloud` points at one server |
|
||||
| Choose the cloud | Per run | Per KAB tenant, by re-registering |
|
||||
| Agents | Need provider rules in their prompts | Know nothing about clouds |
|
||||
| Main risk | 39 tools per agent, and the model could pick the wrong cloud | The tenant serves one cloud at a time; duplicate tool names are untested on the gateway |
|
||||
|
||||
A fits better, because engagements on different clouds run in the same tenant. With a credential stored only for the engagement's cloud, a wrong-cloud call fails visibly. Only agents that read the cloud need these tools.
|
||||
|
||||
## Credentials without Vault
|
||||
|
||||
Kubernetes Secrets can work as a stopgap, but the fallback doesn't exist yet.
|
||||
|
||||
- Each server's pod already loads an optional `<server>-secrets` Secret, and with no Vault the gateway sends no credential.
|
||||
- `REQUIRE_VAULT_CONNECTION=false` is meant to use the server's environment, but `_common/azure_support.py` reads only the request's connection, so calls would fail. `_common/gcp_support.py` has the same gap. The change is to read the environment when no connection arrives. AWS hasn't been checked.
|
||||
- One credential then serves every call to that server. That suits a trial on our own subscription, not real customers. When Vault arrives, turn the setting back on and delete the Secret.
|
||||
|
||||
## Data the cloud servers can't supply
|
||||
|
||||
Scope and sites, CMDB and ITSM, database schemas, flows and link measurements, and the critic's call log and guard hits. Each engagement needs a real source for these, or they become declared gaps. Mixing a real cloud with the sample server's fictional CMDB would be misleading, and `meridian-baseline` can't score a real estate.
|
||||
|
||||
## Data adapters
|
||||
|
||||
A data adapter parses a file type into Neo4j and PostgreSQL. It holds no credentials and reaches no other system, so it doesn't help with Vault or interchange.
|
||||
|
||||
| Gap | Fit |
|
||||
|---|---|
|
||||
| CMDB and ITSM exports | Best fit |
|
||||
| Flow and firewall exports | Partial: a snapshot only |
|
||||
| Schema exports | Partial: only if the customer supplies them |
|
||||
| Scope list | Partial: a file would then decide scope |
|
||||
| Call log and guard hits | None: these record the run itself |
|
||||
|
||||
Costs: one adapter per format, with fixtures that pass conformance. Product docs list only COBOL and JCL as read today, and only `kyn-cobol-ingestion-adapter` is deployed. No way for a workflow step to start ingestion was confirmed. Agents would need `knowledge-graph` access, and its free-form `query_graph` sits awkwardly with the fixed-tool rule. No server of that name was found on the cluster.
|
||||
|
||||
For small estates, the workflow's upload step (`u1`) already passes file text to the planner and inventory agents, so a CMDB export could use that with no adapter.
|
||||
|
||||
This came from reading the platform code and docs. No adapter was tested, and the live catalog wasn't checked.
|
||||
|
||||
## Recommendation to review
|
||||
|
||||
1. Option A for interchange.
|
||||
2. The Secret fallback in each server until Vault arrives.
|
||||
3. The upload step for CMDB and ITSM exports first, with adapters only if real exports are too large or varied.
|
||||
|
||||
## Review of the `provider` input
|
||||
|
||||
Reviewed against runtime 0.35.7 on 2026-11-01. This replaces the recommendation for option A above. The owner has confirmed a metadata-only first discovery, but not this design.
|
||||
|
||||
Option A as first described isn't safe enough:
|
||||
|
||||
- **Nothing enforces the choice.** The runtime offers every declared MCP tool to the model. The `before_model` hook adds instructions only, so a prompt rule is the only guard.
|
||||
- **One unhealthy server stops every agent.** `resolve` fails if any declared server isn't offered by the gateway, and the gateway leaves out unhealthy servers. An `aws-discovery` fault would stop an Azure run.
|
||||
- **Argument and volume differences stay with the model.** Each prompt would carry per-cloud rules, and raw output (5.3 MB on Azure) would reach the model.
|
||||
|
||||
**Recommended instead: A with neutral function tools.** The three servers stay standalone, and the choice moves into the agent's code:
|
||||
|
||||
- Agents declare the servers under `mcp:`, and neutral tools under `functions:`, such as `list_servers(kind)`.
|
||||
- The function validates `provider`, calls `{provider}_{tool}` with `call_tool`, walks every `kind` and page, and returns a normalised summary.
|
||||
- `before_model` withholds the raw tools from the model. The Knowledge Graph Analyst already works this way (`agent.yaml` `mcp:`, `app/graph.py`).
|
||||
- `provider` is an enum in each input schema, which the runtime enforces, as the `unknown_source` failure showed.
|
||||
|
||||
| Gives | Costs |
|
||||
|---|---|
|
||||
| The model can't pick the wrong cloud | The individual agents gain `app/` code and stop being purely declarative |
|
||||
| One normalised shape, with kinds and paging handled | Shared module across the agents, tested per cloud |
|
||||
| Volume is cut before the model sees it | Declared servers must still all be healthy at startup |
|
||||
|
||||
Open points: confirm `before_model` can withhold tools under the ADK runtime these agents use, and decide how to handle the startup rule. Declaring only `azure-discovery` and `gcp-discovery` at first limits the risk but leaves AWS for later. The sub-agent pattern needs less change, because its orchestrator already makes the MCP calls.
|
||||
|
||||
## Secret fallback ruled out
|
||||
|
||||
Tried on 2026-11-01. The MCP Server Builder rejects any server code that reads a credential from the environment (`_credential_env_read` in `mcp_server_service.py`), so a Kubernetes Secret can't feed these servers through rendered code. The check is a deliberate control, and editing `server.py` by hand is forbidden, so the fallback was removed. Without Vault, the gateway has no supported way to send a connection. The section above on credentials without Vault is out of date.
|
||||
|
||||
`azure_list_data_stores` now covers PostgreSQL, MySQL and Redis (azure-discovery 0.1.3), and `gcp-discovery` 0.1.2 takes an `access_token`. Both pass their tests and ran locally against the real environments. Both are deployed to `mcp-servers` since 2026-11-02, on the public Python base like the others.
|
||||
|
||||
## Snapshot trial
|
||||
|
||||
Built on 2026-11-01 so a real estate can go through the unchanged workflow without the Vault Service. `scripts/build_estate_snapshot.py` turns locally collected Google Cloud results into a snapshot. `alz-discovery-sample` 0.2.0 serves it in place of Meridian when `ESTATE_SNAPSHOT` is set, through the server's optional `-dotenv-secrets` Secret. Deleting that Secret and restarting brings Meridian back.
|
||||
|
||||
The snapshot holds resource metadata and no credential. It's kept out of this repository. While it's loaded, the sample isn't available. CMDB, ITSM, schemas, flows and link measurements are empty, so a run shows those gaps. The agents and the workflow don't change. This doesn't test the live path to the cloud, and the Secret's roughly 1 MB limit rules out the Azure estate as it stands.
|
||||
|
||||
### Trial result, RUN-GCP-001
|
||||
|
||||
Run `exec_4d5bd23e38c3` on 2026-11-01: 3 scope entries from one Google Cloud project. All ten steps completed, and the critic's verdict was `pass`. The Secret was deleted afterwards, and the sample is back.
|
||||
|
||||
- **The report overstates what was examined.** No source gaps are declared for CMDB, ITSM, schemas or flows, so "No network findings" and "No data stores were classified" read as clean results, and the critic passes.
|
||||
- **False drift.** With no CMDB at all, all three assets are reported as missing from the CMDB, with `add_ci`.
|
||||
- **Wrong edge.** A VM on a network is typed `app_to_app`.
|
||||
- **Bucket not classified.** The snapshot marks it as a data store, but it has no schema, so it's left out silently.
|
||||
- **Slow model call.** The CMDB step's first model call took about 10 minutes, and the cause isn't known.
|
||||
|
||||
Fixes for all of these are made and checked offline, but not deployed: planner 0.5.0, inventory, CMDB, dependency and critic 0.3.0, classification and report 0.2.0, orchestrator 0.5.0 and sample server 0.2.1. The snapshot now holds 8 entries. The slow call wasn't diagnosed: with no CMDB, that step now replies without calling a tool, and the next run will show whether that helps.
|
||||
|
||||
### Trial result, RUN-GCP-002
|
||||
|
||||
Run `exec_f41bcfa4a04c` on 2026-11-01, on 8 entries: 8 of 8 checks, verdict `pass`, in about 90 seconds. The four missing sources were declared, no false drift was reported, and the critic listed them in `not_checked`. The CMDB step took 1.4 seconds, so the 10-minute call didn't recur. Its cause remains unknown. The Secret was deleted afterwards.
|
||||
|
||||
## Plan: first real discovery
|
||||
## Plan: first real discovery
|
||||
|
||||
The agents' tools are all sample tools today, so a real run changes all eight agents.
|
||||
|
||||
| Step | Agent | Real source | Limit |
|
||||
|---|---|---|---|
|
||||
| s1 | Planner | `list_accounts`, `list_resources` | No engagement scope exists, so scope becomes what `scope_root` shows, and regions stand in for sites |
|
||||
| s2 | Scan | `list_servers`, `list_workloads`, `list_kubernetes_clusters`, `list_data_stores`, `get_resource_metadata` | Plans by resource type, because per-resource tasks won't scale. No OS or service detail inside a VM |
|
||||
| s3 | Inventory | Normalises `s2` output across providers | `list_apps` has no cloud equivalent |
|
||||
| s4 | CMDB | None | Declared gap, unless an export is uploaded |
|
||||
| s5 | Classification | `list_data_stores` | Store metadata only. No schemas, so no PII or card data findings |
|
||||
| s6 | Network | `get_network_topology` | No flows or link measurements, so no bottleneck or unknown-source findings |
|
||||
| s7 | Dependencies | Peerings, private endpoints, resource references | Configured links only, so confidence is lower |
|
||||
| s8 | Critic | Re-reads scope from the cloud | No call log or guard hits. Its tool-call check needs a replacement, or it fails every run |
|
||||
|
||||
Phases:
|
||||
|
||||
1. **Real data, no agents.** Run each server locally, with the connection sent in the call. No cluster change, no model, and nothing leaves the laptop. Proves access and shows real output.
|
||||
2. **Credentials on the cluster.** Add the Secret fallback to both servers, and create the Secrets by hand.
|
||||
3. **Agents.** Add the `provider` input, and change the eight agents and the workflow.
|
||||
4. **Real case.** Write expected facts from phase 1's output, then run each cloud and score it.
|
||||
|
||||
The sample rerun comes first, because phase 3 builds on those fixes.
|
||||
|
||||
## Phase 1 results, 2026-11-01
|
||||
|
||||
Owner's decision: use a short-lived personal token for each environment, not a key or service account.
|
||||
|
||||
**Azure:** run locally against the agreed subscription with an `az` token, through the server's own tools. Sign-in and the scope check passed. It found 644 resources of 51 types in 7 locations, including 16 VMs, 16 scale sets and 15 AKS clusters. No model was involved, and the results stayed on the laptop.
|
||||
|
||||
- **Data store gap:** `azure_list_data_stores` covers SQL, Cosmos DB and storage accounts only. This subscription's 13 PostgreSQL flexible servers and 7 Redis caches aren't in any of its kinds.
|
||||
- **Default kinds:** a tool with no `kind` returns only its first kind, so agents must ask for every kind. `list_data_stores` returned 0 this way, because its default is `sql`.
|
||||
- **Volume:** the tools returned about 5.3 MB, of which 3.2 MB is 1,776 Advisor recommendations. The sample run was 27 KB in total, so this is the real test of ADR 0010, and agents can't be handed this much as it stands.
|
||||
|
||||
**Google Cloud:** run locally against the trial project with a `gcloud` token. Sign-in and the scope search passed, and the tools work, but the project is close to empty: 2 assets (the project and its billing record) and 2 enabled services. Compute isn't enabled. It proves access, not discovery. `list_resource_policies` returned 403, and `list_recommendations` wasn't tested, because it needs `project_id` and `location`, which the probe didn't pass. The account can see two other projects with 87 and 124 enabled services, which weren't scanned.
|
||||
|
||||
At the owner's request, `cloudasset`, `orgpolicy`, `recommender` and `cloudresourcemanager` were then enabled on that project. Compute wasn't, because the server doesn't call it. The rerun found 3 assets, and `list_resource_policies` still returned 403.
|
||||
|
||||
After the owner added resources, the same check found 20 assets in 4 locations, about 40 KB in total: 1 VM, 1 disk, 1 network, 1 subnetwork, 2 routes, 1 storage bucket and 1 service account, plus service and billing records. This is enough for a small end-to-end test, but far smaller than the Azure estate. `list_data_stores` showed 0 only because the probe used its default kind, `cloud_sql`; the bucket appeared in `list_resources`. Firewall rules returned 0.
|
||||
|
||||
**Personal tokens on the cluster:** an `az` token lasts about an hour, so a Secret would need refreshing around each run. It carries the person's own rights, not a Reader role, and anyone who can read the Secret can use it. `gcp-discovery` has no token field yet, so it would need one. Phase 1's local run replaced that server's token step in a throwaway script.
|
||||
|
||||
## Questions for the review
|
||||
|
||||
- How does each engagement supply scope and sites?
|
||||
- Which data gaps are acceptable as declared gaps?
|
||||
- Does a run ever need to cover two clouds?
|
||||
- Is a metadata-only discovery, with these gaps declared, useful enough for a first real run?
|
||||
- Can the gateway's records replace the call log for the critic? Not yet reviewed.
|
||||
@@ -4,6 +4,7 @@
|
||||
|---|---|
|
||||
| Date | 2026-10-30 |
|
||||
| Status | Built to simulated-cloud tests. See [Build status](#build-status-2026-10-30) |
|
||||
| Update 2026-11-04 | `cloud-knowledge` is retired. Guidance is a tool on `azure-discovery` and `gcp-discovery`, on the customer's connection (D9). Sections below that describe `cloud-knowledge` are history |
|
||||
| Scope | Three read-only MCP servers that collect metadata from a customer's cloud estate for CP1 Discovery (ALZ-F01), one folder each under `mcp-servers/` |
|
||||
| Reference for Azure | [Azure MCP Server](https://github.com/microsoft/mcp/blob/main/servers/Azure.Mcp.Server/README.md) |
|
||||
| Platform code read | `modernization-platform` @ `c89aedcb9` |
|
||||
@@ -244,8 +245,8 @@ Exact role lists are confirmed in each server's phase, tested against a real san
|
||||
| D6 | Is `query_resources` in v1? | **Yes, with scope and size limits** | It's the most useful Azure MCP capability for discovery, and the underlying APIs are read-only. The model writes the query, so it needs the boundary check. |
|
||||
| D7 | Are there sandboxes for all three clouds? | **We need one per provider, with an organisation-level structure** | Management group and organisation behaviour can't be tested in a single account. |
|
||||
| D8 | Does discovery look inside Kubernetes clusters (namespaces, deployments, images)? | **Not in v1.** Add it as a later phase | It needs read access granted on each cluster, and private clusters may be unreachable from KAB. |
|
||||
| D9 | Where does Kyndryl's own access to AWS and Google knowledge live? | **Under review.** Options are in [Well-Architected integration](#well-architected-integration) | The platform review showed it can't sit on the discovery servers' vault credential. |
|
||||
| D10 | Should Discovery use Well-Architected guidance, or only Target Architecture? | **Decided: Target Architecture.** Discovery collects findings only | Agreed 2026-10-30. |
|
||||
| D9 | Where does Kyndryl's own access to AWS and Google knowledge live? | **Decided 2026-11-04: nowhere separate.** Guidance is a tool on each provider's discovery server, on the customer's connection, and `cloud-knowledge` is retired. Options were in [Well-Architected integration](#well-architected-integration) | The platform review showed it can't sit on the discovery servers' vault credential. |
|
||||
| D10 | Should Discovery use Well-Architected guidance, or only Target Architecture? | **Decided: Target Architecture.** Discovery collects findings only. **Superseded 2026-11-03** by ALZ-F01 ADR 0014: Discovery runs a Well-Architected review | Agreed 2026-10-30. |
|
||||
| D11 | Are people's names and emails in collected data kept? | **Decided: no, they are anonymised. Finishing this is deferred** | Agreed 2026-10-30, and deferred the same day. See [Later work](#later-work). |
|
||||
|
||||
## 4. Delivery plan
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
# MCP tools and context: individual-agents pattern
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-01 |
|
||||
| Environment | KAB dev (`crucible-kab-dev-aks`), agent runtime 0.35.7 |
|
||||
| Component | ALZ-F01 Discovery, `individual-agents/`; the KAB workflow executor |
|
||||
| Severity | High. The pattern depends on this, and the outputs below reach no later step |
|
||||
| Status | Open. The platform team hasn't confirmed how large step outputs are handled (ADR 0010) |
|
||||
| Found by | Review of the agents' tools and the workflow's wiring |
|
||||
|
||||
For engineers changing these agents, and for reviewers judging their output. Agents collect their own facts, and the workflow carries only what each step returns.
|
||||
|
||||
The main tables are generated from `agent.yaml`, `schemas/` and the workflow export. `scripts/check_context_doc.py`, in the Feature folder, fails if they drift. Run it after changing an agent's tools, a schema or `STEPS`.
|
||||
|
||||
## Summary
|
||||
|
||||
- **Each agent reads its own sources.** It holds only the read-only MCP tools its job needs.
|
||||
- **Each agent starts blank.** It has no memory of earlier steps or runs, and it sees no other agent's reasoning.
|
||||
- **Context is data, not conversation.** A step gets `engagement_id`, `run_id`, and the named fields it's wired to receive.
|
||||
- **There's no shared store.** A step's output exists only as its reply (ADR 0010).
|
||||
|
||||
## Why each agent has MCP tools
|
||||
|
||||
| Reason | What it means |
|
||||
|---|---|
|
||||
| Least privilege | An agent can only use the tools it declares. The classifier can't scan the estate, and the planner can't query the CMDB. |
|
||||
| Evidence from the source | Agents cite real tool results, so a finding can be checked. Each fact carries an evidence reference. |
|
||||
| An independent check | The critic reads the scope and the call log itself, and doesn't rely on what other agents report. |
|
||||
| Read-only | Every tool only reads. The stories' write tools are replaced by each agent's reply (ADR 0003). |
|
||||
|
||||
Only top-level agents can hold MCP tools. That's one reason this pattern differs from the sub-agent pattern, where the orchestrator makes every call.
|
||||
|
||||
| Step | Agent | MCP server: tools | What it fetches |
|
||||
|---|---|---|---|
|
||||
| s1 | `cp1-discovery-planner-agent` | `alz-scope`: `get_scope`, `list_sources`, `get_budgets` | What is in scope, and the limits |
|
||||
| s2 | `cp1-estate-scan-executor-agent` | `alz-estate`: `scan_target`; `alz-cloud`: `get_resource_metadata` | Facts about each estate target |
|
||||
| s3 | `cp1-asset-inventory-normalisation-agent` | `alz-inventory`: `list_apps`, `get_app` | Application records |
|
||||
| s4 | `cp1-cmdb-itsm-reconciliation-agent` | `alz-cmdb`: `ingest_cmdb`, `ingest_itsm`, `cmdb_query`, `diff_records` | CMDB and ITSM records to compare |
|
||||
| s5 | `cp1-data-classification-agent` | `alz-schema`: `scan_schema` | Database schema metadata |
|
||||
| s6 | `cp1-network-discovery-assessment-agent` | `alz-network`: `collect_topology`, `map_flows`, `analyze_bottlenecks` | Network layout and traffic |
|
||||
| s7 | `cp1-dependency-mapping-agent` | `alz-dependency`: `get_dependencies`, `trace_calls`, `trace` | Evidence of how systems connect |
|
||||
| s8 | `cp1-discovery-critic-agent` | `alz-scope`: `get_scope`; `alz-evidence`: `get_call_log`, `get_guard_hits` | The true scope, and what the agents actually called |
|
||||
| s9 | `cp1-discovery-report-agent` | None | It calls no model or MCP tool. It writes `report_markdown`, `report_html` and `final_html` from code, and returns `counts` |
|
||||
|
||||
The server names are logical. In the sample setup, all nine are registered in the `CP1 Agentic Landing Zone` tenant against one server, `alz-discovery-sample`, so each can later point to a real system. The hyperscaler discovery servers aren't declared here, so a run reads only the Meridian sample.
|
||||
|
||||
## How context carries across
|
||||
|
||||
```
|
||||
Run form: engagement_id, run_id ──────────► every step
|
||||
Upload (u1) ── parsed_content ────────────► s1, s3
|
||||
|
||||
s1 Planner ── tasks ──┬─► s2 Scan ── findings ─► s3 Inventory ── records ─┐
|
||||
├─► s4 CMDB ◄───────────────────────────────────────┤
|
||||
└─► s6 Network ◄────────────────────────────────────┤
|
||||
s5 Classification ◄──────┤
|
||||
s7 Dependencies ◄────────┘
|
||||
(also drift_items from s4,
|
||||
flows from s6)
|
||||
s8 Critic ◄── seven fields from earlier steps
|
||||
s9 Report ◄── 21 fields from s1 to s8 ──► Engagement Lead's approval
|
||||
```
|
||||
|
||||
1. **The reply is checked.** It must match the step's output contract. If it doesn't, it's sent back once, then the step fails with `output_contract_violation`.
|
||||
2. **The workflow copies named fields.** `from_steps` wires, for example, `s2.findings` into s3's `findings`.
|
||||
3. **The input is checked.** A request that breaks the input contract is refused before any model call.
|
||||
4. **The build checks the wiring.** `scripts/build_workflow.py` asserts that every field passed between steps exists on both sides.
|
||||
|
||||
| Step | Agent | Receives | Returns |
|
||||
|---|---|---|---|
|
||||
| s1 | `discovery-planner` | `source_documents` from `u1.parsed_content` | `plan_id`, `engagement_id`, `tasks`, `coverage_gaps`, `summary`, `findings_note` |
|
||||
| s2 | `estate-scan-executor` | `tasks` from `s1.tasks` | `findings` |
|
||||
| s3 | `asset-inventory-normalisation` | `findings` from `s2.findings`, `source_documents` from `u1.parsed_content` | `records`, `sme_review_queue` |
|
||||
| s4 | `cmdb-itsm-reconciliation` | `tasks` from `s1.tasks`, `records` from `s3.records`, `source_gaps` from `s1.source_gaps` | `report_id`, `engagement_id`, `drift_items`, `itsm_signals`, `summary` |
|
||||
| s5 | `data-classification` | `records` from `s3.records`, `source_gaps` from `s1.source_gaps` | `stores` |
|
||||
| s6 | `network-discovery-assessment` | `tasks` from `s1.tasks`, `records` from `s3.records` | `ranges`, `flows`, `findings` |
|
||||
| s7 | `dependency-mapping` | `records` from `s3.records`, `drift_items` from `s4.drift_items`, `flows` from `s6.flows` | `graph_id`, `engagement_id`, `edges`, `nodes`, `orphans`, `validation_queue` |
|
||||
| s8 | `discovery-critic` | `coverage_gaps` from `s1.coverage_gaps`, `findings` from `s2.findings`, `records` from `s3.records`, `drift_items` from `s4.drift_items`, `stores` from `s5.stores`, `network_findings` from `s6.findings`, `edges` from `s7.edges`, `source_gaps` from `s1.source_gaps` | `coverage_complete`, `coverage_gaps`, `declared_gaps`, `injections_found`, `obeyed_injection`, `unsupported_claims`, `consistency_issues`, `parse_failures`, `tool_call_issues`, `verdict`, `note` |
|
||||
| s9 | `discovery-report` | `plan_summary` from `s1.summary`, `source_gaps` from `s1.source_gaps`, `coverage_gaps` from `s1.coverage_gaps`, `scan_findings` from `s2.findings`, `records` from `s3.records`, `sme_review_queue` from `s3.sme_review_queue`, `drift_items` from `s4.drift_items`, `itsm_signals` from `s4.itsm_signals`, `stores` from `s5.stores`, `network_findings` from `s6.findings`, `ranges` from `s6.ranges`, `edges` from `s7.edges`, `nodes` from `s7.nodes`, `orphans` from `s7.orphans`, `validation_queue` from `s7.validation_queue`, `verdict` from `s8.verdict`, `critic_note` from `s8.note`, `obeyed_injection` from `s8.obeyed_injection`, `unsupported_claims` from `s8.unsupported_claims`, `consistency_issues` from `s8.consistency_issues`, `parse_failures` from `s8.parse_failures`, `tool_call_issues` from `s8.tool_call_issues` |
|
||||
|
||||
`source_gaps` from the planner reaches s4, s5, s8 and s9, so a missing CMDB, ITSM, flow or schema source is declared and never read as a clean result.
|
||||
|
||||
## What doesn't carry across
|
||||
|
||||
These output fields reach no later step, not even the critic. They're kept in the step's reply, which KAB registers as an `execution_output` artifact, but no agent reads or judges them.
|
||||
|
||||
| Step | Fields |
|
||||
|---|---|
|
||||
| s1 | `plan_id`, `engagement_id`, `findings_note` |
|
||||
| s2 | |
|
||||
| s3 | |
|
||||
| s4 | `report_id`, `engagement_id`, `summary` |
|
||||
| s5 | |
|
||||
| s6 | |
|
||||
| s7 | `graph_id`, `engagement_id` |
|
||||
| s8 | `coverage_complete`, `coverage_gaps`, `declared_gaps`, `injections_found`, `not_checked` |
|
||||
|
||||
Most are identifiers. Since the report step was added (ADR 0012), the findings once listed here reach the reader, so the agents still don't see them, but the Engagement Lead does.
|
||||
|
||||
## Risks to watch
|
||||
|
||||
| Risk | Effect | What to check |
|
||||
|---|---|---|
|
||||
| Unjudged outputs | The critic can't see the fields above | Whether a problem appears only there |
|
||||
| Growing context | `records` reaches four agents, and the critic receives seven fields | Truncation, slow steps, or missed entries on large estates |
|
||||
| Large hand-offs | ADR 0010 relies on the executor handling large outputs, which the platform team hasn't confirmed | Whether a later step received its field in full |
|
||||
| No memory between runs | A rerun starts from nothing | Repeated work and inconsistent ids across runs |
|
||||
| No hash chain | Nothing proves a hand-off wasn't altered. Integrity rests on evidence references and the critic's call-log check | Evidence references that don't match a real call |
|
||||
| Silent gaps | An agent only knows its inputs and tools | Scope entries that disappear between steps |
|
||||
| Shared model | The critic and the executors all use `gpt-5.6-sol` (ADR 0008) | That the critic's `note` says so, and whether its verdict is earned |
|
||||
|
||||
## For the platform team
|
||||
|
||||
- Does the executor externalise large step outputs between steps, and from what size? (ADR 0010)
|
||||
- Can a later step read an earlier step's fields that aren't wired to it, without adding them to every contract?
|
||||
|
||||
## Current wiring (2026-11-05)
|
||||
|
||||
The sections above describe the pattern as it stood on 2026-11-01, on the Meridian sample estate. Since then:
|
||||
|
||||
- **Live providers (ADR 0013).** Each agent also declares read-only tools on the live discovery servers, under the
|
||||
gateway's display names (`Azure Discovery`, `Google Cloud Discovery`). The `alz-*` sample tools still serve
|
||||
`provider: sample`.
|
||||
- **Code answers several steps.** The planner, CMDB step, dependency map, critic's coverage counts,
|
||||
Well-Architected review, report and package are computed in `pre_process` hooks (ADRs 0014, 0015, 0017, 0018,
|
||||
0019, 0020). Where a model is used, its reply is checked in code.
|
||||
- **New steps:** the Well-Architected review (`s11`, ADR 0014), and the Discovery package for the Assessment workflow
|
||||
(`s13`, ADR 0020), which runs after the report's review point and carries every step's output to F02 in one
|
||||
artifact. Fields marked unread below now reach F02 through the package, except the run's own ids.
|
||||
- **Applications and owners (ADR 0021):** the dependency step groups records into applications (CMDB business
|
||||
service, then application tag, then container, then a judged model proposal) and names each asset's owner from the
|
||||
CMDB or an owner tag; the report and the package carry them.
|
||||
|
||||
This section is written from `workflows/cp1-alz-discovery-individual.json` and each `agent.yaml`, and
|
||||
`scripts/check_context_doc.py` checks it.
|
||||
|
||||
### Tools by step
|
||||
|
||||
| Step | Agent | Answered by | MCP tools |
|
||||
|---|---|---|---|
|
||||
| s1 | `cp1-discovery-planner-agent` | code (`hooks`) | alz-scope: `get_scope`, `list_sources`, `get_budgets`; Azure Discovery: `azure_list_accounts`, `azure_list_resources`, `azure_list_servers`, `azure_list_workloads`, `azure_list_kubernetes_clusters`, `azure_list_data_stores`; Google Cloud Discovery: `gcp_list_accounts`, `gcp_list_resources`, `gcp_list_servers`, `gcp_list_workloads`, `gcp_list_kubernetes_clusters`, `gcp_list_data_stores` |
|
||||
| s2 | `cp1-estate-scan-executor-agent` | model | alz-estate: `scan_target`; alz-cloud: `get_resource_metadata`; Azure Discovery: `azure_get_resource_metadata`; Google Cloud Discovery: `gcp_get_resource_metadata` |
|
||||
| s3 | `cp1-asset-inventory-normalisation-agent` | code (`hooks`), with a judged sub-agent | alz-inventory: `list_apps`, `get_app` |
|
||||
| s4 | `cp1-cmdb-itsm-reconciliation-agent` | code (`hooks`), with a judged sub-agent | alz-cmdb: `ingest_cmdb`, `ingest_itsm`, `cmdb_query`, `diff_records`; Azure Discovery: `azure_list_resources`; Google Cloud Discovery: `gcp_list_resources` |
|
||||
| s5 | `cp1-data-classification-agent` | model | alz-schema: `scan_schema`; Azure Discovery: `azure_list_data_stores`; Google Cloud Discovery: `gcp_list_data_stores` |
|
||||
| s6 | `cp1-network-discovery-assessment-agent` | model | alz-network: `collect_topology`, `map_flows`, `analyze_bottlenecks`; Azure Discovery: `azure_get_network_topology`; Google Cloud Discovery: `gcp_get_network_topology` |
|
||||
| s7 | `cp1-dependency-mapping-agent` | code (`hooks`), with a judged sub-agent | alz-dependency: `get_dependencies`, `trace_calls`, `trace` |
|
||||
| s8 | `cp1-discovery-critic-agent` | code (`hooks`) | alz-scope: `get_scope`; alz-evidence: `get_call_log`, `get_guard_hits`; Azure Discovery: `azure_list_resources`, `azure_list_servers`, `azure_list_workloads`, `azure_list_kubernetes_clusters`, `azure_list_data_stores`; Google Cloud Discovery: `gcp_list_resources`, `gcp_list_servers`, `gcp_list_workloads`, `gcp_list_kubernetes_clusters`, `gcp_list_data_stores` |
|
||||
| s11 | `cp1-well-architected-review-agent` | code (`hooks`) | Azure Discovery: `azure_list_servers`, `azure_list_workloads`, `azure_list_kubernetes_clusters`, `azure_list_data_stores`, `azure_get_network_topology`, `azure_list_organization_policies`, `azure_list_resource_policies`, `azure_list_recommendations`, `azure_search_well_architected`; Google Cloud Discovery: `gcp_list_servers`, `gcp_list_workloads`, `gcp_list_kubernetes_clusters`, `gcp_list_data_stores`, `gcp_get_network_topology`, `gcp_list_organization_policies`, `gcp_list_recommendations`, `gcp_search_well_architected` |
|
||||
| s9 | `cp1-discovery-report-agent` | code (`hooks`) | None |
|
||||
| s13 | `cp1-discovery-package-agent` | code (`hooks`) | None |
|
||||
|
||||
### Hand-offs
|
||||
|
||||
| Step | Takes | Returns |
|
||||
|---|---|---|
|
||||
| s1 | `source_documents` from `u1.parsed_content` | `plan_id`, `engagement_id`, `tasks`, `coverage_gaps`, `summary`, `findings_note`, `source_gaps`, `scope_accounting` |
|
||||
| s2 | `tasks` from `s1.tasks` | `findings` |
|
||||
| s3 | `findings` from `s2.findings`, `source_documents` from `u1.parsed_content` | `records`, `sme_review_queue` |
|
||||
| s4 | `tasks` from `s1.tasks`, `records` from `s3.records`, `source_gaps` from `s1.source_gaps`, `source_documents` from `u1.parsed_content` | `report_id`, `engagement_id`, `drift_items`, `itsm_signals`, `summary`, `injections_found`, `cmdb_sources`, `note`, `cis`, `relationships`, `dangling_relationships` |
|
||||
| s5 | `records` from `s3.records`, `source_gaps` from `s1.source_gaps` | `stores` |
|
||||
| s6 | `tasks` from `s1.tasks`, `records` from `s3.records` | `ranges`, `flows`, `findings`, `topology_summary` |
|
||||
| s7 | `records` from `s3.records`, `drift_items` from `s4.drift_items`, `flows` from `s6.flows`, `findings` from `s2.findings`, `cmdb_cis` from `s4.cis`, `cmdb_relationships` from `s4.relationships`, `cmdb_dangling` from `s4.dangling_relationships` | `graph_id`, `engagement_id`, `edges`, `nodes`, `orphans`, `validation_queue`, `folded_dependencies`, `dangling`, `graph_comparison`, `note`, `applications`, `asset_owners`, `ungrouped_assets`, `applications_note` |
|
||||
| s8 | `coverage_gaps` from `s1.coverage_gaps`, `findings` from `s2.findings`, `records` from `s3.records`, `drift_items` from `s4.drift_items`, `stores` from `s5.stores`, `network_findings` from `s6.findings`, `edges` from `s7.edges`, `source_gaps` from `s1.source_gaps`, `cmdb_injections` from `s4.injections_found`, `nodes` from `s7.nodes`, `folded_dependencies` from `s7.folded_dependencies` | `coverage_complete`, `coverage_gaps`, `declared_gaps`, `injections_found`, `obeyed_injection`, `unsupported_claims`, `consistency_issues`, `parse_failures`, `tool_call_issues`, `verdict`, `note`, `not_checked` |
|
||||
| s11 | `records` from `s3.records`, `source_gaps` from `s1.source_gaps`, `verdict` from `s8.verdict` | `engagement_id`, `run_id`, `provider`, `status`, `note`, `pillars`, `checks`, `recommendations`, `call_issues`, `evidence_refs`, `waf_markdown`, `waf_html`, `final_waf_html` |
|
||||
| s9 | `topology_summary` from `s6.topology_summary`, `waf_status` from `s11.status`, `waf_pillars` from `s11.pillars`, `waf_note` from `s11.note`, `plan_summary` from `s1.summary`, `coverage_gaps` from `s1.coverage_gaps`, `source_gaps` from `s1.source_gaps`, `scan_findings` from `s2.findings`, `records` from `s3.records`, `sme_review_queue` from `s3.sme_review_queue`, `drift_items` from `s4.drift_items`, `itsm_signals` from `s4.itsm_signals`, `stores` from `s5.stores`, `network_findings` from `s6.findings`, `ranges` from `s6.ranges`, `edges` from `s7.edges`, `nodes` from `s7.nodes`, `orphans` from `s7.orphans`, `validation_queue` from `s7.validation_queue`, `verdict` from `s8.verdict`, `critic_note` from `s8.note`, `obeyed_injection` from `s8.obeyed_injection`, `unsupported_claims` from `s8.unsupported_claims`, `consistency_issues` from `s8.consistency_issues`, `parse_failures` from `s8.parse_failures`, `tool_call_issues` from `s8.tool_call_issues`, `cmdb_sources` from `s4.cmdb_sources`, `cmdb_injections` from `s4.injections_found`, `graph_comparison` from `s7.graph_comparison`, `folded_dependencies` from `s7.folded_dependencies`, `dangling` from `s7.dangling`, `applications` from `s7.applications`, `asset_owners` from `s7.asset_owners`, `ungrouped_assets` from `s7.ungrouped_assets`, `source_documents` from `u1.parsed_content` | `engagement_id`, `run_id`, `verdict`, `counts`, `report_markdown`, `report_html`, `final_html` |
|
||||
| s13 | `plan_id` from `s1.plan_id`, `plan_summary` from `s1.summary`, `coverage_gaps` from `s1.coverage_gaps`, `source_gaps` from `s1.source_gaps`, `scope_accounting` from `s1.scope_accounting`, `scan_findings` from `s2.findings`, `records` from `s3.records`, `sme_review_queue` from `s3.sme_review_queue`, `drift_items` from `s4.drift_items`, `itsm_signals` from `s4.itsm_signals`, `cmdb_summary` from `s4.summary`, `cmdb_cis` from `s4.cis`, `cmdb_relationships` from `s4.relationships`, `cmdb_dangling` from `s4.dangling_relationships`, `cmdb_sources` from `s4.cmdb_sources`, `cmdb_injections` from `s4.injections_found`, `stores` from `s5.stores`, `ranges` from `s6.ranges`, `flows` from `s6.flows`, `network_findings` from `s6.findings`, `topology_summary` from `s6.topology_summary`, `nodes` from `s7.nodes`, `edges` from `s7.edges`, `orphans` from `s7.orphans`, `validation_queue` from `s7.validation_queue`, `folded_dependencies` from `s7.folded_dependencies`, `dangling` from `s7.dangling`, `graph_comparison` from `s7.graph_comparison`, `applications` from `s7.applications`, `asset_owners` from `s7.asset_owners`, `ungrouped_assets` from `s7.ungrouped_assets`, `applications_note` from `s7.applications_note`, `verdict` from `s8.verdict`, `critic_note` from `s8.note`, `coverage_complete` from `s8.coverage_complete`, `unsupported_claims` from `s8.unsupported_claims`, `consistency_issues` from `s8.consistency_issues`, `obeyed_injection` from `s8.obeyed_injection`, `not_checked` from `s8.not_checked`, `waf_status` from `s11.status`, `waf_pillars` from `s11.pillars`, `waf_checks` from `s11.checks`, `waf_recommendations` from `s11.recommendations`, `waf_note` from `s11.note`, `report_counts` from `s9.counts`, `report_markdown` from `s9.report_markdown` | `engagement_id`, `run_id`, `package_version`, `manifest`, `plan`, `scan`, `inventory`, `cmdb`, `classification`, `network`, `dependencies`, `critic`, `well_architected`, `report`, `applications` |
|
||||
|
||||
### Fields no later step reads
|
||||
|
||||
| Step | Fields |
|
||||
|---|---|
|
||||
| s1 | `engagement_id`, `findings_note` |
|
||||
| s2 | |
|
||||
| s3 | |
|
||||
| s4 | `report_id`, `engagement_id`, `note` |
|
||||
| s5 | |
|
||||
| s6 | |
|
||||
| s7 | `graph_id`, `engagement_id`, `note` |
|
||||
| s8 | `coverage_gaps`, `declared_gaps`, `injections_found` |
|
||||
| s11 | `engagement_id`, `run_id`, `provider`, `call_issues`, `evidence_refs`, `waf_markdown`, `waf_html` |
|
||||
| s13 | `engagement_id`, `run_id`, `package_version`, `manifest`, `plan`, `scan`, `inventory`, `cmdb`, `classification`, `network`, `dependencies`, `critic`, `well_architected`, `report`, `applications` |
|
||||
|
||||
## Changing this
|
||||
|
||||
- **Adding a hand-off:** add it to `STEPS` in `scripts/build_workflow.py`, and to the receiving agent's `schemas/input.json`. Rebuild the workflow, and update this page.
|
||||
- **Adding a tool:** add it to the agent's `mcp:` list and its prompt, and confirm it's read-only and registered at the gateway. A declared tool the gateway doesn't offer stops the agent at startup.
|
||||
- **Changing a requirement:** add an ADR, as `CLAUDE.md` describes.
|
||||
@@ -9,6 +9,19 @@
|
||||
| Status | Open; reported to the platform team for confirmation |
|
||||
| Found by | Code review while building the CP1 Discovery workflows (`sub-agents/` and `individual-agents/`, ALZ-F01) |
|
||||
|
||||
## Update 2026-11-02
|
||||
|
||||
Further code reading, still @ `c89aedcb9`, answers two of the questions below:
|
||||
|
||||
- **Question 3, run-time ids: yes, already possible.** A run input named `<step_id>_artifact_ids` is mapped onto that
|
||||
step's `artifact_ids` (`generic_workflow_executor.py:199-210`). An empty selection still fails.
|
||||
- **Question 4, the `workflow_id` match: one run, chosen arbitrarily.** The query can match several runs' artifacts, but
|
||||
`_resolve` returns the first document found (`artifacts_activities.py:139-146`). The match also has no tenant or
|
||||
project filter: see [select-artifacts-tenant-scope](../select-artifacts-tenant-scope/report-2026-11-02.md).
|
||||
|
||||
Running or chaining one workflow from another is reported separately, in
|
||||
[workflow-composition](../workflow-composition/report-2026-11-02.md).
|
||||
|
||||
## Summary
|
||||
|
||||
You can place an Artifacts step (`select_artifacts`) anywhere in a workflow, including between two agent steps. Within one run, though, it can't select anything an earlier step of that same run produced. The executor registers a run's step outputs as project artifacts only **after every step has finished**. While the run is in progress, its own outputs aren't artifacts yet.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Artifacts step: the lookup isn't scoped to the run's tenant or project
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-02 |
|
||||
| Environment | `modernization-platform` @ `c89aedcb9`, read from source. Not reproduced on `crucible-kab-dev-aks` |
|
||||
| Component | `agent-builder-temporal-workers`: the `select_artifacts` activity ("Artifacts" step). `agent-builder-api`: the execution routes' artifact check |
|
||||
| Severity | Possibly high: a tenant-isolation defect, if it's confirmed. Unconfirmed, because another layer may block it |
|
||||
| Status | Open; needs triage by the platform team |
|
||||
| Found by | Code review of the hand-off from ALZ-F01 Discovery to ALZ-F02 Assessment |
|
||||
|
||||
## Summary
|
||||
|
||||
`SelectArtifactsActivity.execute` reads the raw `artifacts` collection with no tenant or project filter. It matches
|
||||
each requested entry against the artifact's `_id`, `workflow_id` or `name`:
|
||||
|
||||
```python
|
||||
# packages/agent-builder-temporal-workers/src/temporal/activities/artifacts_activities.py, around lines 107-125
|
||||
collection = db.db["artifacts"]
|
||||
...
|
||||
query["$or"].extend([
|
||||
{"_id": {"$in": raw_ids}},
|
||||
{"workflow_id": {"$in": raw_ids}},
|
||||
{"name": {"$in": raw_ids}},
|
||||
])
|
||||
```
|
||||
|
||||
From reading the code, an id, name or workflow id that belongs to another tenant or project would resolve, and its
|
||||
content would be inlined into the run.
|
||||
|
||||
## What happens
|
||||
|
||||
| # | Behaviour | Code |
|
||||
|---|---|---|
|
||||
| 1 | The activity reads the unscoped `db.db["artifacts"]` collection | `temporal/activities/artifacts_activities.py:106` |
|
||||
| 2 | Nothing in the activity refers to `tenant_id` or `project_id`, although `ActivityContext` carries both | `temporal/activities/base_activity.py:41-42` |
|
||||
| 3 | The API's own check, when a run starts, uses the tenant-scoped `db._scoped("artifacts")`. But it only checks file types: it doesn't reject ids from outside the tenant | `agent-builder-api/src/api/workflows/workflow_execution_routes.py:1233` |
|
||||
| 4 | That API check reads only ids set when the workflow was built (`step.inputs.artifact_ids`). Ids given as the run input `<step_id>_artifact_ids` aren't checked at all | Same file, `:1244`; the auto-mapping is in `generic_workflow_executor.py:199-210` |
|
||||
| 5 | A match on `workflow_id` or `name` resolves to the first document found. Given a workflow id, the step returns one run's output, and nothing guarantees which | `artifacts_activities.py:139-146` `_resolve` |
|
||||
|
||||
## Not yet checked
|
||||
|
||||
- Whether Mongo users or roles in the worker limit what it can read.
|
||||
- Whether the UI or the API validates run inputs before the run starts.
|
||||
- Whether artifact names are unique across tenants. They're `{execution_id}_{step_id}` for step outputs, but a name on
|
||||
an uploaded artifact is chosen by the uploader.
|
||||
|
||||
## Impact on the CP1 workflows
|
||||
|
||||
None found so far: CP1 runs in one tenant. Until this is triaged, CP1 workflows select artifacts by exact `_id` only.
|
||||
|
||||
## Questions for the platform team
|
||||
|
||||
1. Can a run in tenant A select an artifact owned by tenant B through this step?
|
||||
2. If it can, could the query be scoped to the run's tenant and project, with any id resolving outside them rejected?
|
||||
3. Should `workflow_id` and `name` matches be kept? If so, could they be made deterministic, for example the latest
|
||||
completed run?
|
||||
|
||||
## Evidence
|
||||
|
||||
Code reading of `modernization-platform` @ `c89aedcb9`. Nothing was run against the platform, and nothing was changed.
|
||||
@@ -0,0 +1,81 @@
|
||||
# A workflow can't run or chain another workflow
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-02 |
|
||||
| Environment | `modernization-platform` @ `c89aedcb9`, read from source. Not reproduced on `crucible-kab-dev-aks` |
|
||||
| Component | `agent-builder-temporal-workers`: the generic workflow executor and its step types. `agent-builder-api`: execution routes |
|
||||
| Severity | Medium. Nothing fails, but a multi-stage engagement has to be wired by hand, workflow by workflow |
|
||||
| Status | Open; for the platform team to confirm |
|
||||
| Found by | Code review while planning the hand-off from ALZ-F01 Discovery to ALZ-F02 Assessment |
|
||||
|
||||
## Summary
|
||||
|
||||
A KAB workflow can't contain another workflow, and nothing starts a workflow when another one completes. Data moves
|
||||
between workflows only through project artifacts. A person has to start the next workflow and name the artifact ids it
|
||||
reads.
|
||||
|
||||
That affects any engagement that runs as a chain of workflows. CP1's Agentic Landing Zone runs as seven:
|
||||
|
||||
1. Discovery
|
||||
2. Assessment
|
||||
3. Strategy
|
||||
4. Business Case
|
||||
5. Target Architecture
|
||||
6. Wave Planning
|
||||
7. Landing Zone
|
||||
|
||||
Each one consumes the approved output of the one before.
|
||||
|
||||
How artifacts are registered, and how `select_artifacts` reads them, is covered in
|
||||
[artifacts-step-within-a-run](../artifacts-step-within-a-run/report-2026-10-29.md). This report covers what's missing
|
||||
around that: running or chaining workflows, and deriving inputs.
|
||||
|
||||
## What happens
|
||||
|
||||
| # | Behaviour | Code (`packages/agent-builder-temporal-workers/src/temporal/`) |
|
||||
|---|---|---|
|
||||
| 1 | The only step types are `sequential`, `parallel` and `conditional`, and every step must name an `agent`. No step type runs a workflow | `workflows/workflow_template_validator.py:14-16`, `:89-108` |
|
||||
| 2 | No step activity starts a workflow. The fallback activity map holds agent calls, file parsing, `combine_inputs`, `format_for_ui` and domain activities | `activity_mapper.py:131-154` |
|
||||
| 3 | Only two built-in pipelines start child workflows, and they're hard-coded. Designer-built and imported workflows can't do it | `workflows/language_ingestion_pipeline_workflow.py:122`, `workflows/business_rules_workflow.py:120` |
|
||||
| 4 | When a run completes, nothing starts another one. The API only broadcasts `execution_complete` over Socket.IO | `agent-builder-api/src/services/execution_service.py:437` |
|
||||
| 5 | Inputs can be derived only from inside the run. Sources are `from_steps` (dot paths), `from_workflow` (run inputs), run inputs named `<step_id>_<input>` (auto-mapped), `static` values, `combine_inputs`, and `==` / `!=` conditions over `steps.*` and `inputs.*`. There's no lookup or expression language, so nothing can resolve "the latest approved run of workflow A" | `workflows/generic_workflow_executor.py:194-210`, `:496-517`, `:600-660` |
|
||||
|
||||
No sub-workflow concept exists in `agent-builder-api/src` or `agent-builder-ui/src`. I searched for `sub_workflow`,
|
||||
`subworkflow` and `nested`.
|
||||
|
||||
## Impact on the CP1 workflows
|
||||
|
||||
- **Assessment can't call Discovery or follow it automatically.** Each Feature runs on its own, and a person starts the
|
||||
next one.
|
||||
- **The artifact ids have to be entered by hand.** The Assessment run can take them as the run input
|
||||
`<step_id>_artifact_ids` (behaviour 5), so the workflow needn't be edited for each engagement. But someone still has to
|
||||
find the ids of the right Discovery run.
|
||||
- **The platform doesn't check that the previous stage passed.** Outputs are registered whether or not the run failed,
|
||||
the critic failed it, or a person rejected it. Each receiving workflow has to check for itself. Each selected entry
|
||||
carries `metadata.status` and `source_execution_id` (`activities/artifacts_activities.py:468-482`), so the check can be
|
||||
made.
|
||||
|
||||
## Workarounds in use
|
||||
|
||||
1. **Each Feature is its own workflow.** It starts with an Artifacts step, given the exact ids of the previous workflow's
|
||||
artifacts as a run input. Never a name or a workflow id: see
|
||||
[select-artifacts-tenant-scope](../select-artifacts-tenant-scope/report-2026-11-02.md).
|
||||
2. **The receiving workflow checks the previous stage.** Its planner refuses to start unless the previous critic's verdict
|
||||
is `pass` and its approval is recorded.
|
||||
3. **References survive between runs.** Evidence references carry the producing `execution_id`.
|
||||
4. **End-to-end tests run as one workflow.** Two stages can be combined into one workflow and wired with `from_steps`.
|
||||
That gives up rerunning one stage on its own.
|
||||
|
||||
## Questions for the platform team
|
||||
|
||||
1. Is a workflow step planned that runs another workflow as a Temporal child workflow, waits for it, and exposes its
|
||||
outputs to `from_steps`?
|
||||
2. Or is a trigger planned: start workflow B when workflow A completes, passing A's registered artifact ids?
|
||||
3. Could registered artifacts record the run's outcome (completed or failed, approved or rejected), and could
|
||||
`select_artifacts` filter on it?
|
||||
4. Could `select_artifacts` take an `execution_id` plus step ids, instead of raw artifact ids?
|
||||
|
||||
## Evidence
|
||||
|
||||
Code reading of `modernization-platform` @ `c89aedcb9`. Nothing was run against the platform, and nothing was changed.
|
||||
Reference in New Issue
Block a user