Files
cp1-agents/docs/deployment.md
Jonathan Boniface f659564208 Platform scripts, MCP servers, deployment and docs for the KAB runs
- Scripts to run workflows from a YAML context or run inputs (kab_run_workflow.py, with
  --input name=@file), store per-project discovery connections, bootstrap agent MCP keys,
  refresh the KAB token, publish reports and build estate snapshots.
- Hyperscaler discovery servers extended in place; cloud-knowledge folded into them.
- deploy/crucible-kab-dev.yaml: the Discovery package agent and the Assessment agents.
- Platform issue write-ups and current-wiring notes under docs/platform-issues/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

10 KiB

Deploying cp1-agents with kustomize

For the platform operator who deploys the CP1 agents and their MCP server to the KAB dev cluster crucible-kab-dev-aks. You deploy by hand with kustomize. Kyndryl's agent CI can deploy the same folders unchanged, because each kubernetes/ folder keeps the CI's placeholders.

Run every command from the repository root. The Feature folder names contain spaces and a colon, so quote them.

What gets deployed

Component Pattern Folder Namespace Port
cp1-alz-discovery-orchestrator-agent Sub-agents sub-agents/ALZ-F01: Discovery Workflow Agent/ agents 8080
cp1-discovery-planner-agent … cp1-discovery-critic-agent (8 agents) Individual agents individual-agents/ALZ-F01: Discovery Workflow Agent/ agents 8080
alz-discovery-sample (sample MCP server, Meridian estate; both patterns use it) — mcp-servers/alz-discovery-sample/ mcp-servers 8110

Images are cruciblecoredevacr.azurecr.io/kab/cp1/<component>:<version>. An agent's version comes from its agent.yaml; the MCP server's comes from mcp-servers/alz-discovery-sample.server.yaml.

The target, registry and gateway addresses are in deploy/crucible-kab-dev.yaml. scripts/render_k8s.py finds each agent by name under sub-agents/ or individual-agents/.

The eight individual agents keep the label repo: cp1-agents-individual. The label is part of each Deployment's selector, which can't be changed in place, and they were first deployed from that repository.

Keys are never in a file. Each component reads its keys from a Kubernetes Secret that you create by hand:

Secret Namespace Key Who creates it
agent-builder-secrets agents litellm-master-key The platform; it already exists
<agent>-secrets (one per agent) agents MCP_AGENT_API_KEY You, after KAB issues the agent's key
alz-discovery-sample-secrets mcp-servers MCP_REGISTRATION_API_KEY You, with a new random key

Before you start

  • kubectl context: crucible-kab-dev-aks. On this workstation it's in the Windows kubeconfig: export KUBECONFIG=/mnt/c/Users/<you>/.kube/config.

  • Registry access: you can push to cruciblecoredevacr (az acr login -n cruciblecoredevacr).

  • Models: the cluster's LiteLLM must serve gpt-5.6-sol and claude-sonnet-5-5 (used by the critic).

  • MCP server registration works. As of 2026-10-29 it fails with a 500 in the MCP gateway; see platform-issues/mcp-gateway-default-tenant.

  • The checks pass:

    find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | xargs -0 -n1 dirname -z \
      | xargs -0 python3 scripts/check_agents.py --runtime-image cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7
    (cd "sub-agents/ALZ-F01: Discovery Workflow Agent" && python3 scripts/dry_run.py cp1-alz-discovery-orchestrator-agent "$(mktemp -d)")
    

1. Build and push the images

az acr login --name cruciblecoredevacr
R=cruciblecoredevacr.azurecr.io/kab/cp1
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | while IFS= read -r -d '' y; do
  d=$(dirname "$y"); a=$(basename "$d"); v=$(sed -n 's/^version: "\{0,1\}\([^"#]*\)"\{0,1\}.*/\1/p' "$y" | tr -d ' ')
  podman build -t "$R/$a:$v" "$d" && podman push "$R/$a:$v"
done
podman build -t $R/alz-discovery-sample:0.1.0 mcp-servers/alz-discovery-sample && podman push $R/alz-discovery-sample:0.1.0

Every agent's Dockerfile builds on cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7. A released version is never pushed twice. For a new build, bump version in agent.yaml and add an entry to RELEASE_NOTES.md.

2. Render the manifests

python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml            # every component
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml <component> # or just one

This writes build/crucible-kab-dev/<component>/ (git-ignored) with every placeholder filled. It leaves out two files:

  • secretnames.yaml, so a deploy can't empty your Secrets;
  • namespace.yaml, so the shared namespaces are never relabelled or deleted.

3. Deploy the MCP server, then register it

read -rs REG_KEY < <(openssl rand -hex 32; echo)
kubectl --context crucible-kab-dev-aks -n mcp-servers create secret generic alz-discovery-sample-secrets \
  --from-literal=MCP_REGISTRATION_API_KEY="$REG_KEY"
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks -n mcp-servers rollout status deploy/alz-discovery-sample

Then register the server in KAB:

  1. Use the address http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110 and the key $REG_KEY.
  2. Register it under each of the nine logical names the agents use: alz-scope, alz-estate, alz-cloud, alz-cmdb, alz-inventory, alz-schema, alz-dependency, alz-network and alz-evidence.
  3. Run unset REG_KEY.

4. Register each agent, then deploy it

In KAB, create the agent at http://<agent>.agents.svc.cluster.local:8080. KAB issues the agent's MCP key, which you store without writing it anywhere:

read -rs MCP_KEY && kubectl --context crucible-kab-dev-aks -n agents create secret generic <agent>-secrets \
  --from-literal=MCP_AGENT_API_KEY="$MCP_KEY"; unset MCP_KEY
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks -n agents rollout status deploy/<agent>

An agent without its key still starts and serves its agent card, but every request fails with MCP_AGENT_API_KEY is not set.

5. Test, then import the workflows

kubectl --context crucible-kab-dev-aks -n agents port-forward svc/<agent> 18080:8080 &
curl -s localhost:18080/.well-known/agent-card.json
python -m kyndryl_agent_runtime.eval --url http://localhost:18080 --cases "<Feature folder>/<agent>/eval/cases.yaml"

Each Feature has its own workflow export in <Feature folder>/workflows/. Import it with your KAB session token in ~/.kab_token. Importing needs the Editor role.

python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id>          # validate
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply  # import as a draft
python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply

Deploying the hyperscaler MCP servers

azure-discovery, aws-discovery and gcp-discovery deploy as in step 3, each with its own <server>-secrets Secret and registration. Their rendered Dockerfile needs a login to kyndryl.jfrog.io, or a local base passed with --build-arg BASE_IMAGE=<image>. Until that's available, build on the public Python image: podman build -f mcp-servers/_common/Dockerfile.python -t <image> mcp-servers/<server>. This image isn't hardened, so don't use it for customer data.

Then store each connection in KAB under Credentials, per project, as described in customer-cloud-access.md. Never tenant-wide. Well-Architected guidance uses the same connection.

Starting a run from a context file

Every run starts from a YAML context file. Its workflow_inputs: become the run's inputs (validated before the run starts). The script generates run_id for every run (UTC time plus a random suffix) and plan_id = PLAN-<run_id>, and adds them as a generated: section; a file that sets run_id itself is refused. The file also declares the customer: (business details only, no personal data) and the engagement plan:. The whole file is uploaded to the workflow's upload step, so the planner, the inventory agent and the report also read its context: section. The template is individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/run-context.template.yaml; the live trials have their own in workflows/cases/. Never put a credential in one.

python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
  --context "individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/gcp-live/context.yaml" --dry-run

Storing the connections from a script

scripts/kab_store_connections.py stores the discovery servers' connections, per project, through the platform's Credentials API, without the UI. It finds each server in the MCP gateway, checks the connection against the server's declared fields, and tests it with connections:verify. Only with --apply does it store the connection (PUT /api/v1/secrets, secretType mcp, format connection, create-only unless --replace).

Credential values never appear in its output, its arguments or a file. They come from the environment (AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET), from a service account key file kept outside the repository, or from az / gcloud access tokens. A token lasts about an hour and carries your own rights.

python3 scripts/kab_store_connections.py --api https://<platform host> --tenant <tenant id> --project-id <project id> \
  --azure-scope-root subscriptions/<id> --azure-credential service-principal \
  --gcp-scope-root projects/<id> --gcp-credential key-file:$HOME/keys/<read-only-sa>.json        # test only
# ... then the same command with --apply

It needs a KAB token ($KAB_TOKEN or ~/.kab_token, --token-file to change) and the Administrator role. API keys work only on /api/external/ routes, so they can't be used here. The servers must already be registered in that platform's MCP gateway, at an address its gateway can reach.

Rolling back and removing

kubectl --context crucible-kab-dev-aks -n agents rollout undo deploy/<agent>          # the previous version
kubectl --context crucible-kab-dev-aks delete -k build/crucible-kab-dev/<agent>       # remove it entirely

A rendered build holds only the component's Deployment, Service and ConfigMap. Never kubectl delete -k an agent's own kubernetes/ folder. That folder lists the shared namespace, and the delete would remove the namespace together with every other agent in it.