Files
cp1-agents/docs/deployment.md
Jonathan Boniface 416c66ced4
All checks were successful
check agents / check (push) Successful in 10s
Add CP1 Discovery agents in both patterns
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight
individual declarative agents (individual-agents/), with their KAB workflow
exports. Shared check, render and import scripts, the alz-discovery-sample
MCP server, the crucible-kab-dev deploy profile, deployment docs, platform
issue reports and the source material. Agent steps time out at 20 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:25:57 +00:00

7.2 KiB

Deploying cp1-agents with kustomize

For the platform operator who deploys the CP1 agents and their MCP server to the KAB dev cluster crucible-kab-dev-aks. You deploy by hand with kustomize. Kyndryl's agent CI can deploy the same folders unchanged, because each kubernetes/ folder keeps the CI's placeholders.

Run every command from the repository root. The Feature folder names contain spaces and a colon, so quote them.

What gets deployed

Component Pattern Folder Namespace Port
cp1-alz-discovery-orchestrator-agent Sub-agents sub-agents/ALZ-F01: Discovery Workflow Agent/ agents 8080
cp1-discovery-planner-agent … cp1-discovery-critic-agent (8 agents) Individual agents individual-agents/ALZ-F01: Discovery Workflow Agent/ agents 8080
alz-discovery-sample (sample MCP server, Meridian estate; both patterns use it) — mcp-servers/alz-discovery-sample/ mcp-servers 8110

Images are cruciblecoredevacr.azurecr.io/kab/cp1/<component>:<version>. An agent's version comes from its agent.yaml; the MCP server's comes from mcp-servers/alz-discovery-sample.server.yaml.

The target, registry and gateway addresses are in deploy/crucible-kab-dev.yaml. scripts/render_k8s.py finds each agent by name under sub-agents/ or individual-agents/.

The eight individual agents keep the label repo: cp1-agents-individual. The label is part of each Deployment's selector, which can't be changed in place, and they were first deployed from that repository.

Keys are never in a file. Each component reads its keys from a Kubernetes Secret that you create by hand:

Secret Namespace Key Who creates it
agent-builder-secrets agents litellm-master-key The platform; it already exists
<agent>-secrets (one per agent) agents MCP_AGENT_API_KEY You, after KAB issues the agent's key
alz-discovery-sample-secrets mcp-servers MCP_REGISTRATION_API_KEY You, with a new random key

Before you start

  • kubectl context: crucible-kab-dev-aks. On this workstation it's in the Windows kubeconfig: export KUBECONFIG=/mnt/c/Users/<you>/.kube/config.

  • Registry access: you can push to cruciblecoredevacr (az acr login -n cruciblecoredevacr).

  • Models: the cluster's LiteLLM must serve gpt-5.6-sol and claude-sonnet-5-5 (used by the critic).

  • MCP server registration works. As of 2026-10-29 it fails with a 500 in the MCP gateway; see platform-issues/mcp-gateway-default-tenant.

  • The checks pass:

    find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | xargs -0 -n1 dirname -z \
      | xargs -0 python3 scripts/check_agents.py --runtime-image cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7
    (cd "sub-agents/ALZ-F01: Discovery Workflow Agent" && python3 scripts/dry_run.py cp1-alz-discovery-orchestrator-agent "$(mktemp -d)")
    

1. Build and push the images

az acr login --name cruciblecoredevacr
R=cruciblecoredevacr.azurecr.io/kab/cp1
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | while IFS= read -r -d '' y; do
  d=$(dirname "$y"); a=$(basename "$d"); v=$(sed -n 's/^version: "\{0,1\}\([^"#]*\)"\{0,1\}.*/\1/p' "$y" | tr -d ' ')
  podman build -t "$R/$a:$v" "$d" && podman push "$R/$a:$v"
done
podman build -t $R/alz-discovery-sample:0.1.0 mcp-servers/alz-discovery-sample && podman push $R/alz-discovery-sample:0.1.0

Every agent's Dockerfile builds on cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7. A released version is never pushed twice. For a new build, bump version in agent.yaml and add an entry to RELEASE_NOTES.md.

2. Render the manifests

python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml            # every component
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml <component> # or just one

This writes build/crucible-kab-dev/<component>/ (git-ignored) with every placeholder filled. It leaves out two files:

  • secretnames.yaml, so a deploy can't empty your Secrets;
  • namespace.yaml, so the shared namespaces are never relabelled or deleted.

3. Deploy the MCP server, then register it

read -rs REG_KEY < <(openssl rand -hex 32; echo)
kubectl --context crucible-kab-dev-aks -n mcp-servers create secret generic alz-discovery-sample-secrets \
  --from-literal=MCP_REGISTRATION_API_KEY="$REG_KEY"
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks -n mcp-servers rollout status deploy/alz-discovery-sample

Then register the server in KAB:

  1. Use the address http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110 and the key $REG_KEY.
  2. Register it under each of the nine logical names the agents use: alz-scope, alz-estate, alz-cloud, alz-cmdb, alz-inventory, alz-schema, alz-dependency, alz-network and alz-evidence.
  3. Run unset REG_KEY.

4. Register each agent, then deploy it

In KAB, create the agent at http://<agent>.agents.svc.cluster.local:8080. KAB issues the agent's MCP key, which you store without writing it anywhere:

read -rs MCP_KEY && kubectl --context crucible-kab-dev-aks -n agents create secret generic <agent>-secrets \
  --from-literal=MCP_AGENT_API_KEY="$MCP_KEY"; unset MCP_KEY
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks -n agents rollout status deploy/<agent>

An agent without its key still starts and serves its agent card, but every request fails with MCP_AGENT_API_KEY is not set.

5. Test, then import the workflows

kubectl --context crucible-kab-dev-aks -n agents port-forward svc/<agent> 18080:8080 &
curl -s localhost:18080/.well-known/agent-card.json
python -m kyndryl_agent_runtime.eval --url http://localhost:18080 --cases "<Feature folder>/<agent>/eval/cases.yaml"

Each Feature has its own workflow export in <Feature folder>/workflows/. Import it with your KAB session token in ~/.kab_token. Importing needs the Editor role.

python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id>          # validate
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply  # import as a draft
python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply

Rolling back and removing

kubectl --context crucible-kab-dev-aks -n agents rollout undo deploy/<agent>          # the previous version
kubectl --context crucible-kab-dev-aks delete -k build/crucible-kab-dev/<agent>       # remove it entirely

A rendered build holds only the component's Deployment, Service and ConfigMap. Never kubectl delete -k an agent's own kubernetes/ folder. That folder lists the shared namespace, and the delete would remove the namespace together with every other agent in it.