# Deploying cp1-agents with kustomize For the platform operator who deploys the CP1 agents and their MCP server to the KAB dev cluster `crucible-kab-dev-aks`. You deploy by hand with kustomize. Kyndryl's agent CI can deploy the same folders unchanged, because each `kubernetes/` folder keeps the CI's placeholders. Run every command from the repository root. The Feature folder names contain spaces and a colon, so quote them. ## What gets deployed | Component | Pattern | Folder | Namespace | Port | |---|---|---|---|---| | `cp1-alz-discovery-orchestrator-agent` | Sub-agents | `sub-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 | | `cp1-discovery-planner-agent` … `cp1-discovery-critic-agent` (8 agents) | Individual agents | `individual-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 | | `alz-discovery-sample` (sample MCP server, Meridian estate; both patterns use it) | — | `mcp-servers/alz-discovery-sample/` | `mcp-servers` | 8110 | Images are `cruciblecoredevacr.azurecr.io/kab/cp1/:`. An agent's version comes from its `agent.yaml`; the MCP server's comes from `mcp-servers/alz-discovery-sample.server.yaml`. The target, registry and gateway addresses are in [`deploy/crucible-kab-dev.yaml`](../deploy/crucible-kab-dev.yaml). `scripts/render_k8s.py` finds each agent by name under `sub-agents/` or `individual-agents/`. The eight individual agents keep the label `repo: cp1-agents-individual`. The label is part of each Deployment's selector, which can't be changed in place, and they were first deployed from that repository. Keys are never in a file. Each component reads its keys from a Kubernetes Secret that you create by hand: | Secret | Namespace | Key | Who creates it | |---|---|---|---| | `agent-builder-secrets` | `agents` | `litellm-master-key` | The platform; it already exists | | `-secrets` (one per agent) | `agents` | `MCP_AGENT_API_KEY` | You, after KAB issues the agent's key | | `alz-discovery-sample-secrets` | `mcp-servers` | `MCP_REGISTRATION_API_KEY` | You, with a new random key | ## Before you start - **kubectl context:** `crucible-kab-dev-aks`. On this workstation it's in the Windows kubeconfig: `export KUBECONFIG=/mnt/c/Users//.kube/config`. - **Registry access:** you can push to `cruciblecoredevacr` (`az acr login -n cruciblecoredevacr`). - **Models:** the cluster's LiteLLM must serve `gpt-5.6-sol` and `claude-sonnet-5-5` (used by the critic). - **MCP server registration works.** As of 2026-10-29 it fails with a 500 in the MCP gateway; see [platform-issues/mcp-gateway-default-tenant](platform-issues/mcp-gateway-default-tenant/report-2026-10-29.md). - **The checks pass:** ```sh find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | xargs -0 -n1 dirname -z \ | xargs -0 python3 scripts/check_agents.py --runtime-image cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7 (cd "sub-agents/ALZ-F01: Discovery Workflow Agent" && python3 scripts/dry_run.py cp1-alz-discovery-orchestrator-agent "$(mktemp -d)") ``` ## 1. Build and push the images ```sh az acr login --name cruciblecoredevacr R=cruciblecoredevacr.azurecr.io/kab/cp1 find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | while IFS= read -r -d '' y; do d=$(dirname "$y"); a=$(basename "$d"); v=$(sed -n 's/^version: "\{0,1\}\([^"#]*\)"\{0,1\}.*/\1/p' "$y" | tr -d ' ') podman build -t "$R/$a:$v" "$d" && podman push "$R/$a:$v" done podman build -t $R/alz-discovery-sample:0.1.0 mcp-servers/alz-discovery-sample && podman push $R/alz-discovery-sample:0.1.0 ``` Every agent's Dockerfile builds on `cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7`. A released version is never pushed twice. For a new build, bump `version` in `agent.yaml` and add an entry to `RELEASE_NOTES.md`. ## 2. Render the manifests ```sh python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml # every component python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml # or just one ``` This writes `build/crucible-kab-dev//` (git-ignored) with every placeholder filled. It leaves out two files: - `secretnames.yaml`, so a deploy can't empty your Secrets; - `namespace.yaml`, so the shared namespaces are never relabelled or deleted. ## 3. Deploy the MCP server, then register it ```sh read -rs REG_KEY < <(openssl rand -hex 32; echo) kubectl --context crucible-kab-dev-aks -n mcp-servers create secret generic alz-discovery-sample-secrets \ --from-literal=MCP_REGISTRATION_API_KEY="$REG_KEY" kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/alz-discovery-sample kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/alz-discovery-sample kubectl --context crucible-kab-dev-aks -n mcp-servers rollout status deploy/alz-discovery-sample ``` Then register the server in KAB: 1. Use the address `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` and the key `$REG_KEY`. 2. Register it under each of the nine logical names the agents use: `alz-scope`, `alz-estate`, `alz-cloud`, `alz-cmdb`, `alz-inventory`, `alz-schema`, `alz-dependency`, `alz-network` and `alz-evidence`. 3. Run `unset REG_KEY`. ## 4. Register each agent, then deploy it In KAB, create the agent at `http://.agents.svc.cluster.local:8080`. KAB issues the agent's MCP key, which you store without writing it anywhere: ```sh read -rs MCP_KEY && kubectl --context crucible-kab-dev-aks -n agents create secret generic -secrets \ --from-literal=MCP_AGENT_API_KEY="$MCP_KEY"; unset MCP_KEY kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/ kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/ kubectl --context crucible-kab-dev-aks -n agents rollout status deploy/ ``` An agent without its key still starts and serves its agent card, but every request fails with `MCP_AGENT_API_KEY is not set`. ## 5. Test, then import the workflows ```sh kubectl --context crucible-kab-dev-aks -n agents port-forward svc/ 18080:8080 & curl -s localhost:18080/.well-known/agent-card.json python -m kyndryl_agent_runtime.eval --url http://localhost:18080 --cases "//eval/cases.yaml" ``` Each Feature has its own workflow export in `/workflows/`. Import it with your KAB session token in `~/.kab_token`. Importing needs the Editor role. ```sh python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id # validate python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id --apply # import as a draft python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Workflow Agent" --project-id --apply ``` ## Rolling back and removing ```sh kubectl --context crucible-kab-dev-aks -n agents rollout undo deploy/ # the previous version kubectl --context crucible-kab-dev-aks delete -k build/crucible-kab-dev/ # remove it entirely ``` A rendered build holds only the component's Deployment, Service and ConfigMap. **Never `kubectl delete -k` an agent's own `kubernetes/` folder.** That folder lists the shared namespace, and the delete would remove the namespace together with every other agent in it.