Files
cp1-agents/docs/deployment.md
Jonathan Boniface 66c32745aa
Some checks failed
check agents / check (push) Failing after 14s
F02: Assessment report and package (ADR 0014, 0015); register agents in KAB
- Report (cp1-assessment-report-agent 0.2.0, step s7): a model writes the
  Assessment report from facts worked out in code (register, gaps, scorecard,
  rule traces, checks, and what each application is from the Discovery
  package); code sets the verdict and renders the pages. The
  disposition_signoff review point moves from the critic to the report.
- Package (cp1-assessment-package-agent 0.1.1, step s8): after sign-off, every
  step's output in one package, with a register, a gaps register, the
  sign-off record, lineage and a sha256 per part, in code. Page step s9.
- scripts/kab_register_agents.py: registers deployed agents in KAB's agent
  catalog (an in-place workflow update doesn't).
- Runs exec_1b1257380df7 (014) and exec_374d5d577077 (015) completed on
  crucible-kab-dev: critic pass, report figures all in its facts, package
  complete with every digest recomputing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:19:10 +00:00

169 lines
11 KiB
Markdown

# Deploying cp1-agents with kustomize
For the platform operator who deploys the CP1 agents and their MCP server to the KAB dev cluster `crucible-kab-dev-aks`. You deploy by hand with kustomize. Kyndryl's agent CI can deploy the same folders unchanged, because each `kubernetes/` folder keeps the CI's placeholders.
Run every command from the repository root. The Feature folder names contain spaces and a colon, so quote them.
## What gets deployed
| Component | Pattern | Folder | Namespace | Port |
|---|---|---|---|---|
| `cp1-alz-discovery-orchestrator-agent` | Sub-agents | `sub-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 |
| `cp1-discovery-planner-agent` … `cp1-discovery-critic-agent` (8 agents) | Individual agents | `individual-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 |
| `alz-discovery-sample` (sample MCP server, Meridian estate; both patterns use it) | — | `mcp-servers/alz-discovery-sample/` | `mcp-servers` | 8110 |
Images are `cruciblecoredevacr.azurecr.io/kab/cp1/<component>:<version>`. An agent's version comes from its `agent.yaml`; the MCP server's comes from `mcp-servers/alz-discovery-sample.server.yaml`.
The target, registry and gateway addresses are in [`deploy/crucible-kab-dev.yaml`](../deploy/crucible-kab-dev.yaml). `scripts/render_k8s.py` finds each agent by name under `sub-agents/` or `individual-agents/`.
The eight individual agents keep the label `repo: cp1-agents-individual`. The label is part of each Deployment's selector, which can't be changed in place, and they were first deployed from that repository.
Keys are never in a file. Each component reads its keys from a Kubernetes Secret that you create by hand:
| Secret | Namespace | Key | Who creates it |
|---|---|---|---|
| `agent-builder-secrets` | `agents` | `litellm-master-key` | The platform; it already exists |
| `<agent>-secrets` (one per agent) | `agents` | `MCP_AGENT_API_KEY` | You, after KAB issues the agent's key |
| `alz-discovery-sample-secrets` | `mcp-servers` | `MCP_REGISTRATION_API_KEY` | You, with a new random key |
## Before you start
- **kubectl context:** `crucible-kab-dev-aks`. On this workstation it's in the Windows kubeconfig: `export KUBECONFIG=/mnt/c/Users/<you>/.kube/config`.
- **Registry access:** you can push to `cruciblecoredevacr` (`az acr login -n cruciblecoredevacr`).
- **Models:** the cluster's LiteLLM must serve `gpt-5.6-sol` and `claude-sonnet-5-5` (used by the critic).
- **MCP server registration works.** As of 2026-10-29 it fails with a 500 in the MCP gateway; see [platform-issues/mcp-gateway-default-tenant](platform-issues/mcp-gateway-default-tenant/report-2026-10-29.md).
- **The checks pass:**
```sh
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | xargs -0 -n1 dirname -z \
| xargs -0 python3 scripts/check_agents.py --runtime-image cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7
(cd "sub-agents/ALZ-F01: Discovery Workflow Agent" && python3 scripts/dry_run.py cp1-alz-discovery-orchestrator-agent "$(mktemp -d)")
```
## 1. Build and push the images
```sh
az acr login --name cruciblecoredevacr
R=cruciblecoredevacr.azurecr.io/kab/cp1
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | while IFS= read -r -d '' y; do
d=$(dirname "$y"); a=$(basename "$d"); v=$(sed -n 's/^version: "\{0,1\}\([^"#]*\)"\{0,1\}.*/\1/p' "$y" | tr -d ' ')
podman build -t "$R/$a:$v" "$d" && podman push "$R/$a:$v"
done
podman build -t $R/alz-discovery-sample:0.1.0 mcp-servers/alz-discovery-sample && podman push $R/alz-discovery-sample:0.1.0
```
Every agent's Dockerfile builds on `cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7`. A released version is never pushed twice. For a new build, bump `version` in `agent.yaml` and add an entry to `RELEASE_NOTES.md`.
## 2. Render the manifests
```sh
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml # every component
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml <component> # or just one
```
This writes `build/crucible-kab-dev/<component>/` (git-ignored) with every placeholder filled. It leaves out two files:
- `secretnames.yaml`, so a deploy can't empty your Secrets;
- `namespace.yaml`, so the shared namespaces are never relabelled or deleted.
## 3. Deploy the MCP server, then register it
```sh
read -rs REG_KEY < <(openssl rand -hex 32; echo)
kubectl --context crucible-kab-dev-aks -n mcp-servers create secret generic alz-discovery-sample-secrets \
--from-literal=MCP_REGISTRATION_API_KEY="$REG_KEY"
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/alz-discovery-sample
kubectl --context crucible-kab-dev-aks -n mcp-servers rollout status deploy/alz-discovery-sample
```
Then register the server in KAB:
1. Use the address `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` and the key `$REG_KEY`.
2. Register it under each of the nine logical names the agents use: `alz-scope`, `alz-estate`, `alz-cloud`, `alz-cmdb`, `alz-inventory`, `alz-schema`, `alz-dependency`, `alz-network` and `alz-evidence`.
3. Run `unset REG_KEY`.
## 4. Register each agent, then deploy it
In KAB, create the agent at `http://<agent>.agents.svc.cluster.local:8080`, or, once it is deployed, register it from
the command line: `python3 scripts/kab_register_agents.py <agent> ... --apply` (KAB's `discover-and-register`, which reads
the agent card). A workflow that uses an unregistered agent fails to validate, and `kab_import_workflow.py --update`
doesn't register agents. An agent with MCP tools also needs its key. KAB issues it, and you store it without writing it
anywhere:
```sh
read -rs MCP_KEY && kubectl --context crucible-kab-dev-aks -n agents create secret generic <agent>-secrets \
--from-literal=MCP_AGENT_API_KEY="$MCP_KEY"; unset MCP_KEY
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/<agent>
kubectl --context crucible-kab-dev-aks -n agents rollout status deploy/<agent>
```
An agent without its key still starts and serves its agent card, but every request fails with `MCP_AGENT_API_KEY is not set`.
## 5. Test, then import the workflows
```sh
kubectl --context crucible-kab-dev-aks -n agents port-forward svc/<agent> 18080:8080 &
curl -s localhost:18080/.well-known/agent-card.json
python -m kyndryl_agent_runtime.eval --url http://localhost:18080 --cases "<Feature folder>/<agent>/eval/cases.yaml"
```
Each Feature has its own workflow export in `<Feature folder>/workflows/`. Import it with your KAB session token in `~/.kab_token`. Importing needs the Editor role.
```sh
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> # validate
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply # import as a draft
python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply
```
## Deploying the hyperscaler MCP servers
`azure-discovery`, `aws-discovery` and `gcp-discovery` deploy as in step 3, each with its own `<server>-secrets` Secret and registration. Their rendered Dockerfile needs a login to `kyndryl.jfrog.io`, or a local base passed with `--build-arg BASE_IMAGE=<image>`. Until that's available, build on the public Python image: `podman build -f mcp-servers/_common/Dockerfile.python -t <image> mcp-servers/<server>`. This image isn't hardened, so don't use it for customer data.
Then store each connection in KAB under **Credentials**, per project, as described in
[customer-cloud-access.md](customer-cloud-access.md). Never tenant-wide. Well-Architected guidance uses the same connection.
### Starting a run from a context file
Every run starts from a YAML context file. Its `workflow_inputs:` become the run's inputs (validated before the run
starts). The script generates `run_id` for every run (UTC time plus a random suffix) and `plan_id` = `PLAN-<run_id>`,
and adds them as a `generated:` section; a file that sets `run_id` itself is refused. The file also declares the
`customer:` (business details only, no personal data) and the engagement `plan:`. The whole file is uploaded to the workflow's upload step, so the planner, the inventory agent and the
report also read its `context:` section. The template is
`individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/run-context.template.yaml`; the live trials have their
own in `workflows/cases/`. Never put a credential in one.
```sh
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
--context "individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/gcp-live/context.yaml" --dry-run
```
### Storing the connections from a script
`scripts/kab_store_connections.py` stores the discovery servers' connections, per project, through the platform's
Credentials API, without the UI. It finds each server in the MCP gateway, checks the connection against the
server's declared fields, and tests it with `connections:verify`. Only with `--apply` does it store the connection
(`PUT /api/v1/secrets`, `secretType` `mcp`, format `connection`, create-only unless `--replace`).
Credential values never appear in its output, its arguments or a file. They come from the environment
(`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`), from a service account key file kept outside the
repository, or from `az` / `gcloud` access tokens. A token lasts about an hour and carries your own rights.
```sh
python3 scripts/kab_store_connections.py --api https://<platform host> --tenant <tenant id> --project-id <project id> \
--azure-scope-root subscriptions/<id> --azure-credential service-principal \
--gcp-scope-root projects/<id> --gcp-credential key-file:$HOME/keys/<read-only-sa>.json # test only
# ... then the same command with --apply
```
It needs a KAB token (`$KAB_TOKEN` or `~/.kab_token`, `--token-file` to change) and the Administrator role. API keys work only on `/api/external/` routes, so they can't be used here. The
servers must already be registered in that platform's MCP gateway, at an address its gateway can reach.
## Rolling back and removing
```sh
kubectl --context crucible-kab-dev-aks -n agents rollout undo deploy/<agent> # the previous version
kubectl --context crucible-kab-dev-aks delete -k build/crucible-kab-dev/<agent> # remove it entirely
```
A rendered build holds only the component's Deployment, Service and ConfigMap. **Never `kubectl delete -k` an agent's own `kubernetes/` folder.** That folder lists the shared namespace, and the delete would remove the namespace together with every other agent in it.