Some checks failed
check agents / check (push) Failing after 14s
- Report (cp1-assessment-report-agent 0.2.0, step s7): a model writes the Assessment report from facts worked out in code (register, gaps, scorecard, rule traces, checks, and what each application is from the Discovery package); code sets the verdict and renders the pages. The disposition_signoff review point moves from the critic to the report. - Package (cp1-assessment-package-agent 0.1.1, step s8): after sign-off, every step's output in one package, with a register, a gaps register, the sign-off record, lineage and a sha256 per part, in code. Page step s9. - scripts/kab_register_agents.py: registers deployed agents in KAB's agent catalog (an in-place workflow update doesn't). - Runs exec_1b1257380df7 (014) and exec_374d5d577077 (015) completed on crucible-kab-dev: critic pass, report figures all in its facts, package complete with every digest recomputing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
169 lines
11 KiB
Markdown
169 lines
11 KiB
Markdown
# Deploying cp1-agents with kustomize
|
|
|
|
For the platform operator who deploys the CP1 agents and their MCP server to the KAB dev cluster `crucible-kab-dev-aks`. You deploy by hand with kustomize. Kyndryl's agent CI can deploy the same folders unchanged, because each `kubernetes/` folder keeps the CI's placeholders.
|
|
|
|
Run every command from the repository root. The Feature folder names contain spaces and a colon, so quote them.
|
|
|
|
## What gets deployed
|
|
|
|
| Component | Pattern | Folder | Namespace | Port |
|
|
|---|---|---|---|---|
|
|
| `cp1-alz-discovery-orchestrator-agent` | Sub-agents | `sub-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 |
|
|
| `cp1-discovery-planner-agent` … `cp1-discovery-critic-agent` (8 agents) | Individual agents | `individual-agents/ALZ-F01: Discovery Workflow Agent/` | `agents` | 8080 |
|
|
| `alz-discovery-sample` (sample MCP server, Meridian estate; both patterns use it) | — | `mcp-servers/alz-discovery-sample/` | `mcp-servers` | 8110 |
|
|
|
|
Images are `cruciblecoredevacr.azurecr.io/kab/cp1/<component>:<version>`. An agent's version comes from its `agent.yaml`; the MCP server's comes from `mcp-servers/alz-discovery-sample.server.yaml`.
|
|
|
|
The target, registry and gateway addresses are in [`deploy/crucible-kab-dev.yaml`](../deploy/crucible-kab-dev.yaml). `scripts/render_k8s.py` finds each agent by name under `sub-agents/` or `individual-agents/`.
|
|
|
|
The eight individual agents keep the label `repo: cp1-agents-individual`. The label is part of each Deployment's selector, which can't be changed in place, and they were first deployed from that repository.
|
|
|
|
Keys are never in a file. Each component reads its keys from a Kubernetes Secret that you create by hand:
|
|
|
|
| Secret | Namespace | Key | Who creates it |
|
|
|---|---|---|---|
|
|
| `agent-builder-secrets` | `agents` | `litellm-master-key` | The platform; it already exists |
|
|
| `<agent>-secrets` (one per agent) | `agents` | `MCP_AGENT_API_KEY` | You, after KAB issues the agent's key |
|
|
| `alz-discovery-sample-secrets` | `mcp-servers` | `MCP_REGISTRATION_API_KEY` | You, with a new random key |
|
|
|
|
## Before you start
|
|
|
|
- **kubectl context:** `crucible-kab-dev-aks`. On this workstation it's in the Windows kubeconfig: `export KUBECONFIG=/mnt/c/Users/<you>/.kube/config`.
|
|
- **Registry access:** you can push to `cruciblecoredevacr` (`az acr login -n cruciblecoredevacr`).
|
|
- **Models:** the cluster's LiteLLM must serve `gpt-5.6-sol` and `claude-sonnet-5-5` (used by the critic).
|
|
- **MCP server registration works.** As of 2026-10-29 it fails with a 500 in the MCP gateway; see [platform-issues/mcp-gateway-default-tenant](platform-issues/mcp-gateway-default-tenant/report-2026-10-29.md).
|
|
- **The checks pass:**
|
|
|
|
```sh
|
|
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | xargs -0 -n1 dirname -z \
|
|
| xargs -0 python3 scripts/check_agents.py --runtime-image cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7
|
|
(cd "sub-agents/ALZ-F01: Discovery Workflow Agent" && python3 scripts/dry_run.py cp1-alz-discovery-orchestrator-agent "$(mktemp -d)")
|
|
```
|
|
|
|
## 1. Build and push the images
|
|
|
|
```sh
|
|
az acr login --name cruciblecoredevacr
|
|
R=cruciblecoredevacr.azurecr.io/kab/cp1
|
|
find sub-agents individual-agents -mindepth 3 -maxdepth 3 -name agent.yaml -print0 | while IFS= read -r -d '' y; do
|
|
d=$(dirname "$y"); a=$(basename "$d"); v=$(sed -n 's/^version: "\{0,1\}\([^"#]*\)"\{0,1\}.*/\1/p' "$y" | tr -d ' ')
|
|
podman build -t "$R/$a:$v" "$d" && podman push "$R/$a:$v"
|
|
done
|
|
podman build -t $R/alz-discovery-sample:0.1.0 mcp-servers/alz-discovery-sample && podman push $R/alz-discovery-sample:0.1.0
|
|
```
|
|
|
|
Every agent's Dockerfile builds on `cruciblecoredevacr.azurecr.io/kab/agent-runtime-adk:0.35.7`. A released version is never pushed twice. For a new build, bump `version` in `agent.yaml` and add an entry to `RELEASE_NOTES.md`.
|
|
|
|
## 2. Render the manifests
|
|
|
|
```sh
|
|
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml # every component
|
|
python3 scripts/render_k8s.py deploy/crucible-kab-dev.yaml <component> # or just one
|
|
```
|
|
|
|
This writes `build/crucible-kab-dev/<component>/` (git-ignored) with every placeholder filled. It leaves out two files:
|
|
- `secretnames.yaml`, so a deploy can't empty your Secrets;
|
|
- `namespace.yaml`, so the shared namespaces are never relabelled or deleted.
|
|
|
|
## 3. Deploy the MCP server, then register it
|
|
|
|
```sh
|
|
read -rs REG_KEY < <(openssl rand -hex 32; echo)
|
|
kubectl --context crucible-kab-dev-aks -n mcp-servers create secret generic alz-discovery-sample-secrets \
|
|
--from-literal=MCP_REGISTRATION_API_KEY="$REG_KEY"
|
|
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/alz-discovery-sample
|
|
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/alz-discovery-sample
|
|
kubectl --context crucible-kab-dev-aks -n mcp-servers rollout status deploy/alz-discovery-sample
|
|
```
|
|
|
|
Then register the server in KAB:
|
|
1. Use the address `http://alz-discovery-sample.mcp-servers.svc.cluster.local:8110` and the key `$REG_KEY`.
|
|
2. Register it under each of the nine logical names the agents use: `alz-scope`, `alz-estate`, `alz-cloud`, `alz-cmdb`, `alz-inventory`, `alz-schema`, `alz-dependency`, `alz-network` and `alz-evidence`.
|
|
3. Run `unset REG_KEY`.
|
|
|
|
## 4. Register each agent, then deploy it
|
|
|
|
In KAB, create the agent at `http://<agent>.agents.svc.cluster.local:8080`, or, once it is deployed, register it from
|
|
the command line: `python3 scripts/kab_register_agents.py <agent> ... --apply` (KAB's `discover-and-register`, which reads
|
|
the agent card). A workflow that uses an unregistered agent fails to validate, and `kab_import_workflow.py --update`
|
|
doesn't register agents. An agent with MCP tools also needs its key. KAB issues it, and you store it without writing it
|
|
anywhere:
|
|
|
|
```sh
|
|
read -rs MCP_KEY && kubectl --context crucible-kab-dev-aks -n agents create secret generic <agent>-secrets \
|
|
--from-literal=MCP_AGENT_API_KEY="$MCP_KEY"; unset MCP_KEY
|
|
kubectl --context crucible-kab-dev-aks diff -k build/crucible-kab-dev/<agent>
|
|
kubectl --context crucible-kab-dev-aks apply -k build/crucible-kab-dev/<agent>
|
|
kubectl --context crucible-kab-dev-aks -n agents rollout status deploy/<agent>
|
|
```
|
|
|
|
An agent without its key still starts and serves its agent card, but every request fails with `MCP_AGENT_API_KEY is not set`.
|
|
|
|
## 5. Test, then import the workflows
|
|
|
|
```sh
|
|
kubectl --context crucible-kab-dev-aks -n agents port-forward svc/<agent> 18080:8080 &
|
|
curl -s localhost:18080/.well-known/agent-card.json
|
|
python -m kyndryl_agent_runtime.eval --url http://localhost:18080 --cases "<Feature folder>/<agent>/eval/cases.yaml"
|
|
```
|
|
|
|
Each Feature has its own workflow export in `<Feature folder>/workflows/`. Import it with your KAB session token in `~/.kab_token`. Importing needs the Editor role.
|
|
|
|
```sh
|
|
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> # validate
|
|
python3 scripts/kab_import_workflow.py "sub-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply # import as a draft
|
|
python3 scripts/kab_import_workflow.py "individual-agents/ALZ-F01: Discovery Workflow Agent" --project-id <id> --apply
|
|
```
|
|
|
|
## Deploying the hyperscaler MCP servers
|
|
|
|
`azure-discovery`, `aws-discovery` and `gcp-discovery` deploy as in step 3, each with its own `<server>-secrets` Secret and registration. Their rendered Dockerfile needs a login to `kyndryl.jfrog.io`, or a local base passed with `--build-arg BASE_IMAGE=<image>`. Until that's available, build on the public Python image: `podman build -f mcp-servers/_common/Dockerfile.python -t <image> mcp-servers/<server>`. This image isn't hardened, so don't use it for customer data.
|
|
|
|
Then store each connection in KAB under **Credentials**, per project, as described in
|
|
[customer-cloud-access.md](customer-cloud-access.md). Never tenant-wide. Well-Architected guidance uses the same connection.
|
|
|
|
### Starting a run from a context file
|
|
|
|
Every run starts from a YAML context file. Its `workflow_inputs:` become the run's inputs (validated before the run
|
|
starts). The script generates `run_id` for every run (UTC time plus a random suffix) and `plan_id` = `PLAN-<run_id>`,
|
|
and adds them as a `generated:` section; a file that sets `run_id` itself is refused. The file also declares the
|
|
`customer:` (business details only, no personal data) and the engagement `plan:`. The whole file is uploaded to the workflow's upload step, so the planner, the inventory agent and the
|
|
report also read its `context:` section. The template is
|
|
`individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/run-context.template.yaml`; the live trials have their
|
|
own in `workflows/cases/`. Never put a credential in one.
|
|
|
|
```sh
|
|
python3 scripts/kab_run_workflow.py start --workflow <id> --project <id> \
|
|
--context "individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/gcp-live/context.yaml" --dry-run
|
|
```
|
|
|
|
### Storing the connections from a script
|
|
|
|
`scripts/kab_store_connections.py` stores the discovery servers' connections, per project, through the platform's
|
|
Credentials API, without the UI. It finds each server in the MCP gateway, checks the connection against the
|
|
server's declared fields, and tests it with `connections:verify`. Only with `--apply` does it store the connection
|
|
(`PUT /api/v1/secrets`, `secretType` `mcp`, format `connection`, create-only unless `--replace`).
|
|
|
|
Credential values never appear in its output, its arguments or a file. They come from the environment
|
|
(`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`), from a service account key file kept outside the
|
|
repository, or from `az` / `gcloud` access tokens. A token lasts about an hour and carries your own rights.
|
|
|
|
```sh
|
|
python3 scripts/kab_store_connections.py --api https://<platform host> --tenant <tenant id> --project-id <project id> \
|
|
--azure-scope-root subscriptions/<id> --azure-credential service-principal \
|
|
--gcp-scope-root projects/<id> --gcp-credential key-file:$HOME/keys/<read-only-sa>.json # test only
|
|
# ... then the same command with --apply
|
|
```
|
|
|
|
It needs a KAB token (`$KAB_TOKEN` or `~/.kab_token`, `--token-file` to change) and the Administrator role. API keys work only on `/api/external/` routes, so they can't be used here. The
|
|
servers must already be registered in that platform's MCP gateway, at an address its gateway can reach.
|
|
|
|
## Rolling back and removing
|
|
|
|
```sh
|
|
kubectl --context crucible-kab-dev-aks -n agents rollout undo deploy/<agent> # the previous version
|
|
kubectl --context crucible-kab-dev-aks delete -k build/crucible-kab-dev/<agent> # remove it entirely
|
|
```
|
|
|
|
A rendered build holds only the component's Deployment, Service and ConfigMap. **Never `kubectl delete -k` an agent's own `kubernetes/` folder.** That folder lists the shared namespace, and the delete would remove the namespace together with every other agent in it.
|