Files
cp1-agents/docs/source-material/agent-shapes/subagent-pattern/README.md
Jonathan Boniface 416c66ced4
All checks were successful
check agents / check (push) Successful in 10s
Add CP1 Discovery agents in both patterns
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight
individual declarative agents (individual-agents/), with their KAB workflow
exports. Shared check, render and import scripts, the alz-discovery-sample
MCP server, the crucible-kab-dev deploy profile, deployment docs, platform
issue reports and the source material. Agent steps time out at 20 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:25:57 +00:00

21 KiB
Raw Blame History

expense-policy-agent

An A2A-protocol agent on Google's ADK, built on the platform's shared runtime image (agent-runtime-adk).

The runtime handles everything that is the same for every agent — building the agent, serving the A2A protocol, reaching the LLM gateway, correlation headers, structured logging, tracing, and the agent-loop guard rails. What is left in this repository is the part that is yours: agent.yaml (what the agent is), the instructions in prompt.md, its schemas, skills, tools and hooks. There is no app/agent.py: the runtime builds the agent from agent.yaml.

  • Model: claude-sonnet-4-5
  • Cloud / region: local / localhost

Quick start

./run-local.sh                     # build if needed, run on :8080
DEV_RELOAD=true ./run-local.sh     # same, with hot reload

.env (generated at creation, gitignored) must hold LITELLM_PROXY_BASE and LITELLM_MASTER_KEY. Then, in a second terminal:

pip install -r tests/requirements.txt
python tests/test_chatbot.py --url http://localhost:8080

tests/test_chatbot.py is an interactive chat client. It speaks the same A2A wire format the platform's Temporal worker uses to call this agent in production, so what you see locally is what a workflow will get.

run-local.sh builds the image from the Dockerfile — the runtime image plus this agent's files — and runs it with the project bind-mounted over /agent, so an edit is live on the next start. With DEV_RELOAD=true the server restarts on any change in the project — app/, schemas/, prompt.md, agent.yaml — which are the files you will iterate on most. It rebuilds the image only when the Dockerfile, the runtime image or requirements.txt changes. The runtime image comes from the dev kit's GHCR (RUNTIME_REGISTRY, or RUNTIME_IMAGE for one outright; localhost for a runtime built from source).

It is also the closest local equivalent to a deployed pod. In Kubernetes, deployment.yaml does envFrom: configMapRef, which injects every key of kubernetes/configmap.yaml into the container. run-local.sh reads that same file and passes each key as -e, so a setting you change in the ConfigMap takes effect in both places. A plain podman run with hand-written -e flags drifts from the deployed configuration silently, and the drift only shows up after deploy.

Secrets are not in the ConfigMap; they come from .env locally and from the Secrets in kubernetes/secretnames.yaml in production.

Running it against the platform

The steps above run the agent on its own. To use it in a workflow it also has to be reachable by the platform and registered in the agent catalog.

1. Put your gateway credentials in .env (once per checkout):

cp .env.example .env      # then fill in LITELLM_PROXY_BASE and LITELLM_MASTER_KEY

run-local.sh sources .env for secrets and takes everything else from kubernetes/configmap.yaml.

2. Start it on the platform's network:

PORT_HOST=10021 \
CONTAINER_NETWORK=agent-builder-local-dev-kit_agent-builder-network \
  ./run-local.sh

CONTAINER_NETWORK is what makes the agent reachable by the API and the Temporal worker; without it the agent runs fine but nothing can call it. Pick a PORT_HOST nobody else is using — it is only for your own testing.

3. Check it is up, from both sides:

curl -s localhost:10021/health                     # from your machine
python tests/test_chatbot.py --url http://localhost:10021

4. Register it in the agent catalog. In the platform UI, go to Access Control → Agents → Discover, and enter:

http://<agent-name>:8080

That is the container name on the shared network and the container's port (8080) — not localhost and not your PORT_HOST. The platform resolves that name over the podman network; localhost from inside the API container means the API itself. <agent-name> is this project's directory name, which is what run-local.sh names the container and registers as its DNS alias.

The agent then appears in the catalog and can be dropped into a workflow.

Redeploying after a change. Edit, then re-run step 2 — the script rebuilds and replaces the container. The registration is by URL and survives, so you do not need to re-register. For prompt-only changes, add DEV_RELOAD=true and skip the rebuild entirely.

Configuration

The agent itself: agent.yaml

name: my-agent
description: What this agent does, for someone choosing it from the catalog.
version: 1.0.0            # #.#.# or DEV -- the CI releases nothing else
framework: adk            # must match the runtime image in the Dockerfile
tags: [adk]
capabilities:
  streaming: true
  push_notifications: false
skills:
  - id: my_agent
    name: My Agent
    description: What this skill does.
    tags: []
    examples: []
prompt: prompt.md
contract:
  input: schemas/input.json
  output: schemas/output.json
functions:
  - app.tools:search
hooks: app.business_logic

agent.yaml, at the root of the project, is the agent: the runtime reads it at startup and builds the agent from it — its instructions, contract, tools, skills and hooks — and it is the source of truth for everything the agent card says. None of it is read from environment variables. Skills are listed either as folders under skills/ (instructions and tools the model loads when it needs them -- see skills/README.md) or as plain entries that are only advertised. Skills never change the agent's input and output: the first skill carries its schemas. To change any of it, edit the file, redeploy, and refresh the agent in the catalog. A new name registers as a new agent. Every field is described in the platform's developer guide (Help → Developer Guide → agent.yaml reference).

The runtime it is built on: the Dockerfile

The Dockerfile builds FROM the shared runtime image for this framework, named with its runtime version (.../agent-runtime-adk:<version>), and copies this agent's files into /agent. Moving to a newer runtime is a change of that line plus a new agent version — the image changes, so the version must.

Releasing a version: RELEASE_NOTES.md

The CI takes the version from agent.yaml and refuses to release it unless RELEASE_NOTES.md has an entry for it, and a released version cannot be released again. So a release is three edits: bump version in agent.yaml, add its entry at the top of RELEASE_NOTES.md, redeploy.

## [1.1.0] - 2026-10-01

### Added
- New capability or tool the agent now supports.
### Fixed
- Bug fixes included in this release.

version: DEV is the exception: it needs no entry and may be overwritten, for development.

Everything else: kubernetes/configmap.yaml

kubernetes/configmap.yaml is the contract for every other setting — each one is documented there, in the file that actually ships. This is the summary.

Required

Variable Description
LITELLM_MODEL Model ID (e.g. gpt-5.4, claude-sonnet-4-6)
LITELLM_PROXY_BASE LiteLLM gateway URL — on a cluster, set by the platform for every agent
LITELLM_MASTER_KEY LiteLLM gateway key — on a cluster, set by the platform

Optional

Variable Default Description
LITELLM_TEMPERATURE 0.0 Model temperature
LITELLM_MAX_TOKENS 16384 Max response tokens
AGENT_GUARDRAILS_ENABLED true Enforce the loop bounds below
AGENT_MAX_ITERATIONS 25 Model calls per run: the agent's, or one sub-agent's
AGENT_MAX_REQUEST_MODEL_CALLS 100 Model calls in the whole request, the sub-agents the model calls included (default 4 × AGENT_MAX_ITERATIONS)
AGENT_MAX_DURATION_SECONDS 1500 Wall clock per request (25 minutes)
AGENT_MAX_TOKENS 0 Total tokens per request; 0 = no limit
OBSERVABILITY_TYPE unset Local runs only: otel (with OTEL_EXPORTER_OTLP_ENDPOINT, the OpenTelemetry Collector) or none. On a cluster the platform sets it for every agent
OBSERVABILITY_CAPTURE_CONTENT true false keeps this agent's prompts, responses, tool data and approval comments out of traces; an agent may record less than the platform, never more
MCP_GATEWAY_URL The MCP gateway, for the MCP tools declared under mcp: in agent.yaml. Local runs only: on a cluster the platform sets it
LOG_LEVEL INFO Applies to every logger; prompts and payloads are logged only at DEBUG
LOG_FORMAT json json for a log store, console for reading on your machine
AGENT_MAX_CONCURRENT_REQUESTS 100 Requests worked on at once; one more gets HTTP 503, which Temporal retries
AGENT_INPUT_GUARDRAIL A guardrail on the LLM gateway that screens each request before the agent runs
AGENT_EGRESS log log, enforce or off: what happens to a connection to a host that is not one of the agent's endpoints
AGENT_EGRESS_ALLOW More hosts the agent may connect to, comma-separated
PORT 8080

The LITELLM_ prefix on the model settings is load-bearing. The unprefixed LLM_TEMPERATURE / LLM_MAX_TOKENS names are read by a different code path that does not configure this agent's model, so setting them is silently ignored — no error, just an agent that stays at the defaults.

On a cluster the model gateway, the MCP gateway and telemetry are the platform's, the same for every agent: the Deployment reads the platform's ConfigMap agent-platform-connections after this agent's own, so they are not in kubernetes/; traces go through the platform's OpenTelemetry Collector, which holds any keys. Locally, .env holds them, and traces go to the dev kit's collector the same way. A telemetry problem never stops the agent: the log says why (observability_off). The in-app help (Architecture & design > Telemetry) explains the path.

Which configuration wins

Environment variables are read once, at process start, by whichever path started the agent:

How it runs Configuration source
./run-local.sh kubernetes/configmap.yaml + secrets from .env
Deployed pod kubernetes/configmap.yaml via envFrom, plus Secrets

.env is never copied into the image and is never read by a deployed agent, so nothing you put there can affect production. Conversely, changing a ConfigMap value on a running deployment needs a restart — kubectl rollout restart — not just an apply.

Agent loop guard rails

An agent with tools makes as many model calls as the model decides it needs. The runtime bounds that automatically; this agent writes no code for it. Any one of the three limits ends the loop, and the caller gets an answer explaining why rather than an error or a hang.

Every run logs its own numbers, enforced or not:

agent_loop_completed iterations=4 tokens=18204 stopped_by_guardrail=False

That line is how you find the right limits for your agent: watch the typical iterations for real work, then set AGENT_MAX_ITERATIONS above it with room to spare. The defaults are deliberately loose so they catch runaways — most often a tool bug that keeps telling the model to call it again — without touching an agent that is behaving.

Setting AGENT_GUARDRAILS_ENABLED=false removes the bounds but keeps the counting and the log line, so you can measure first and enforce afterwards.

An agent with no tools cannot loop. One request is exactly one model call, and these settings have nothing to bound.

Customizing

Instructions

Edit prompt.md. It is a Markdown file loaded verbatim as the system instruction, and it is where most of an agent's behaviour actually lives.

Tools

Tools are plain Python functions in app/tools.py, each listed under functions: in agent.yaml. The framework derives the schema the model sees from the signature and docstring, so both are part of the interface:

# app/tools.py
def search(symbol: str) -> dict:
    """Find every line where `symbol` appears.

    Args:
        symbol: the identifier to look for.
    """
    ...
# agent.yaml
functions:
  - app.tools:search

MCP tools are declared under mcp: in agent.yaml instead, and reached through the platform's MCP gateway; the model sees both kinds the same way.

Keep tool results bounded. A tool that can return thousands of rows will fill the context window and drive the loop into the iteration cap; cap the result inside the tool and say in the return value that it was capped, so the model can decide what to do about it.

Business logic: before and after the model

app/business_logic.py has four hooks, all already wired up (hooks: in agent.yaml) and doing nothing until you fill them in. These are the per-call pair:

Hook Runs Use it to
before_model before each model call change what is asked — add context to the instructions
after_model after each reply inspect it — check its shape, record a metric

Use these when the work must happen on every request. Use a tool when the model should decide whether and when to do it — tools create the agent loop, callbacks do not.

A worked example, reading a file and putting it in front of the model:

@lru_cache(maxsize=1)
def load_reference() -> str:
    try:
        return (Path(__file__).parent.parent / "reference" / "notes.txt").read_text()
    except OSError:
        return ""            # never fatal: answer without it rather than 500

def before_model(call):
    if load_reference():
        call.add_instruction(load_reference())

call is the runtime's ModelCall, the same on every framework: call.add_instruction(text) before the model, call.reply_text after it, and call.native for the framework's own arguments.

Two things that catch people out:

  • COPY your data files in the Dockerfile. ./run-local.sh bind-mounts the project, so a file you forgot to copy still works locally and is missing in Kubernetes. This is the most common local-passes-deploy-fails bug.
  • before_model runs once per model call, not per request. In a tool loop that is several times. Cache anything you read from disk.

Where configuration lives

Two files, and they do not overlap:

holds read by
kubernetes/configmap.yaml every non-secret runtime setting — name, model, guard rails, observability Kubernetes (envFrom), ./run-local.sh (as -e)
.env secrets, and values specific to your machine ./run-local.sh

The ConfigMap is not a Kubernetes-only file. That is the thing worth knowing. ./run-local.sh parses it and passes every key through as -e, exactly as envFrom: configMapRef does on a cluster. Change the model there once and the local container and the deployed pod agree.

.env holds only what a ConfigMap cannot: LITELLM_MASTER_KEY and MCP_AGENT_API_KEY because a ConfigMap is stored unencrypted, and LITELLM_PROXY_BASE because the gateway URL is yours and nobody else's. DEV_RELOAD is there too, being meaningless once deployed.

Precedence is shell environment → .env → ConfigMap, so

LITELLM_MODEL=claude-sonnet-4-6 ./run-local.sh

overrides the model for one run without editing anything.

If you are hunting for a setting and it is not in .env, it is in the ConfigMap. These values used to be listed in both files; they drifted, and since the ConfigMap wins on both container paths, the .env copy was inert — you could edit it, restart, and see no change at all.

.env is generated for you at creation time and is gitignored. There is no cp .env.example .env step.

The agent's contract

Two JSON Schemas describe this agent: schemas/input.json says what it accepts, schemas/output.json says what it returns, and contract: in agent.yaml points at them. Both are written when the agent is created, from the schemas supplied then. Both are optional — a schema that was not supplied is left empty and its contract: line commented out, and an agent with no contract takes and returns free text.

Declaring them buys three things:

  • The agent card publishes both as skills[0].inputSchema / outputSchema, so the workflow builder can check this agent's output against the next agent's input before anything runs.
  • A payload that does not match the input schema is refused before a model call is spent, and the caller gets the JSON path that was wrong.
  • A reply that does not match the output schema earns one more attempt quoting the specific failure. Enforcement sits at the invocation boundary rather than in a hook because a retry needs another model call, and after_model cannot cause one.

To change the contract, edit the JSON files (and uncomment the contract: line for one you add).

Where your code goes

app/business_logic.py and app/tools.py. The hooks come in two pairs:

Hook Runs Receives
pre_process once per request the parsed input payload
before_model once per model call the runtime's ModelCall
after_model once per model call the runtime's ModelCall
post_process once per request the parsed output payload

The distinction is the one that bites. A lookup in before_model runs four times in a four-step tool loop and you pay for it four times; the same lookup in pre_process runs once. Request work goes in the outer pair, per-turn work in the inner pair.

Testing

python tests/test_chatbot.py --url http://localhost:8080     # interactive
python tests/test_chatbot.py --url http://localhost:8080 --test-message "..."

Raw protocol checks:

curl http://localhost:8080/.well-known/agent-card.json    # discovery; also the k8s probe
curl http://localhost:8080/health

curl -X POST http://localhost:8080/ \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"message/send","id":"1",
       "params":{"message":{"parts":[{"type":"text","text":"Hello!"}]}}}'

Layout

expense-policy-agent/
├── app/
│   ├── __init__.py
│   ├── business_logic.py   # your four hooks
│   └── tools.py            # your functions; list them under functions: in agent.yaml
├── schemas/
│   ├── input.json          # what the agent accepts (contract.input)
│   └── output.json         # what it returns (contract.output)
├── skills/                 # one folder per skill; see skills/README.md
├── eval/
│   └── cases.yaml          # eval suite; python -m kyndryl_agent_runtime.eval
├── kubernetes/
│   ├── configmap.yaml      # the configuration contract — start here
│   ├── deployment.yaml
│   ├── secretnames.yaml    # credentials; never in the ConfigMap
│   ├── service.yaml
│   └── kustomization.yaml
├── tests/
│   ├── a2a_client.py       # A2A client over httpx
│   ├── test_chatbot.py     # interactive chat client
│   └── requirements.txt    # host-side only; not in the image
├── agent.yaml              # the agent: identity, prompt, contract, tools, skills, hooks
├── RELEASE_NOTES.md        # one entry per released version; the CI checks it
├── prompt.md               # system instruction
├── requirements.txt        # extra Python packages, if any (built into the image)
├── run-local.sh            # container, configured from the ConfigMap (+ DEV_RELOAD)
└── Dockerfile              # FROM the runtime image, plus this agent's files

Deploying

Commit and push. CI/CD builds the image and applies kubernetes/, replacing <placeholder_for_agent_name>, <placeholder_for_image>, and <placeholder_for_port>. See docs/agent-deployment/DEPLOYMENT.md (in the platform repo) for the walkthrough, including the repository secrets that need to be set once.

To apply by hand against a cluster you already have credentials for:

kubectl apply -k kubernetes/

Troubleshooting

Symptom Cause
No traces arrive Look for observability_off (a missing key or setting) or observability_destination_unreachable in the log
Temperature/max-tokens changes do nothing Using LLM_* instead of LITELLM_*
ConfigMap edit had no effect Env vars are read at startup; restart the deployment
DEV_RELOAD does not pick up changes Set WATCHFILES_FORCE_POLLING=true — run-local.sh already does
Agent stops early with a "reached the maximum" answer It hit a guard rail; see the agent_loop_stopped log line
Connection refused LITELLM_PROXY_BASE unreachable from where the agent runs
Agent stops at startup naming a function, skill or hook agent.yaml names something that is not there: a functions: entry missing from app/tools.py, a skill folder not COPYed, a typo in hooks:
Import errors for a package you added The runtime image has only the runtime and the framework; list extra packages in requirements.txt; the next ./run-local.sh (and the pipeline) builds them in