ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
21 KiB
expense-policy-agent
An A2A-protocol agent on Google's ADK, built on the platform's shared runtime
image (agent-runtime-adk).
The runtime handles everything that is the same for every agent — building the
agent, serving the A2A protocol, reaching the LLM gateway, correlation headers,
structured logging, tracing, and the agent-loop guard rails. What is left in
this repository is the part that is yours: agent.yaml (what the agent is),
the instructions in prompt.md, its schemas, skills, tools and hooks. There is
no app/agent.py: the runtime builds the agent from agent.yaml.
- Model: claude-sonnet-4-5
- Cloud / region: local / localhost
Quick start
./run-local.sh # build if needed, run on :8080
DEV_RELOAD=true ./run-local.sh # same, with hot reload
.env (generated at creation, gitignored) must hold LITELLM_PROXY_BASE and
LITELLM_MASTER_KEY. Then, in a second terminal:
pip install -r tests/requirements.txt
python tests/test_chatbot.py --url http://localhost:8080
tests/test_chatbot.py is an interactive chat client. It speaks the same A2A
wire format the platform's Temporal worker uses to call this agent in
production, so what you see locally is what a workflow will get.
run-local.sh builds the image from the Dockerfile — the runtime image plus
this agent's files — and runs it with the project bind-mounted over /agent,
so an edit is live on the next start. With DEV_RELOAD=true the server
restarts on any change in the project — app/, schemas/, prompt.md,
agent.yaml — which are the files you will iterate on most. It rebuilds the
image only when the Dockerfile, the runtime image or requirements.txt
changes. The runtime image comes from the dev kit's GHCR (RUNTIME_REGISTRY,
or RUNTIME_IMAGE for one outright; localhost for a runtime built from
source).
It is also the closest local equivalent to a deployed pod. In Kubernetes,
deployment.yaml does envFrom: configMapRef, which injects every key of
kubernetes/configmap.yaml into the container. run-local.sh reads that same
file and passes each key as -e, so a setting you change in the ConfigMap
takes effect in both places. A plain podman run with hand-written -e flags
drifts from the deployed configuration silently, and the drift only shows up
after deploy.
Secrets are not in the ConfigMap; they come from .env locally and from the
Secrets in kubernetes/secretnames.yaml in production.
Running it against the platform
The steps above run the agent on its own. To use it in a workflow it also has to be reachable by the platform and registered in the agent catalog.
1. Put your gateway credentials in .env (once per checkout):
cp .env.example .env # then fill in LITELLM_PROXY_BASE and LITELLM_MASTER_KEY
run-local.sh sources .env for secrets and takes everything else from
kubernetes/configmap.yaml.
2. Start it on the platform's network:
PORT_HOST=10021 \
CONTAINER_NETWORK=agent-builder-local-dev-kit_agent-builder-network \
./run-local.sh
CONTAINER_NETWORK is what makes the agent reachable by the API and the
Temporal worker; without it the agent runs fine but nothing can call it.
Pick a PORT_HOST nobody else is using — it is only for your own testing.
3. Check it is up, from both sides:
curl -s localhost:10021/health # from your machine
python tests/test_chatbot.py --url http://localhost:10021
4. Register it in the agent catalog. In the platform UI, go to Access Control → Agents → Discover, and enter:
http://<agent-name>:8080
That is the container name on the shared network and the container's port
(8080) — not localhost and not your PORT_HOST. The platform resolves that
name over the podman network; localhost from inside the API container means
the API itself. <agent-name> is this project's directory name, which is what
run-local.sh names the container and registers as its DNS alias.
The agent then appears in the catalog and can be dropped into a workflow.
Redeploying after a change. Edit, then re-run step 2 — the script rebuilds
and replaces the container. The registration is by URL and survives, so you do
not need to re-register. For prompt-only changes, add DEV_RELOAD=true and
skip the rebuild entirely.
Configuration
The agent itself: agent.yaml
name: my-agent
description: What this agent does, for someone choosing it from the catalog.
version: 1.0.0 # #.#.# or DEV -- the CI releases nothing else
framework: adk # must match the runtime image in the Dockerfile
tags: [adk]
capabilities:
streaming: true
push_notifications: false
skills:
- id: my_agent
name: My Agent
description: What this skill does.
tags: []
examples: []
prompt: prompt.md
contract:
input: schemas/input.json
output: schemas/output.json
functions:
- app.tools:search
hooks: app.business_logic
agent.yaml, at the root of the project, is the agent: the runtime reads it at
startup and builds the agent from it — its instructions, contract, tools,
skills and hooks — and it is the source of truth for everything the agent card
says. None of it is read from environment variables. Skills are listed either
as folders under skills/ (instructions and tools the model loads when it
needs them -- see skills/README.md) or as plain entries that are only
advertised. Skills never change the agent's input and output: the first skill
carries its schemas. To change any of it, edit the file, redeploy, and refresh
the agent in the catalog. A new name registers as a new agent. Every field is
described in the platform's developer guide (Help → Developer Guide →
agent.yaml reference).
The runtime it is built on: the Dockerfile
The Dockerfile builds FROM the shared runtime image for this framework, named
with its runtime version (.../agent-runtime-adk:<version>), and copies this
agent's files into /agent. Moving to a newer runtime is a change of that line
plus a new agent version — the image changes, so the version must.
Releasing a version: RELEASE_NOTES.md
The CI takes the version from agent.yaml and refuses to release it unless
RELEASE_NOTES.md has an entry for it, and a released version cannot be
released again. So a release is three edits: bump version in agent.yaml,
add its entry at the top of RELEASE_NOTES.md, redeploy.
## [1.1.0] - 2026-10-01
### Added
- New capability or tool the agent now supports.
### Fixed
- Bug fixes included in this release.
version: DEV is the exception: it needs no entry and may be overwritten, for
development.
Everything else: kubernetes/configmap.yaml
kubernetes/configmap.yaml is the contract for every other setting — each one
is documented there, in the file that actually ships. This is the summary.
Required
| Variable | Description |
|---|---|
LITELLM_MODEL |
Model ID (e.g. gpt-5.4, claude-sonnet-4-6) |
LITELLM_PROXY_BASE |
LiteLLM gateway URL — on a cluster, set by the platform for every agent |
LITELLM_MASTER_KEY |
LiteLLM gateway key — on a cluster, set by the platform |
Optional
| Variable | Default | Description |
|---|---|---|
LITELLM_TEMPERATURE |
0.0 |
Model temperature |
LITELLM_MAX_TOKENS |
16384 |
Max response tokens |
AGENT_GUARDRAILS_ENABLED |
true |
Enforce the loop bounds below |
AGENT_MAX_ITERATIONS |
25 |
Model calls per run: the agent's, or one sub-agent's |
AGENT_MAX_REQUEST_MODEL_CALLS |
100 |
Model calls in the whole request, the sub-agents the model calls included (default 4 × AGENT_MAX_ITERATIONS) |
AGENT_MAX_DURATION_SECONDS |
1500 |
Wall clock per request (25 minutes) |
AGENT_MAX_TOKENS |
0 |
Total tokens per request; 0 = no limit |
OBSERVABILITY_TYPE |
unset | Local runs only: otel (with OTEL_EXPORTER_OTLP_ENDPOINT, the OpenTelemetry Collector) or none. On a cluster the platform sets it for every agent |
OBSERVABILITY_CAPTURE_CONTENT |
true |
false keeps this agent's prompts, responses, tool data and approval comments out of traces; an agent may record less than the platform, never more |
MCP_GATEWAY_URL |
The MCP gateway, for the MCP tools declared under mcp: in agent.yaml. Local runs only: on a cluster the platform sets it |
|
LOG_LEVEL |
INFO |
Applies to every logger; prompts and payloads are logged only at DEBUG |
LOG_FORMAT |
json |
json for a log store, console for reading on your machine |
AGENT_MAX_CONCURRENT_REQUESTS |
100 |
Requests worked on at once; one more gets HTTP 503, which Temporal retries |
AGENT_INPUT_GUARDRAIL |
A guardrail on the LLM gateway that screens each request before the agent runs | |
AGENT_EGRESS |
log |
log, enforce or off: what happens to a connection to a host that is not one of the agent's endpoints |
AGENT_EGRESS_ALLOW |
More hosts the agent may connect to, comma-separated | |
PORT |
8080 |
The
LITELLM_prefix on the model settings is load-bearing. The unprefixedLLM_TEMPERATURE/LLM_MAX_TOKENSnames are read by a different code path that does not configure this agent's model, so setting them is silently ignored — no error, just an agent that stays at the defaults.
On a cluster the model gateway, the MCP gateway and telemetry are the platform's, the same for every agent: the Deployment reads the platform's ConfigMap
agent-platform-connectionsafter this agent's own, so they are not inkubernetes/; traces go through the platform's OpenTelemetry Collector, which holds any keys. Locally,.envholds them, and traces go to the dev kit's collector the same way. A telemetry problem never stops the agent: the log says why (observability_off). The in-app help (Architecture & design > Telemetry) explains the path.
Which configuration wins
Environment variables are read once, at process start, by whichever path started the agent:
| How it runs | Configuration source |
|---|---|
./run-local.sh |
kubernetes/configmap.yaml + secrets from .env |
| Deployed pod | kubernetes/configmap.yaml via envFrom, plus Secrets |
.env is never copied into the image and is never read by a deployed agent, so
nothing you put there can affect production. Conversely, changing a ConfigMap
value on a running deployment needs a restart — kubectl rollout restart — not
just an apply.
Agent loop guard rails
An agent with tools makes as many model calls as the model decides it needs. The runtime bounds that automatically; this agent writes no code for it. Any one of the three limits ends the loop, and the caller gets an answer explaining why rather than an error or a hang.
Every run logs its own numbers, enforced or not:
agent_loop_completed iterations=4 tokens=18204 stopped_by_guardrail=False
That line is how you find the right limits for your agent: watch the typical
iterations for real work, then set AGENT_MAX_ITERATIONS above it with room
to spare. The defaults are deliberately loose so they catch runaways — most
often a tool bug that keeps telling the model to call it again — without
touching an agent that is behaving.
Setting AGENT_GUARDRAILS_ENABLED=false removes the bounds but keeps the
counting and the log line, so you can measure first and enforce afterwards.
An agent with no tools cannot loop. One request is exactly one model call, and these settings have nothing to bound.
Customizing
Instructions
Edit prompt.md. It is a Markdown file loaded verbatim as the system
instruction, and it is where most of an agent's behaviour actually lives.
Tools
Tools are plain Python functions in app/tools.py, each listed under
functions: in agent.yaml. The framework derives the schema the model sees
from the signature and docstring, so both are part of the interface:
# app/tools.py
def search(symbol: str) -> dict:
"""Find every line where `symbol` appears.
Args:
symbol: the identifier to look for.
"""
...
# agent.yaml
functions:
- app.tools:search
MCP tools are declared under mcp: in agent.yaml instead, and reached
through the platform's MCP gateway; the model sees both kinds the same way.
Keep tool results bounded. A tool that can return thousands of rows will fill the context window and drive the loop into the iteration cap; cap the result inside the tool and say in the return value that it was capped, so the model can decide what to do about it.
Business logic: before and after the model
app/business_logic.py has four hooks, all already wired up (hooks: in
agent.yaml) and doing nothing until you fill them in. These are the per-call
pair:
| Hook | Runs | Use it to |
|---|---|---|
before_model |
before each model call | change what is asked — add context to the instructions |
after_model |
after each reply | inspect it — check its shape, record a metric |
Use these when the work must happen on every request. Use a tool when the model should decide whether and when to do it — tools create the agent loop, callbacks do not.
A worked example, reading a file and putting it in front of the model:
@lru_cache(maxsize=1)
def load_reference() -> str:
try:
return (Path(__file__).parent.parent / "reference" / "notes.txt").read_text()
except OSError:
return "" # never fatal: answer without it rather than 500
def before_model(call):
if load_reference():
call.add_instruction(load_reference())
call is the runtime's ModelCall, the same on every framework:
call.add_instruction(text) before the model, call.reply_text after it, and
call.native for the framework's own arguments.
Two things that catch people out:
COPYyour data files in the Dockerfile../run-local.shbind-mounts the project, so a file you forgot to copy still works locally and is missing in Kubernetes. This is the most common local-passes-deploy-fails bug.before_modelruns once per model call, not per request. In a tool loop that is several times. Cache anything you read from disk.
Where configuration lives
Two files, and they do not overlap:
| holds | read by | |
|---|---|---|
kubernetes/configmap.yaml |
every non-secret runtime setting — name, model, guard rails, observability | Kubernetes (envFrom), ./run-local.sh (as -e) |
.env |
secrets, and values specific to your machine | ./run-local.sh |
The ConfigMap is not a Kubernetes-only file. That is the thing worth
knowing. ./run-local.sh parses it and passes every key through as -e,
exactly as envFrom: configMapRef does on a cluster. Change the model there
once and the local container and the deployed pod agree.
.env holds only what a ConfigMap cannot: LITELLM_MASTER_KEY and
MCP_AGENT_API_KEY because a ConfigMap is stored unencrypted, and
LITELLM_PROXY_BASE because the gateway URL is yours and nobody else's.
DEV_RELOAD is there too, being meaningless once deployed.
Precedence is shell environment → .env → ConfigMap, so
LITELLM_MODEL=claude-sonnet-4-6 ./run-local.sh
overrides the model for one run without editing anything.
If you are hunting for a setting and it is not in .env, it is in the
ConfigMap. These values used to be listed in both files; they drifted, and
since the ConfigMap wins on both container paths, the .env copy was inert —
you could edit it, restart, and see no change at all.
.env is generated for you at creation time and is gitignored. There is no
cp .env.example .env step.
The agent's contract
Two JSON Schemas describe this agent: schemas/input.json says what it
accepts, schemas/output.json says what it returns, and contract: in
agent.yaml points at them. Both are written when the agent is created, from
the schemas supplied then. Both are optional — a schema that was not supplied
is left empty and its contract: line commented out, and an agent with no
contract takes and returns free text.
Declaring them buys three things:
- The agent card publishes both as
skills[0].inputSchema/outputSchema, so the workflow builder can check this agent's output against the next agent's input before anything runs. - A payload that does not match the input schema is refused before a model call is spent, and the caller gets the JSON path that was wrong.
- A reply that does not match the output schema earns one more attempt quoting
the specific failure. Enforcement sits at the invocation boundary rather than
in a hook because a retry needs another model call, and
after_modelcannot cause one.
To change the contract, edit the JSON files (and uncomment the contract:
line for one you add).
Where your code goes
app/business_logic.py and app/tools.py. The hooks come in two pairs:
| Hook | Runs | Receives |
|---|---|---|
pre_process |
once per request | the parsed input payload |
before_model |
once per model call | the runtime's ModelCall |
after_model |
once per model call | the runtime's ModelCall |
post_process |
once per request | the parsed output payload |
The distinction is the one that bites. A lookup in before_model runs four
times in a four-step tool loop and you pay for it four times; the same lookup in
pre_process runs once. Request work goes in the outer pair, per-turn work in
the inner pair.
Testing
python tests/test_chatbot.py --url http://localhost:8080 # interactive
python tests/test_chatbot.py --url http://localhost:8080 --test-message "..."
Raw protocol checks:
curl http://localhost:8080/.well-known/agent-card.json # discovery; also the k8s probe
curl http://localhost:8080/health
curl -X POST http://localhost:8080/ \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"message/send","id":"1",
"params":{"message":{"parts":[{"type":"text","text":"Hello!"}]}}}'
Layout
expense-policy-agent/
├── app/
│ ├── __init__.py
│ ├── business_logic.py # your four hooks
│ └── tools.py # your functions; list them under functions: in agent.yaml
├── schemas/
│ ├── input.json # what the agent accepts (contract.input)
│ └── output.json # what it returns (contract.output)
├── skills/ # one folder per skill; see skills/README.md
├── eval/
│ └── cases.yaml # eval suite; python -m kyndryl_agent_runtime.eval
├── kubernetes/
│ ├── configmap.yaml # the configuration contract — start here
│ ├── deployment.yaml
│ ├── secretnames.yaml # credentials; never in the ConfigMap
│ ├── service.yaml
│ └── kustomization.yaml
├── tests/
│ ├── a2a_client.py # A2A client over httpx
│ ├── test_chatbot.py # interactive chat client
│ └── requirements.txt # host-side only; not in the image
├── agent.yaml # the agent: identity, prompt, contract, tools, skills, hooks
├── RELEASE_NOTES.md # one entry per released version; the CI checks it
├── prompt.md # system instruction
├── requirements.txt # extra Python packages, if any (built into the image)
├── run-local.sh # container, configured from the ConfigMap (+ DEV_RELOAD)
└── Dockerfile # FROM the runtime image, plus this agent's files
Deploying
Commit and push. CI/CD builds the image and applies kubernetes/, replacing
<placeholder_for_agent_name>, <placeholder_for_image>, and
<placeholder_for_port>. See docs/agent-deployment/DEPLOYMENT.md (in the platform repo) for the walkthrough,
including the repository secrets that need to be set once.
To apply by hand against a cluster you already have credentials for:
kubectl apply -k kubernetes/
Troubleshooting
| Symptom | Cause |
|---|---|
| No traces arrive | Look for observability_off (a missing key or setting) or observability_destination_unreachable in the log |
| Temperature/max-tokens changes do nothing | Using LLM_* instead of LITELLM_* |
| ConfigMap edit had no effect | Env vars are read at startup; restart the deployment |
DEV_RELOAD does not pick up changes |
Set WATCHFILES_FORCE_POLLING=true — run-local.sh already does |
| Agent stops early with a "reached the maximum" answer | It hit a guard rail; see the agent_loop_stopped log line |
| Connection refused | LITELLM_PROXY_BASE unreachable from where the agent runs |
| Agent stops at startup naming a function, skill or hook | agent.yaml names something that is not there: a functions: entry missing from app/tools.py, a skill folder not COPYed, a typo in hooks: |
| Import errors for a package you added | The runtime image has only the runtime and the framework; list extra packages in requirements.txt; the next ./run-local.sh (and the pipeline) builds them in |