All checks were successful
check agents / check (push) Successful in 10s
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight individual declarative agents (individual-agents/), with their KAB workflow exports. Shared check, render and import scripts, the alz-discovery-sample MCP server, the crucible-kab-dev deploy profile, deployment docs, platform issue reports and the source material. Agent steps time out at 20 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
496 lines
21 KiB
Markdown
496 lines
21 KiB
Markdown
# expense-policy-agent
|
||
|
||
An A2A-protocol agent on Google's ADK, built on the platform's shared runtime
|
||
image (`agent-runtime-adk`).
|
||
|
||
The runtime handles everything that is the same for every agent — building the
|
||
agent, serving the A2A protocol, reaching the LLM gateway, correlation headers,
|
||
structured logging, tracing, and the agent-loop guard rails. What is left in
|
||
this repository is the part that is yours: `agent.yaml` (what the agent is),
|
||
the instructions in `prompt.md`, its schemas, skills, tools and hooks. There is
|
||
no `app/agent.py`: the runtime builds the agent from `agent.yaml`.
|
||
|
||
- **Model**: claude-sonnet-4-5
|
||
- **Cloud / region**: local / localhost
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
./run-local.sh # build if needed, run on :8080
|
||
DEV_RELOAD=true ./run-local.sh # same, with hot reload
|
||
```
|
||
|
||
`.env` (generated at creation, gitignored) must hold `LITELLM_PROXY_BASE` and
|
||
`LITELLM_MASTER_KEY`. Then, in a second terminal:
|
||
|
||
```bash
|
||
pip install -r tests/requirements.txt
|
||
python tests/test_chatbot.py --url http://localhost:8080
|
||
```
|
||
|
||
`tests/test_chatbot.py` is an interactive chat client. It speaks the same A2A
|
||
wire format the platform's Temporal worker uses to call this agent in
|
||
production, so what you see locally is what a workflow will get.
|
||
|
||
`run-local.sh` builds the image from the Dockerfile — the runtime image plus
|
||
this agent's files — and runs it with the project bind-mounted over `/agent`,
|
||
so an edit is live on the next start. With `DEV_RELOAD=true` the server
|
||
restarts on any change in the project — `app/`, `schemas/`, `prompt.md`,
|
||
`agent.yaml` — which are the files you will iterate on most. It rebuilds the
|
||
image only when the Dockerfile, the runtime image or `requirements.txt`
|
||
changes. The runtime image comes from the dev kit's GHCR (`RUNTIME_REGISTRY`,
|
||
or `RUNTIME_IMAGE` for one outright; `localhost` for a runtime built from
|
||
source).
|
||
|
||
It is also the closest local equivalent to a deployed pod. In Kubernetes,
|
||
`deployment.yaml` does `envFrom: configMapRef`, which injects every key of
|
||
`kubernetes/configmap.yaml` into the container. `run-local.sh` reads that same
|
||
file and passes each key as `-e`, so a setting you change in the ConfigMap
|
||
takes effect in both places. A plain `podman run` with hand-written `-e` flags
|
||
drifts from the deployed configuration silently, and the drift only shows up
|
||
after deploy.
|
||
|
||
Secrets are not in the ConfigMap; they come from `.env` locally and from the
|
||
Secrets in `kubernetes/secretnames.yaml` in production.
|
||
|
||
## Running it against the platform
|
||
|
||
The steps above run the agent on its own. To use it in a workflow it also has
|
||
to be reachable by the platform and registered in the agent catalog.
|
||
|
||
**1. Put your gateway credentials in `.env`** (once per checkout):
|
||
|
||
```bash
|
||
cp .env.example .env # then fill in LITELLM_PROXY_BASE and LITELLM_MASTER_KEY
|
||
```
|
||
|
||
`run-local.sh` sources `.env` for secrets and takes everything else from
|
||
`kubernetes/configmap.yaml`.
|
||
|
||
**2. Start it on the platform's network:**
|
||
|
||
```bash
|
||
PORT_HOST=10021 \
|
||
CONTAINER_NETWORK=agent-builder-local-dev-kit_agent-builder-network \
|
||
./run-local.sh
|
||
```
|
||
|
||
`CONTAINER_NETWORK` is what makes the agent reachable by the API and the
|
||
Temporal worker; without it the agent runs fine but nothing can call it.
|
||
Pick a `PORT_HOST` nobody else is using — it is only for your own testing.
|
||
|
||
**3. Check it is up, from both sides:**
|
||
|
||
```bash
|
||
curl -s localhost:10021/health # from your machine
|
||
python tests/test_chatbot.py --url http://localhost:10021
|
||
```
|
||
|
||
**4. Register it in the agent catalog.** In the platform UI, go to
|
||
**Access Control → Agents → Discover**, and enter:
|
||
|
||
```
|
||
http://<agent-name>:8080
|
||
```
|
||
|
||
That is the *container* name on the shared network and the *container's* port
|
||
(8080) — not `localhost` and not your `PORT_HOST`. The platform resolves that
|
||
name over the podman network; `localhost` from inside the API container means
|
||
the API itself. `<agent-name>` is this project's directory name, which is what
|
||
`run-local.sh` names the container and registers as its DNS alias.
|
||
|
||
The agent then appears in the catalog and can be dropped into a workflow.
|
||
|
||
**Redeploying after a change.** Edit, then re-run step 2 — the script rebuilds
|
||
and replaces the container. The registration is by URL and survives, so you do
|
||
not need to re-register. For prompt-only changes, add `DEV_RELOAD=true` and
|
||
skip the rebuild entirely.
|
||
|
||
## Configuration
|
||
|
||
### The agent itself: `agent.yaml`
|
||
|
||
```yaml
|
||
name: my-agent
|
||
description: What this agent does, for someone choosing it from the catalog.
|
||
version: 1.0.0 # #.#.# or DEV -- the CI releases nothing else
|
||
framework: adk # must match the runtime image in the Dockerfile
|
||
tags: [adk]
|
||
capabilities:
|
||
streaming: true
|
||
push_notifications: false
|
||
skills:
|
||
- id: my_agent
|
||
name: My Agent
|
||
description: What this skill does.
|
||
tags: []
|
||
examples: []
|
||
prompt: prompt.md
|
||
contract:
|
||
input: schemas/input.json
|
||
output: schemas/output.json
|
||
functions:
|
||
- app.tools:search
|
||
hooks: app.business_logic
|
||
```
|
||
|
||
`agent.yaml`, at the root of the project, is the agent: the runtime reads it at
|
||
startup and builds the agent from it — its instructions, contract, tools,
|
||
skills and hooks — and it is the source of truth for everything the agent card
|
||
says. None of it is read from environment variables. Skills are listed either
|
||
as folders under `skills/` (instructions and tools the model loads when it
|
||
needs them -- see `skills/README.md`) or as plain entries that are only
|
||
advertised. Skills never change the agent's input and output: the first skill
|
||
carries its schemas. To change any of it, edit the file, redeploy, and refresh
|
||
the agent in the catalog. A new `name` registers as a new agent. Every field is
|
||
described in the platform's developer guide (Help → Developer Guide →
|
||
agent.yaml reference).
|
||
|
||
### The runtime it is built on: the Dockerfile
|
||
|
||
The Dockerfile builds `FROM` the shared runtime image for this framework, named
|
||
with its runtime version (`.../agent-runtime-adk:<version>`), and copies this
|
||
agent's files into `/agent`. Moving to a newer runtime is a change of that line
|
||
plus a new agent version — the image changes, so the version must.
|
||
|
||
### Releasing a version: `RELEASE_NOTES.md`
|
||
|
||
The CI takes the version from `agent.yaml` and refuses to release it unless
|
||
`RELEASE_NOTES.md` has an entry for it, and a released version cannot be
|
||
released again. So a release is three edits: bump `version` in `agent.yaml`,
|
||
add its entry at the top of `RELEASE_NOTES.md`, redeploy.
|
||
|
||
```markdown
|
||
## [1.1.0] - 2026-10-01
|
||
|
||
### Added
|
||
- New capability or tool the agent now supports.
|
||
### Fixed
|
||
- Bug fixes included in this release.
|
||
```
|
||
|
||
`version: DEV` is the exception: it needs no entry and may be overwritten, for
|
||
development.
|
||
|
||
### Everything else: `kubernetes/configmap.yaml`
|
||
|
||
`kubernetes/configmap.yaml` is the contract for every other setting — each one
|
||
is documented there, in the file that actually ships. This is the summary.
|
||
|
||
### Required
|
||
|
||
| Variable | Description |
|
||
|---|---|
|
||
| `LITELLM_MODEL` | Model ID (e.g. `gpt-5.4`, `claude-sonnet-4-6`) |
|
||
| `LITELLM_PROXY_BASE` | LiteLLM gateway URL — on a cluster, set by the platform for every agent |
|
||
| `LITELLM_MASTER_KEY` | LiteLLM gateway key — on a cluster, set by the platform |
|
||
|
||
### Optional
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `LITELLM_TEMPERATURE` | `0.0` | Model temperature |
|
||
| `LITELLM_MAX_TOKENS` | `16384` | Max response tokens |
|
||
| `AGENT_GUARDRAILS_ENABLED` | `true` | Enforce the loop bounds below |
|
||
| `AGENT_MAX_ITERATIONS` | `25` | Model calls per run: the agent's, or one sub-agent's |
|
||
| `AGENT_MAX_REQUEST_MODEL_CALLS` | `100` | Model calls in the whole request, the sub-agents the model calls included (default 4 × `AGENT_MAX_ITERATIONS`) |
|
||
| `AGENT_MAX_DURATION_SECONDS` | `1500` | Wall clock per request (25 minutes) |
|
||
| `AGENT_MAX_TOKENS` | `0` | Total tokens per request; `0` = no limit |
|
||
| `OBSERVABILITY_TYPE` | unset | Local runs only: `otel` (with `OTEL_EXPORTER_OTLP_ENDPOINT`, the OpenTelemetry Collector) or `none`. On a cluster the platform sets it for every agent |
|
||
| `OBSERVABILITY_CAPTURE_CONTENT` | `true` | `false` keeps this agent's prompts, responses, tool data and approval comments out of traces; an agent may record less than the platform, never more |
|
||
| `MCP_GATEWAY_URL` | | The MCP gateway, for the MCP tools declared under `mcp:` in `agent.yaml`. Local runs only: on a cluster the platform sets it |
|
||
| `LOG_LEVEL` | `INFO` | Applies to every logger; prompts and payloads are logged only at `DEBUG` |
|
||
| `LOG_FORMAT` | `json` | `json` for a log store, `console` for reading on your machine |
|
||
| `AGENT_MAX_CONCURRENT_REQUESTS` | `100` | Requests worked on at once; one more gets HTTP 503, which Temporal retries |
|
||
| `AGENT_INPUT_GUARDRAIL` | | A guardrail on the LLM gateway that screens each request before the agent runs |
|
||
| `AGENT_EGRESS` | `log` | `log`, `enforce` or `off`: what happens to a connection to a host that is not one of the agent's endpoints |
|
||
| `AGENT_EGRESS_ALLOW` | | More hosts the agent may connect to, comma-separated |
|
||
| `PORT` | `8080` | |
|
||
|
||
> The `LITELLM_` prefix on the model settings is load-bearing. The unprefixed
|
||
> `LLM_TEMPERATURE` / `LLM_MAX_TOKENS` names are read by a different code path
|
||
> that does not configure this agent's model, so setting them is silently
|
||
> ignored — no error, just an agent that stays at the defaults.
|
||
|
||
> On a cluster the model gateway, the MCP gateway and telemetry are the
|
||
> platform's, the same for every agent: the Deployment reads the platform's
|
||
> ConfigMap `agent-platform-connections` after this agent's own, so they are
|
||
> not in `kubernetes/`; traces go through the platform's OpenTelemetry
|
||
> Collector, which holds any keys. Locally, `.env`
|
||
> holds them, and traces go to the dev kit's collector the same way. A
|
||
> telemetry problem never stops the agent: the log
|
||
> says why (`observability_off`). The in-app help (Architecture & design >
|
||
> Telemetry) explains the path.
|
||
|
||
### Which configuration wins
|
||
|
||
Environment variables are read once, at process start, by whichever path
|
||
started the agent:
|
||
|
||
| How it runs | Configuration source |
|
||
|---|---|
|
||
| `./run-local.sh` | `kubernetes/configmap.yaml` + secrets from `.env` |
|
||
| Deployed pod | `kubernetes/configmap.yaml` via `envFrom`, plus Secrets |
|
||
|
||
`.env` is never copied into the image and is never read by a deployed agent, so
|
||
nothing you put there can affect production. Conversely, changing a ConfigMap
|
||
value on a running deployment needs a restart — `kubectl rollout restart` — not
|
||
just an apply.
|
||
|
||
## Agent loop guard rails
|
||
|
||
An agent with tools makes as many model calls as the model decides it needs.
|
||
The runtime bounds that automatically; this agent writes no code for it. Any
|
||
one of the three limits ends the loop, and the caller gets an answer explaining
|
||
why rather than an error or a hang.
|
||
|
||
Every run logs its own numbers, enforced or not:
|
||
|
||
```
|
||
agent_loop_completed iterations=4 tokens=18204 stopped_by_guardrail=False
|
||
```
|
||
|
||
That line is how you find the right limits for your agent: watch the typical
|
||
`iterations` for real work, then set `AGENT_MAX_ITERATIONS` above it with room
|
||
to spare. The defaults are deliberately loose so they catch runaways — most
|
||
often a tool bug that keeps telling the model to call it again — without
|
||
touching an agent that is behaving.
|
||
|
||
Setting `AGENT_GUARDRAILS_ENABLED=false` removes the bounds but keeps the
|
||
counting and the log line, so you can measure first and enforce afterwards.
|
||
|
||
**An agent with no tools cannot loop.** One request is exactly one model call,
|
||
and these settings have nothing to bound.
|
||
|
||
## Customizing
|
||
|
||
### Instructions
|
||
|
||
Edit `prompt.md`. It is a Markdown file loaded verbatim as the system
|
||
instruction, and it is where most of an agent's behaviour actually lives.
|
||
|
||
### Tools
|
||
|
||
Tools are plain Python functions in `app/tools.py`, each listed under
|
||
`functions:` in `agent.yaml`. The framework derives the schema the model sees
|
||
from the signature and docstring, so both are part of the interface:
|
||
|
||
```python
|
||
# app/tools.py
|
||
def search(symbol: str) -> dict:
|
||
"""Find every line where `symbol` appears.
|
||
|
||
Args:
|
||
symbol: the identifier to look for.
|
||
"""
|
||
...
|
||
```
|
||
|
||
```yaml
|
||
# agent.yaml
|
||
functions:
|
||
- app.tools:search
|
||
```
|
||
|
||
MCP tools are declared under `mcp:` in `agent.yaml` instead, and reached
|
||
through the platform's MCP gateway; the model sees both kinds the same way.
|
||
|
||
Keep tool results bounded. A tool that can return thousands of rows will fill
|
||
the context window and drive the loop into the iteration cap; cap the result
|
||
inside the tool and say in the return value that it was capped, so the model
|
||
can decide what to do about it.
|
||
|
||
### Business logic: before and after the model
|
||
|
||
`app/business_logic.py` has four hooks, all already wired up (`hooks:` in
|
||
`agent.yaml`) and doing nothing until you fill them in. These are the per-call
|
||
pair:
|
||
|
||
| Hook | Runs | Use it to |
|
||
|---|---|---|
|
||
| `before_model` | before each model call | change what is asked — add context to the instructions |
|
||
| `after_model` | after each reply | inspect it — check its shape, record a metric |
|
||
|
||
Use these when the work must happen on **every** request. Use a **tool** when
|
||
the model should decide whether and when to do it — tools create the agent
|
||
loop, callbacks do not.
|
||
|
||
A worked example, reading a file and putting it in front of the model:
|
||
|
||
```python
|
||
@lru_cache(maxsize=1)
|
||
def load_reference() -> str:
|
||
try:
|
||
return (Path(__file__).parent.parent / "reference" / "notes.txt").read_text()
|
||
except OSError:
|
||
return "" # never fatal: answer without it rather than 500
|
||
|
||
def before_model(call):
|
||
if load_reference():
|
||
call.add_instruction(load_reference())
|
||
```
|
||
|
||
`call` is the runtime's `ModelCall`, the same on every framework:
|
||
`call.add_instruction(text)` before the model, `call.reply_text` after it, and
|
||
`call.native` for the framework's own arguments.
|
||
|
||
Two things that catch people out:
|
||
|
||
- **`COPY` your data files in the Dockerfile.** `./run-local.sh` bind-mounts the
|
||
project, so a file you forgot to copy still works locally and is missing in
|
||
Kubernetes. This is the most common local-passes-deploy-fails bug.
|
||
- **`before_model` runs once per *model call*, not per request.** In a tool loop
|
||
that is several times. Cache anything you read from disk.
|
||
|
||
### Where configuration lives
|
||
|
||
Two files, and they do not overlap:
|
||
|
||
| | holds | read by |
|
||
|---|---|---|
|
||
| `kubernetes/configmap.yaml` | every non-secret runtime setting — name, model, guard rails, observability | Kubernetes (`envFrom`), `./run-local.sh` (as `-e`) |
|
||
| `.env` | secrets, and values specific to your machine | `./run-local.sh` |
|
||
|
||
**The ConfigMap is not a Kubernetes-only file.** That is the thing worth
|
||
knowing. `./run-local.sh` parses it and passes every key through as `-e`,
|
||
exactly as `envFrom: configMapRef` does on a cluster. Change the model there
|
||
once and the local container and the deployed pod agree.
|
||
|
||
`.env` holds only what a ConfigMap cannot: `LITELLM_MASTER_KEY` and
|
||
`MCP_AGENT_API_KEY` because a ConfigMap is stored unencrypted, and
|
||
`LITELLM_PROXY_BASE` because the gateway URL is yours and nobody else's.
|
||
`DEV_RELOAD` is there too, being meaningless once deployed.
|
||
|
||
Precedence is `shell environment` → `.env` → ConfigMap, so
|
||
|
||
```bash
|
||
LITELLM_MODEL=claude-sonnet-4-6 ./run-local.sh
|
||
```
|
||
|
||
overrides the model for one run without editing anything.
|
||
|
||
If you are hunting for a setting and it is not in `.env`, it is in the
|
||
ConfigMap. These values used to be listed in both files; they drifted, and
|
||
since the ConfigMap wins on both container paths, the `.env` copy was inert —
|
||
you could edit it, restart, and see no change at all.
|
||
|
||
`.env` is generated for you at creation time and is gitignored. There is no
|
||
`cp .env.example .env` step.
|
||
|
||
### The agent's contract
|
||
|
||
Two JSON Schemas describe this agent: `schemas/input.json` says what it
|
||
accepts, `schemas/output.json` says what it returns, and `contract:` in
|
||
`agent.yaml` points at them. Both are written when the agent is created, from
|
||
the schemas supplied then. Both are optional — a schema that was not supplied
|
||
is left empty and its `contract:` line commented out, and an agent with no
|
||
contract takes and returns free text.
|
||
|
||
Declaring them buys three things:
|
||
|
||
- The agent card publishes both as `skills[0].inputSchema` / `outputSchema`, so
|
||
the workflow builder can check this agent's output against the next agent's
|
||
input *before* anything runs.
|
||
- A payload that does not match the input schema is refused before a model call
|
||
is spent, and the caller gets the JSON path that was wrong.
|
||
- A reply that does not match the output schema earns one more attempt quoting
|
||
the specific failure. Enforcement sits at the invocation boundary rather than
|
||
in a hook because a retry needs another model call, and `after_model` cannot
|
||
cause one.
|
||
|
||
To change the contract, edit the JSON files (and uncomment the `contract:`
|
||
line for one you add).
|
||
|
||
### Where your code goes
|
||
|
||
`app/business_logic.py` and `app/tools.py`. The hooks come in two pairs:
|
||
|
||
| Hook | Runs | Receives |
|
||
|---|---|---|
|
||
| `pre_process` | once per request | the parsed input payload |
|
||
| `before_model` | once per **model call** | the runtime's `ModelCall` |
|
||
| `after_model` | once per **model call** | the runtime's `ModelCall` |
|
||
| `post_process` | once per request | the parsed output payload |
|
||
|
||
The distinction is the one that bites. A lookup in `before_model` runs four
|
||
times in a four-step tool loop and you pay for it four times; the same lookup in
|
||
`pre_process` runs once. Request work goes in the outer pair, per-turn work in
|
||
the inner pair.
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
python tests/test_chatbot.py --url http://localhost:8080 # interactive
|
||
python tests/test_chatbot.py --url http://localhost:8080 --test-message "..."
|
||
```
|
||
|
||
Raw protocol checks:
|
||
|
||
```bash
|
||
curl http://localhost:8080/.well-known/agent-card.json # discovery; also the k8s probe
|
||
curl http://localhost:8080/health
|
||
|
||
curl -X POST http://localhost:8080/ \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"jsonrpc":"2.0","method":"message/send","id":"1",
|
||
"params":{"message":{"parts":[{"type":"text","text":"Hello!"}]}}}'
|
||
```
|
||
|
||
## Layout
|
||
|
||
```
|
||
expense-policy-agent/
|
||
├── app/
|
||
│ ├── __init__.py
|
||
│ ├── business_logic.py # your four hooks
|
||
│ └── tools.py # your functions; list them under functions: in agent.yaml
|
||
├── schemas/
|
||
│ ├── input.json # what the agent accepts (contract.input)
|
||
│ └── output.json # what it returns (contract.output)
|
||
├── skills/ # one folder per skill; see skills/README.md
|
||
├── eval/
|
||
│ └── cases.yaml # eval suite; python -m kyndryl_agent_runtime.eval
|
||
├── kubernetes/
|
||
│ ├── configmap.yaml # the configuration contract — start here
|
||
│ ├── deployment.yaml
|
||
│ ├── secretnames.yaml # credentials; never in the ConfigMap
|
||
│ ├── service.yaml
|
||
│ └── kustomization.yaml
|
||
├── tests/
|
||
│ ├── a2a_client.py # A2A client over httpx
|
||
│ ├── test_chatbot.py # interactive chat client
|
||
│ └── requirements.txt # host-side only; not in the image
|
||
├── agent.yaml # the agent: identity, prompt, contract, tools, skills, hooks
|
||
├── RELEASE_NOTES.md # one entry per released version; the CI checks it
|
||
├── prompt.md # system instruction
|
||
├── requirements.txt # extra Python packages, if any (built into the image)
|
||
├── run-local.sh # container, configured from the ConfigMap (+ DEV_RELOAD)
|
||
└── Dockerfile # FROM the runtime image, plus this agent's files
|
||
```
|
||
|
||
## Deploying
|
||
|
||
Commit and push. CI/CD builds the image and applies `kubernetes/`, replacing
|
||
`<placeholder_for_agent_name>`, `<placeholder_for_image>`, and
|
||
`<placeholder_for_port>`. See `docs/agent-deployment/DEPLOYMENT.md` (in the platform repo) for the walkthrough,
|
||
including the repository secrets that need to be set once.
|
||
|
||
To apply by hand against a cluster you already have credentials for:
|
||
|
||
```bash
|
||
kubectl apply -k kubernetes/
|
||
```
|
||
|
||
## Troubleshooting
|
||
|
||
| Symptom | Cause |
|
||
|---|---|
|
||
| No traces arrive | Look for `observability_off` (a missing key or setting) or `observability_destination_unreachable` in the log |
|
||
| Temperature/max-tokens changes do nothing | Using `LLM_*` instead of `LITELLM_*` |
|
||
| ConfigMap edit had no effect | Env vars are read at startup; restart the deployment |
|
||
| `DEV_RELOAD` does not pick up changes | Set `WATCHFILES_FORCE_POLLING=true` — `run-local.sh` already does |
|
||
| Agent stops early with a "reached the maximum" answer | It hit a guard rail; see the `agent_loop_stopped` log line |
|
||
| Connection refused | `LITELLM_PROXY_BASE` unreachable from where the agent runs |
|
||
| Agent stops at startup naming a function, skill or hook | `agent.yaml` names something that is not there: a `functions:` entry missing from `app/tools.py`, a skill folder not COPYed, a typo in `hooks:` |
|
||
| Import errors for a package you added | The runtime image has only the runtime and the framework; list extra packages in `requirements.txt`; the next `./run-local.sh` (and the pipeline) builds them in |
|