Files
cp1-agents/docs/source-material/agent-shapes/subagent-pattern/README.md
Jonathan Boniface 416c66ced4
All checks were successful
check agents / check (push) Successful in 10s
Add CP1 Discovery agents in both patterns
ALZ-F01 Discovery as a sub-agent orchestrator (sub-agents/) and as eight
individual declarative agents (individual-agents/), with their KAB workflow
exports. Shared check, render and import scripts, the alz-discovery-sample
MCP server, the crucible-kab-dev deploy profile, deployment docs, platform
issue reports and the source material. Agent steps time out at 20 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 18:25:57 +00:00

496 lines
21 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# expense-policy-agent
An A2A-protocol agent on Google's ADK, built on the platform's shared runtime
image (`agent-runtime-adk`).
The runtime handles everything that is the same for every agent — building the
agent, serving the A2A protocol, reaching the LLM gateway, correlation headers,
structured logging, tracing, and the agent-loop guard rails. What is left in
this repository is the part that is yours: `agent.yaml` (what the agent is),
the instructions in `prompt.md`, its schemas, skills, tools and hooks. There is
no `app/agent.py`: the runtime builds the agent from `agent.yaml`.
- **Model**: claude-sonnet-4-5
- **Cloud / region**: local / localhost
## Quick start
```bash
./run-local.sh # build if needed, run on :8080
DEV_RELOAD=true ./run-local.sh # same, with hot reload
```
`.env` (generated at creation, gitignored) must hold `LITELLM_PROXY_BASE` and
`LITELLM_MASTER_KEY`. Then, in a second terminal:
```bash
pip install -r tests/requirements.txt
python tests/test_chatbot.py --url http://localhost:8080
```
`tests/test_chatbot.py` is an interactive chat client. It speaks the same A2A
wire format the platform's Temporal worker uses to call this agent in
production, so what you see locally is what a workflow will get.
`run-local.sh` builds the image from the Dockerfile — the runtime image plus
this agent's files — and runs it with the project bind-mounted over `/agent`,
so an edit is live on the next start. With `DEV_RELOAD=true` the server
restarts on any change in the project — `app/`, `schemas/`, `prompt.md`,
`agent.yaml` — which are the files you will iterate on most. It rebuilds the
image only when the Dockerfile, the runtime image or `requirements.txt`
changes. The runtime image comes from the dev kit's GHCR (`RUNTIME_REGISTRY`,
or `RUNTIME_IMAGE` for one outright; `localhost` for a runtime built from
source).
It is also the closest local equivalent to a deployed pod. In Kubernetes,
`deployment.yaml` does `envFrom: configMapRef`, which injects every key of
`kubernetes/configmap.yaml` into the container. `run-local.sh` reads that same
file and passes each key as `-e`, so a setting you change in the ConfigMap
takes effect in both places. A plain `podman run` with hand-written `-e` flags
drifts from the deployed configuration silently, and the drift only shows up
after deploy.
Secrets are not in the ConfigMap; they come from `.env` locally and from the
Secrets in `kubernetes/secretnames.yaml` in production.
## Running it against the platform
The steps above run the agent on its own. To use it in a workflow it also has
to be reachable by the platform and registered in the agent catalog.
**1. Put your gateway credentials in `.env`** (once per checkout):
```bash
cp .env.example .env # then fill in LITELLM_PROXY_BASE and LITELLM_MASTER_KEY
```
`run-local.sh` sources `.env` for secrets and takes everything else from
`kubernetes/configmap.yaml`.
**2. Start it on the platform's network:**
```bash
PORT_HOST=10021 \
CONTAINER_NETWORK=agent-builder-local-dev-kit_agent-builder-network \
./run-local.sh
```
`CONTAINER_NETWORK` is what makes the agent reachable by the API and the
Temporal worker; without it the agent runs fine but nothing can call it.
Pick a `PORT_HOST` nobody else is using — it is only for your own testing.
**3. Check it is up, from both sides:**
```bash
curl -s localhost:10021/health # from your machine
python tests/test_chatbot.py --url http://localhost:10021
```
**4. Register it in the agent catalog.** In the platform UI, go to
**Access Control → Agents → Discover**, and enter:
```
http://<agent-name>:8080
```
That is the *container* name on the shared network and the *container's* port
(8080) — not `localhost` and not your `PORT_HOST`. The platform resolves that
name over the podman network; `localhost` from inside the API container means
the API itself. `<agent-name>` is this project's directory name, which is what
`run-local.sh` names the container and registers as its DNS alias.
The agent then appears in the catalog and can be dropped into a workflow.
**Redeploying after a change.** Edit, then re-run step 2 — the script rebuilds
and replaces the container. The registration is by URL and survives, so you do
not need to re-register. For prompt-only changes, add `DEV_RELOAD=true` and
skip the rebuild entirely.
## Configuration
### The agent itself: `agent.yaml`
```yaml
name: my-agent
description: What this agent does, for someone choosing it from the catalog.
version: 1.0.0 # #.#.# or DEV -- the CI releases nothing else
framework: adk # must match the runtime image in the Dockerfile
tags: [adk]
capabilities:
streaming: true
push_notifications: false
skills:
- id: my_agent
name: My Agent
description: What this skill does.
tags: []
examples: []
prompt: prompt.md
contract:
input: schemas/input.json
output: schemas/output.json
functions:
- app.tools:search
hooks: app.business_logic
```
`agent.yaml`, at the root of the project, is the agent: the runtime reads it at
startup and builds the agent from it — its instructions, contract, tools,
skills and hooks — and it is the source of truth for everything the agent card
says. None of it is read from environment variables. Skills are listed either
as folders under `skills/` (instructions and tools the model loads when it
needs them -- see `skills/README.md`) or as plain entries that are only
advertised. Skills never change the agent's input and output: the first skill
carries its schemas. To change any of it, edit the file, redeploy, and refresh
the agent in the catalog. A new `name` registers as a new agent. Every field is
described in the platform's developer guide (Help → Developer Guide →
agent.yaml reference).
### The runtime it is built on: the Dockerfile
The Dockerfile builds `FROM` the shared runtime image for this framework, named
with its runtime version (`.../agent-runtime-adk:<version>`), and copies this
agent's files into `/agent`. Moving to a newer runtime is a change of that line
plus a new agent version — the image changes, so the version must.
### Releasing a version: `RELEASE_NOTES.md`
The CI takes the version from `agent.yaml` and refuses to release it unless
`RELEASE_NOTES.md` has an entry for it, and a released version cannot be
released again. So a release is three edits: bump `version` in `agent.yaml`,
add its entry at the top of `RELEASE_NOTES.md`, redeploy.
```markdown
## [1.1.0] - 2026-10-01
### Added
- New capability or tool the agent now supports.
### Fixed
- Bug fixes included in this release.
```
`version: DEV` is the exception: it needs no entry and may be overwritten, for
development.
### Everything else: `kubernetes/configmap.yaml`
`kubernetes/configmap.yaml` is the contract for every other setting — each one
is documented there, in the file that actually ships. This is the summary.
### Required
| Variable | Description |
|---|---|
| `LITELLM_MODEL` | Model ID (e.g. `gpt-5.4`, `claude-sonnet-4-6`) |
| `LITELLM_PROXY_BASE` | LiteLLM gateway URL — on a cluster, set by the platform for every agent |
| `LITELLM_MASTER_KEY` | LiteLLM gateway key — on a cluster, set by the platform |
### Optional
| Variable | Default | Description |
|---|---|---|
| `LITELLM_TEMPERATURE` | `0.0` | Model temperature |
| `LITELLM_MAX_TOKENS` | `16384` | Max response tokens |
| `AGENT_GUARDRAILS_ENABLED` | `true` | Enforce the loop bounds below |
| `AGENT_MAX_ITERATIONS` | `25` | Model calls per run: the agent's, or one sub-agent's |
| `AGENT_MAX_REQUEST_MODEL_CALLS` | `100` | Model calls in the whole request, the sub-agents the model calls included (default 4 × `AGENT_MAX_ITERATIONS`) |
| `AGENT_MAX_DURATION_SECONDS` | `1500` | Wall clock per request (25 minutes) |
| `AGENT_MAX_TOKENS` | `0` | Total tokens per request; `0` = no limit |
| `OBSERVABILITY_TYPE` | unset | Local runs only: `otel` (with `OTEL_EXPORTER_OTLP_ENDPOINT`, the OpenTelemetry Collector) or `none`. On a cluster the platform sets it for every agent |
| `OBSERVABILITY_CAPTURE_CONTENT` | `true` | `false` keeps this agent's prompts, responses, tool data and approval comments out of traces; an agent may record less than the platform, never more |
| `MCP_GATEWAY_URL` | | The MCP gateway, for the MCP tools declared under `mcp:` in `agent.yaml`. Local runs only: on a cluster the platform sets it |
| `LOG_LEVEL` | `INFO` | Applies to every logger; prompts and payloads are logged only at `DEBUG` |
| `LOG_FORMAT` | `json` | `json` for a log store, `console` for reading on your machine |
| `AGENT_MAX_CONCURRENT_REQUESTS` | `100` | Requests worked on at once; one more gets HTTP 503, which Temporal retries |
| `AGENT_INPUT_GUARDRAIL` | | A guardrail on the LLM gateway that screens each request before the agent runs |
| `AGENT_EGRESS` | `log` | `log`, `enforce` or `off`: what happens to a connection to a host that is not one of the agent's endpoints |
| `AGENT_EGRESS_ALLOW` | | More hosts the agent may connect to, comma-separated |
| `PORT` | `8080` | |
> The `LITELLM_` prefix on the model settings is load-bearing. The unprefixed
> `LLM_TEMPERATURE` / `LLM_MAX_TOKENS` names are read by a different code path
> that does not configure this agent's model, so setting them is silently
> ignored — no error, just an agent that stays at the defaults.
> On a cluster the model gateway, the MCP gateway and telemetry are the
> platform's, the same for every agent: the Deployment reads the platform's
> ConfigMap `agent-platform-connections` after this agent's own, so they are
> not in `kubernetes/`; traces go through the platform's OpenTelemetry
> Collector, which holds any keys. Locally, `.env`
> holds them, and traces go to the dev kit's collector the same way. A
> telemetry problem never stops the agent: the log
> says why (`observability_off`). The in-app help (Architecture & design >
> Telemetry) explains the path.
### Which configuration wins
Environment variables are read once, at process start, by whichever path
started the agent:
| How it runs | Configuration source |
|---|---|
| `./run-local.sh` | `kubernetes/configmap.yaml` + secrets from `.env` |
| Deployed pod | `kubernetes/configmap.yaml` via `envFrom`, plus Secrets |
`.env` is never copied into the image and is never read by a deployed agent, so
nothing you put there can affect production. Conversely, changing a ConfigMap
value on a running deployment needs a restart — `kubectl rollout restart` — not
just an apply.
## Agent loop guard rails
An agent with tools makes as many model calls as the model decides it needs.
The runtime bounds that automatically; this agent writes no code for it. Any
one of the three limits ends the loop, and the caller gets an answer explaining
why rather than an error or a hang.
Every run logs its own numbers, enforced or not:
```
agent_loop_completed iterations=4 tokens=18204 stopped_by_guardrail=False
```
That line is how you find the right limits for your agent: watch the typical
`iterations` for real work, then set `AGENT_MAX_ITERATIONS` above it with room
to spare. The defaults are deliberately loose so they catch runaways — most
often a tool bug that keeps telling the model to call it again — without
touching an agent that is behaving.
Setting `AGENT_GUARDRAILS_ENABLED=false` removes the bounds but keeps the
counting and the log line, so you can measure first and enforce afterwards.
**An agent with no tools cannot loop.** One request is exactly one model call,
and these settings have nothing to bound.
## Customizing
### Instructions
Edit `prompt.md`. It is a Markdown file loaded verbatim as the system
instruction, and it is where most of an agent's behaviour actually lives.
### Tools
Tools are plain Python functions in `app/tools.py`, each listed under
`functions:` in `agent.yaml`. The framework derives the schema the model sees
from the signature and docstring, so both are part of the interface:
```python
# app/tools.py
def search(symbol: str) -> dict:
"""Find every line where `symbol` appears.
Args:
symbol: the identifier to look for.
"""
...
```
```yaml
# agent.yaml
functions:
- app.tools:search
```
MCP tools are declared under `mcp:` in `agent.yaml` instead, and reached
through the platform's MCP gateway; the model sees both kinds the same way.
Keep tool results bounded. A tool that can return thousands of rows will fill
the context window and drive the loop into the iteration cap; cap the result
inside the tool and say in the return value that it was capped, so the model
can decide what to do about it.
### Business logic: before and after the model
`app/business_logic.py` has four hooks, all already wired up (`hooks:` in
`agent.yaml`) and doing nothing until you fill them in. These are the per-call
pair:
| Hook | Runs | Use it to |
|---|---|---|
| `before_model` | before each model call | change what is asked — add context to the instructions |
| `after_model` | after each reply | inspect it — check its shape, record a metric |
Use these when the work must happen on **every** request. Use a **tool** when
the model should decide whether and when to do it — tools create the agent
loop, callbacks do not.
A worked example, reading a file and putting it in front of the model:
```python
@lru_cache(maxsize=1)
def load_reference() -> str:
try:
return (Path(__file__).parent.parent / "reference" / "notes.txt").read_text()
except OSError:
return "" # never fatal: answer without it rather than 500
def before_model(call):
if load_reference():
call.add_instruction(load_reference())
```
`call` is the runtime's `ModelCall`, the same on every framework:
`call.add_instruction(text)` before the model, `call.reply_text` after it, and
`call.native` for the framework's own arguments.
Two things that catch people out:
- **`COPY` your data files in the Dockerfile.** `./run-local.sh` bind-mounts the
project, so a file you forgot to copy still works locally and is missing in
Kubernetes. This is the most common local-passes-deploy-fails bug.
- **`before_model` runs once per *model call*, not per request.** In a tool loop
that is several times. Cache anything you read from disk.
### Where configuration lives
Two files, and they do not overlap:
| | holds | read by |
|---|---|---|
| `kubernetes/configmap.yaml` | every non-secret runtime setting — name, model, guard rails, observability | Kubernetes (`envFrom`), `./run-local.sh` (as `-e`) |
| `.env` | secrets, and values specific to your machine | `./run-local.sh` |
**The ConfigMap is not a Kubernetes-only file.** That is the thing worth
knowing. `./run-local.sh` parses it and passes every key through as `-e`,
exactly as `envFrom: configMapRef` does on a cluster. Change the model there
once and the local container and the deployed pod agree.
`.env` holds only what a ConfigMap cannot: `LITELLM_MASTER_KEY` and
`MCP_AGENT_API_KEY` because a ConfigMap is stored unencrypted, and
`LITELLM_PROXY_BASE` because the gateway URL is yours and nobody else's.
`DEV_RELOAD` is there too, being meaningless once deployed.
Precedence is `shell environment` → `.env` → ConfigMap, so
```bash
LITELLM_MODEL=claude-sonnet-4-6 ./run-local.sh
```
overrides the model for one run without editing anything.
If you are hunting for a setting and it is not in `.env`, it is in the
ConfigMap. These values used to be listed in both files; they drifted, and
since the ConfigMap wins on both container paths, the `.env` copy was inert —
you could edit it, restart, and see no change at all.
`.env` is generated for you at creation time and is gitignored. There is no
`cp .env.example .env` step.
### The agent's contract
Two JSON Schemas describe this agent: `schemas/input.json` says what it
accepts, `schemas/output.json` says what it returns, and `contract:` in
`agent.yaml` points at them. Both are written when the agent is created, from
the schemas supplied then. Both are optional — a schema that was not supplied
is left empty and its `contract:` line commented out, and an agent with no
contract takes and returns free text.
Declaring them buys three things:
- The agent card publishes both as `skills[0].inputSchema` / `outputSchema`, so
the workflow builder can check this agent's output against the next agent's
input *before* anything runs.
- A payload that does not match the input schema is refused before a model call
is spent, and the caller gets the JSON path that was wrong.
- A reply that does not match the output schema earns one more attempt quoting
the specific failure. Enforcement sits at the invocation boundary rather than
in a hook because a retry needs another model call, and `after_model` cannot
cause one.
To change the contract, edit the JSON files (and uncomment the `contract:`
line for one you add).
### Where your code goes
`app/business_logic.py` and `app/tools.py`. The hooks come in two pairs:
| Hook | Runs | Receives |
|---|---|---|
| `pre_process` | once per request | the parsed input payload |
| `before_model` | once per **model call** | the runtime's `ModelCall` |
| `after_model` | once per **model call** | the runtime's `ModelCall` |
| `post_process` | once per request | the parsed output payload |
The distinction is the one that bites. A lookup in `before_model` runs four
times in a four-step tool loop and you pay for it four times; the same lookup in
`pre_process` runs once. Request work goes in the outer pair, per-turn work in
the inner pair.
## Testing
```bash
python tests/test_chatbot.py --url http://localhost:8080 # interactive
python tests/test_chatbot.py --url http://localhost:8080 --test-message "..."
```
Raw protocol checks:
```bash
curl http://localhost:8080/.well-known/agent-card.json # discovery; also the k8s probe
curl http://localhost:8080/health
curl -X POST http://localhost:8080/ \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"message/send","id":"1",
"params":{"message":{"parts":[{"type":"text","text":"Hello!"}]}}}'
```
## Layout
```
expense-policy-agent/
├── app/
│ ├── __init__.py
│ ├── business_logic.py # your four hooks
│ └── tools.py # your functions; list them under functions: in agent.yaml
├── schemas/
│ ├── input.json # what the agent accepts (contract.input)
│ └── output.json # what it returns (contract.output)
├── skills/ # one folder per skill; see skills/README.md
├── eval/
│ └── cases.yaml # eval suite; python -m kyndryl_agent_runtime.eval
├── kubernetes/
│ ├── configmap.yaml # the configuration contract — start here
│ ├── deployment.yaml
│ ├── secretnames.yaml # credentials; never in the ConfigMap
│ ├── service.yaml
│ └── kustomization.yaml
├── tests/
│ ├── a2a_client.py # A2A client over httpx
│ ├── test_chatbot.py # interactive chat client
│ └── requirements.txt # host-side only; not in the image
├── agent.yaml # the agent: identity, prompt, contract, tools, skills, hooks
├── RELEASE_NOTES.md # one entry per released version; the CI checks it
├── prompt.md # system instruction
├── requirements.txt # extra Python packages, if any (built into the image)
├── run-local.sh # container, configured from the ConfigMap (+ DEV_RELOAD)
└── Dockerfile # FROM the runtime image, plus this agent's files
```
## Deploying
Commit and push. CI/CD builds the image and applies `kubernetes/`, replacing
`<placeholder_for_agent_name>`, `<placeholder_for_image>`, and
`<placeholder_for_port>`. See `docs/agent-deployment/DEPLOYMENT.md` (in the platform repo) for the walkthrough,
including the repository secrets that need to be set once.
To apply by hand against a cluster you already have credentials for:
```bash
kubectl apply -k kubernetes/
```
## Troubleshooting
| Symptom | Cause |
|---|---|
| No traces arrive | Look for `observability_off` (a missing key or setting) or `observability_destination_unreachable` in the log |
| Temperature/max-tokens changes do nothing | Using `LLM_*` instead of `LITELLM_*` |
| ConfigMap edit had no effect | Env vars are read at startup; restart the deployment |
| `DEV_RELOAD` does not pick up changes | Set `WATCHFILES_FORCE_POLLING=true` — `run-local.sh` already does |
| Agent stops early with a "reached the maximum" answer | It hit a guard rail; see the `agent_loop_stopped` log line |
| Connection refused | `LITELLM_PROXY_BASE` unreachable from where the agent runs |
| Agent stops at startup naming a function, skill or hook | `agent.yaml` names something that is not there: a `functions:` entry missing from `app/tools.py`, a skill folder not COPYed, a typo in `hooks:` |
| Import errors for a package you added | The runtime image has only the runtime and the framework; list extra packages in `requirements.txt`; the next `./run-local.sh` (and the pipeline) builds them in |