# expense-policy-agent An A2A-protocol agent on Google's ADK, built on the platform's shared runtime image (`agent-runtime-adk`). The runtime handles everything that is the same for every agent — building the agent, serving the A2A protocol, reaching the LLM gateway, correlation headers, structured logging, tracing, and the agent-loop guard rails. What is left in this repository is the part that is yours: `agent.yaml` (what the agent is), the instructions in `prompt.md`, its schemas, skills, tools and hooks. There is no `app/agent.py`: the runtime builds the agent from `agent.yaml`. - **Model**: claude-sonnet-4-5 - **Cloud / region**: local / localhost ## Quick start ```bash ./run-local.sh # build if needed, run on :8080 DEV_RELOAD=true ./run-local.sh # same, with hot reload ``` `.env` (generated at creation, gitignored) must hold `LITELLM_PROXY_BASE` and `LITELLM_MASTER_KEY`. Then, in a second terminal: ```bash pip install -r tests/requirements.txt python tests/test_chatbot.py --url http://localhost:8080 ``` `tests/test_chatbot.py` is an interactive chat client. It speaks the same A2A wire format the platform's Temporal worker uses to call this agent in production, so what you see locally is what a workflow will get. `run-local.sh` builds the image from the Dockerfile — the runtime image plus this agent's files — and runs it with the project bind-mounted over `/agent`, so an edit is live on the next start. With `DEV_RELOAD=true` the server restarts on any change in the project — `app/`, `schemas/`, `prompt.md`, `agent.yaml` — which are the files you will iterate on most. It rebuilds the image only when the Dockerfile, the runtime image or `requirements.txt` changes. The runtime image comes from the dev kit's GHCR (`RUNTIME_REGISTRY`, or `RUNTIME_IMAGE` for one outright; `localhost` for a runtime built from source). It is also the closest local equivalent to a deployed pod. In Kubernetes, `deployment.yaml` does `envFrom: configMapRef`, which injects every key of `kubernetes/configmap.yaml` into the container. `run-local.sh` reads that same file and passes each key as `-e`, so a setting you change in the ConfigMap takes effect in both places. A plain `podman run` with hand-written `-e` flags drifts from the deployed configuration silently, and the drift only shows up after deploy. Secrets are not in the ConfigMap; they come from `.env` locally and from the Secrets in `kubernetes/secretnames.yaml` in production. ## Running it against the platform The steps above run the agent on its own. To use it in a workflow it also has to be reachable by the platform and registered in the agent catalog. **1. Put your gateway credentials in `.env`** (once per checkout): ```bash cp .env.example .env # then fill in LITELLM_PROXY_BASE and LITELLM_MASTER_KEY ``` `run-local.sh` sources `.env` for secrets and takes everything else from `kubernetes/configmap.yaml`. **2. Start it on the platform's network:** ```bash PORT_HOST=10021 \ CONTAINER_NETWORK=agent-builder-local-dev-kit_agent-builder-network \ ./run-local.sh ``` `CONTAINER_NETWORK` is what makes the agent reachable by the API and the Temporal worker; without it the agent runs fine but nothing can call it. Pick a `PORT_HOST` nobody else is using — it is only for your own testing. **3. Check it is up, from both sides:** ```bash curl -s localhost:10021/health # from your machine python tests/test_chatbot.py --url http://localhost:10021 ``` **4. Register it in the agent catalog.** In the platform UI, go to **Access Control → Agents → Discover**, and enter: ``` http://:8080 ``` That is the *container* name on the shared network and the *container's* port (8080) — not `localhost` and not your `PORT_HOST`. The platform resolves that name over the podman network; `localhost` from inside the API container means the API itself. `` is this project's directory name, which is what `run-local.sh` names the container and registers as its DNS alias. The agent then appears in the catalog and can be dropped into a workflow. **Redeploying after a change.** Edit, then re-run step 2 — the script rebuilds and replaces the container. The registration is by URL and survives, so you do not need to re-register. For prompt-only changes, add `DEV_RELOAD=true` and skip the rebuild entirely. ## Configuration ### The agent itself: `agent.yaml` ```yaml name: my-agent description: What this agent does, for someone choosing it from the catalog. version: 1.0.0 # #.#.# or DEV -- the CI releases nothing else framework: adk # must match the runtime image in the Dockerfile tags: [adk] capabilities: streaming: true push_notifications: false skills: - id: my_agent name: My Agent description: What this skill does. tags: [] examples: [] prompt: prompt.md contract: input: schemas/input.json output: schemas/output.json functions: - app.tools:search hooks: app.business_logic ``` `agent.yaml`, at the root of the project, is the agent: the runtime reads it at startup and builds the agent from it — its instructions, contract, tools, skills and hooks — and it is the source of truth for everything the agent card says. None of it is read from environment variables. Skills are listed either as folders under `skills/` (instructions and tools the model loads when it needs them -- see `skills/README.md`) or as plain entries that are only advertised. Skills never change the agent's input and output: the first skill carries its schemas. To change any of it, edit the file, redeploy, and refresh the agent in the catalog. A new `name` registers as a new agent. Every field is described in the platform's developer guide (Help → Developer Guide → agent.yaml reference). ### The runtime it is built on: the Dockerfile The Dockerfile builds `FROM` the shared runtime image for this framework, named with its runtime version (`.../agent-runtime-adk:`), and copies this agent's files into `/agent`. Moving to a newer runtime is a change of that line plus a new agent version — the image changes, so the version must. ### Releasing a version: `RELEASE_NOTES.md` The CI takes the version from `agent.yaml` and refuses to release it unless `RELEASE_NOTES.md` has an entry for it, and a released version cannot be released again. So a release is three edits: bump `version` in `agent.yaml`, add its entry at the top of `RELEASE_NOTES.md`, redeploy. ```markdown ## [1.1.0] - 2026-10-01 ### Added - New capability or tool the agent now supports. ### Fixed - Bug fixes included in this release. ``` `version: DEV` is the exception: it needs no entry and may be overwritten, for development. ### Everything else: `kubernetes/configmap.yaml` `kubernetes/configmap.yaml` is the contract for every other setting — each one is documented there, in the file that actually ships. This is the summary. ### Required | Variable | Description | |---|---| | `LITELLM_MODEL` | Model ID (e.g. `gpt-5.4`, `claude-sonnet-4-6`) | | `LITELLM_PROXY_BASE` | LiteLLM gateway URL — on a cluster, set by the platform for every agent | | `LITELLM_MASTER_KEY` | LiteLLM gateway key — on a cluster, set by the platform | ### Optional | Variable | Default | Description | |---|---|---| | `LITELLM_TEMPERATURE` | `0.0` | Model temperature | | `LITELLM_MAX_TOKENS` | `16384` | Max response tokens | | `AGENT_GUARDRAILS_ENABLED` | `true` | Enforce the loop bounds below | | `AGENT_MAX_ITERATIONS` | `25` | Model calls per run: the agent's, or one sub-agent's | | `AGENT_MAX_REQUEST_MODEL_CALLS` | `100` | Model calls in the whole request, the sub-agents the model calls included (default 4 × `AGENT_MAX_ITERATIONS`) | | `AGENT_MAX_DURATION_SECONDS` | `1500` | Wall clock per request (25 minutes) | | `AGENT_MAX_TOKENS` | `0` | Total tokens per request; `0` = no limit | | `OBSERVABILITY_TYPE` | unset | Local runs only: `otel` (with `OTEL_EXPORTER_OTLP_ENDPOINT`, the OpenTelemetry Collector) or `none`. On a cluster the platform sets it for every agent | | `OBSERVABILITY_CAPTURE_CONTENT` | `true` | `false` keeps this agent's prompts, responses, tool data and approval comments out of traces; an agent may record less than the platform, never more | | `MCP_GATEWAY_URL` | | The MCP gateway, for the MCP tools declared under `mcp:` in `agent.yaml`. Local runs only: on a cluster the platform sets it | | `LOG_LEVEL` | `INFO` | Applies to every logger; prompts and payloads are logged only at `DEBUG` | | `LOG_FORMAT` | `json` | `json` for a log store, `console` for reading on your machine | | `AGENT_MAX_CONCURRENT_REQUESTS` | `100` | Requests worked on at once; one more gets HTTP 503, which Temporal retries | | `AGENT_INPUT_GUARDRAIL` | | A guardrail on the LLM gateway that screens each request before the agent runs | | `AGENT_EGRESS` | `log` | `log`, `enforce` or `off`: what happens to a connection to a host that is not one of the agent's endpoints | | `AGENT_EGRESS_ALLOW` | | More hosts the agent may connect to, comma-separated | | `PORT` | `8080` | | > The `LITELLM_` prefix on the model settings is load-bearing. The unprefixed > `LLM_TEMPERATURE` / `LLM_MAX_TOKENS` names are read by a different code path > that does not configure this agent's model, so setting them is silently > ignored — no error, just an agent that stays at the defaults. > On a cluster the model gateway, the MCP gateway and telemetry are the > platform's, the same for every agent: the Deployment reads the platform's > ConfigMap `agent-platform-connections` after this agent's own, so they are > not in `kubernetes/`; traces go through the platform's OpenTelemetry > Collector, which holds any keys. Locally, `.env` > holds them, and traces go to the dev kit's collector the same way. A > telemetry problem never stops the agent: the log > says why (`observability_off`). The in-app help (Architecture & design > > Telemetry) explains the path. ### Which configuration wins Environment variables are read once, at process start, by whichever path started the agent: | How it runs | Configuration source | |---|---| | `./run-local.sh` | `kubernetes/configmap.yaml` + secrets from `.env` | | Deployed pod | `kubernetes/configmap.yaml` via `envFrom`, plus Secrets | `.env` is never copied into the image and is never read by a deployed agent, so nothing you put there can affect production. Conversely, changing a ConfigMap value on a running deployment needs a restart — `kubectl rollout restart` — not just an apply. ## Agent loop guard rails An agent with tools makes as many model calls as the model decides it needs. The runtime bounds that automatically; this agent writes no code for it. Any one of the three limits ends the loop, and the caller gets an answer explaining why rather than an error or a hang. Every run logs its own numbers, enforced or not: ``` agent_loop_completed iterations=4 tokens=18204 stopped_by_guardrail=False ``` That line is how you find the right limits for your agent: watch the typical `iterations` for real work, then set `AGENT_MAX_ITERATIONS` above it with room to spare. The defaults are deliberately loose so they catch runaways — most often a tool bug that keeps telling the model to call it again — without touching an agent that is behaving. Setting `AGENT_GUARDRAILS_ENABLED=false` removes the bounds but keeps the counting and the log line, so you can measure first and enforce afterwards. **An agent with no tools cannot loop.** One request is exactly one model call, and these settings have nothing to bound. ## Customizing ### Instructions Edit `prompt.md`. It is a Markdown file loaded verbatim as the system instruction, and it is where most of an agent's behaviour actually lives. ### Tools Tools are plain Python functions in `app/tools.py`, each listed under `functions:` in `agent.yaml`. The framework derives the schema the model sees from the signature and docstring, so both are part of the interface: ```python # app/tools.py def search(symbol: str) -> dict: """Find every line where `symbol` appears. Args: symbol: the identifier to look for. """ ... ``` ```yaml # agent.yaml functions: - app.tools:search ``` MCP tools are declared under `mcp:` in `agent.yaml` instead, and reached through the platform's MCP gateway; the model sees both kinds the same way. Keep tool results bounded. A tool that can return thousands of rows will fill the context window and drive the loop into the iteration cap; cap the result inside the tool and say in the return value that it was capped, so the model can decide what to do about it. ### Business logic: before and after the model `app/business_logic.py` has four hooks, all already wired up (`hooks:` in `agent.yaml`) and doing nothing until you fill them in. These are the per-call pair: | Hook | Runs | Use it to | |---|---|---| | `before_model` | before each model call | change what is asked — add context to the instructions | | `after_model` | after each reply | inspect it — check its shape, record a metric | Use these when the work must happen on **every** request. Use a **tool** when the model should decide whether and when to do it — tools create the agent loop, callbacks do not. A worked example, reading a file and putting it in front of the model: ```python @lru_cache(maxsize=1) def load_reference() -> str: try: return (Path(__file__).parent.parent / "reference" / "notes.txt").read_text() except OSError: return "" # never fatal: answer without it rather than 500 def before_model(call): if load_reference(): call.add_instruction(load_reference()) ``` `call` is the runtime's `ModelCall`, the same on every framework: `call.add_instruction(text)` before the model, `call.reply_text` after it, and `call.native` for the framework's own arguments. Two things that catch people out: - **`COPY` your data files in the Dockerfile.** `./run-local.sh` bind-mounts the project, so a file you forgot to copy still works locally and is missing in Kubernetes. This is the most common local-passes-deploy-fails bug. - **`before_model` runs once per *model call*, not per request.** In a tool loop that is several times. Cache anything you read from disk. ### Where configuration lives Two files, and they do not overlap: | | holds | read by | |---|---|---| | `kubernetes/configmap.yaml` | every non-secret runtime setting — name, model, guard rails, observability | Kubernetes (`envFrom`), `./run-local.sh` (as `-e`) | | `.env` | secrets, and values specific to your machine | `./run-local.sh` | **The ConfigMap is not a Kubernetes-only file.** That is the thing worth knowing. `./run-local.sh` parses it and passes every key through as `-e`, exactly as `envFrom: configMapRef` does on a cluster. Change the model there once and the local container and the deployed pod agree. `.env` holds only what a ConfigMap cannot: `LITELLM_MASTER_KEY` and `MCP_AGENT_API_KEY` because a ConfigMap is stored unencrypted, and `LITELLM_PROXY_BASE` because the gateway URL is yours and nobody else's. `DEV_RELOAD` is there too, being meaningless once deployed. Precedence is `shell environment` → `.env` → ConfigMap, so ```bash LITELLM_MODEL=claude-sonnet-4-6 ./run-local.sh ``` overrides the model for one run without editing anything. If you are hunting for a setting and it is not in `.env`, it is in the ConfigMap. These values used to be listed in both files; they drifted, and since the ConfigMap wins on both container paths, the `.env` copy was inert — you could edit it, restart, and see no change at all. `.env` is generated for you at creation time and is gitignored. There is no `cp .env.example .env` step. ### The agent's contract Two JSON Schemas describe this agent: `schemas/input.json` says what it accepts, `schemas/output.json` says what it returns, and `contract:` in `agent.yaml` points at them. Both are written when the agent is created, from the schemas supplied then. Both are optional — a schema that was not supplied is left empty and its `contract:` line commented out, and an agent with no contract takes and returns free text. Declaring them buys three things: - The agent card publishes both as `skills[0].inputSchema` / `outputSchema`, so the workflow builder can check this agent's output against the next agent's input *before* anything runs. - A payload that does not match the input schema is refused before a model call is spent, and the caller gets the JSON path that was wrong. - A reply that does not match the output schema earns one more attempt quoting the specific failure. Enforcement sits at the invocation boundary rather than in a hook because a retry needs another model call, and `after_model` cannot cause one. To change the contract, edit the JSON files (and uncomment the `contract:` line for one you add). ### Where your code goes `app/business_logic.py` and `app/tools.py`. The hooks come in two pairs: | Hook | Runs | Receives | |---|---|---| | `pre_process` | once per request | the parsed input payload | | `before_model` | once per **model call** | the runtime's `ModelCall` | | `after_model` | once per **model call** | the runtime's `ModelCall` | | `post_process` | once per request | the parsed output payload | The distinction is the one that bites. A lookup in `before_model` runs four times in a four-step tool loop and you pay for it four times; the same lookup in `pre_process` runs once. Request work goes in the outer pair, per-turn work in the inner pair. ## Testing ```bash python tests/test_chatbot.py --url http://localhost:8080 # interactive python tests/test_chatbot.py --url http://localhost:8080 --test-message "..." ``` Raw protocol checks: ```bash curl http://localhost:8080/.well-known/agent-card.json # discovery; also the k8s probe curl http://localhost:8080/health curl -X POST http://localhost:8080/ \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"message/send","id":"1", "params":{"message":{"parts":[{"type":"text","text":"Hello!"}]}}}' ``` ## Layout ``` expense-policy-agent/ ├── app/ │ ├── __init__.py │ ├── business_logic.py # your four hooks │ └── tools.py # your functions; list them under functions: in agent.yaml ├── schemas/ │ ├── input.json # what the agent accepts (contract.input) │ └── output.json # what it returns (contract.output) ├── skills/ # one folder per skill; see skills/README.md ├── eval/ │ └── cases.yaml # eval suite; python -m kyndryl_agent_runtime.eval ├── kubernetes/ │ ├── configmap.yaml # the configuration contract — start here │ ├── deployment.yaml │ ├── secretnames.yaml # credentials; never in the ConfigMap │ ├── service.yaml │ └── kustomization.yaml ├── tests/ │ ├── a2a_client.py # A2A client over httpx │ ├── test_chatbot.py # interactive chat client │ └── requirements.txt # host-side only; not in the image ├── agent.yaml # the agent: identity, prompt, contract, tools, skills, hooks ├── RELEASE_NOTES.md # one entry per released version; the CI checks it ├── prompt.md # system instruction ├── requirements.txt # extra Python packages, if any (built into the image) ├── run-local.sh # container, configured from the ConfigMap (+ DEV_RELOAD) └── Dockerfile # FROM the runtime image, plus this agent's files ``` ## Deploying Commit and push. CI/CD builds the image and applies `kubernetes/`, replacing ``, ``, and ``. See `docs/agent-deployment/DEPLOYMENT.md` (in the platform repo) for the walkthrough, including the repository secrets that need to be set once. To apply by hand against a cluster you already have credentials for: ```bash kubectl apply -k kubernetes/ ``` ## Troubleshooting | Symptom | Cause | |---|---| | No traces arrive | Look for `observability_off` (a missing key or setting) or `observability_destination_unreachable` in the log | | Temperature/max-tokens changes do nothing | Using `LLM_*` instead of `LITELLM_*` | | ConfigMap edit had no effect | Env vars are read at startup; restart the deployment | | `DEV_RELOAD` does not pick up changes | Set `WATCHFILES_FORCE_POLLING=true` — `run-local.sh` already does | | Agent stops early with a "reached the maximum" answer | It hit a guard rail; see the `agent_loop_stopped` log line | | Connection refused | `LITELLM_PROXY_BASE` unreachable from where the agent runs | | Agent stops at startup naming a function, skill or hook | `agent.yaml` names something that is not there: a `functions:` entry missing from `app/tools.py`, a skill folder not COPYed, a typo in `hooks:` | | Import errors for a package you added | The runtime image has only the runtime and the framework; list extra packages in `requirements.txt`; the next `./run-local.sh` (and the pipeline) builds them in |