platform/agent-harness-spec
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: initial harness spec v1.0.0 + reference template

Browse Source 
Adds the normative contract every platform agent must conform to, plus a
working reference template under template/ that ticks every box out of
the box.

Spec is derived from the production agents shipped in
cjot-backstage-az/agents/ on branch sandbox/jonathan:
  - agent-registry, agent-factory, decomposer, discovery-agent,
    golden-path, modernization-factory, modernization-factory-v2,
    policy-transformer, scaffold-agent, support-intake-agent, the-watcher

Contents:
  - SPEC.md          normative contract (13 sections + conformance checklist)
  - CHANGELOG.md     spec versioning (1.0.0)
  - docs/
      registration.md   self-registration with the agent-gateway
      observability.md  OTel logs/metrics/traces wiring
      manifest.md       /.well-known/agent.json schema + AgentSkill
                        serialisation (pydantic vs protobuf)
      kubernetes.md     k8s deployment shape, mandatory cross-refs,
                        per-namespace agents, resource sizing
      deployment.md     .image-version, ACR build, deploy scripts, rollback
      deviations.md     tracked debts against the spec
  - template/
      .env.local.example, .gitignore, .image-version (0.1.0),
      Dockerfile (python:3.12-slim, non-root UID 1001, HEALTHCHECK),
      requirements.txt (harness floor + optional LLM stack),
      app/agent.py (Starlette entry, lifespan + self-registration,
                    defensive _skill_dict for pydantic vs protobuf),
      app/logging_setup.py (canonical OTel log bridge — copy verbatim),
      app/metrics.py (meter + example counter/histogram),
      app/skills.py (AGENT_CONFIG + AgentSkill list),
      k8s/configmap.yaml + deployment.yaml (Deployment + Service,
                                            OTel annotations + 6-var env block,
                                            agents-sa + agents-kv-spc bindings),
      scripts/deploy.sh (auto-bump + az acr build + apply + rollout),
      scripts/full-deploy.sh (preflight + deploy + post-deploy smoke),
      scripts/deploy-local.sh (docker/podman + .env.local)
This commit is contained in:
Jonathan Boniface
2026-06-09 15:54:18 +01:00
parent c87dd3fa55
commit e6f10ba8c9
25 changed files with 1933 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

133
docs/deployment.md Normal file
View File

@@ -0,0 +1,133 @@
# Build & deployment
> Normative requirements: [SPEC.md §10](../SPEC.md#10-build--deployment-must)
Every agent ships three deploy scripts under `scripts/` that wrap the same build → push → apply → roll-out pipeline at three layers of fidelity.
## `.image-version`
A single-line file at the agent root:
```
1.0.3
```
Conventions:
- Semver only.
- **MUST** match `OTEL_RESOURCE_ATTRIBUTES`' `service.version` value at all times.
- Bumped whenever a code change ships. The patch number is auto-incremented by `scripts/deploy.sh` when no `--tag` is given.
- Pure image-version-only commits are reserved for *redeploy reruns* (e.g. rebuilding without code changes to refresh dependencies). They **SHOULD** be rare.
## ACR
All images are pushed to:
```
bstagecjotdevacr.azurecr.io/<agent-name>:<X.Y.Z>
bstagecjotdevacr.azurecr.io/<agent-name>:latest
```
Both tags are pushed every build so that pinning to `:latest` works for sandbox/dev clusters and pinning to `:<version>` works for prod clusters.
## `scripts/deploy.sh`
The fast path. Used for routine redeploys.
```bash
#!/bin/bash
set -euo pipefail
SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
ACR_NAME="bstagecjotdevacr"
APP_NAME="<agent-name>"
NAMESPACE="agents"
VERSION_FILE="${SCRIPT_DIR}/../.image-version"
TAG=""
while [[ "$#" -gt 0 ]]; do
case $1 in
--tag) TAG="$2"; shift ;;
*) echo "unknown arg: $1"; exit 2 ;;
esac
shift
done
if [ -z "$TAG" ]; then
CURRENT=$(cat "$VERSION_FILE" 2>/dev/null || echo "1.0.0")
IFS='.' read -r MAJOR MINOR PATCH <<< "$CURRENT"
PATCH=$((PATCH + 1))
TAG="${MAJOR}.${MINOR}.${PATCH}"
echo "$TAG" > "$VERSION_FILE"
fi
IMAGE="${ACR_NAME}.azurecr.io/${APP_NAME}:${TAG}"
echo "Building ${IMAGE} ..."
az acr build --registry "$ACR_NAME" \
--image "${APP_NAME}:${TAG}" \
--image "${APP_NAME}:latest" \
"${SCRIPT_DIR}/.."
echo "Applying manifests ..."
kubectl apply -f "${SCRIPT_DIR}/../k8s/configmap.yaml"
kubectl apply -f "${SCRIPT_DIR}/../k8s/deployment.yaml"
kubectl -n "$NAMESPACE" rollout status "deployment/${APP_NAME}" --timeout=180s
```
Behaviour:
- Auto-bumps the patch on `.image-version` if `--tag` is not given.
- Rebuilds and re-pushes both `:<tag>` and `:latest`.
- Applies `configmap.yaml` and `deployment.yaml` in order.
- Blocks until the rollout completes (or fails) within 180 s.
## `scripts/full-deploy.sh`
The "first time" path. Adds preflight + post-deploy smoke.
Preflight (refuse to continue if any check fails):
- `kubectl` available and pointed at a cluster.
- `az` available and logged in.
- Target namespace exists.
- ServiceAccount `agents-sa` exists in the namespace.
- SecretProviderClass `agents-kv-spc` exists in the namespace.
After `rollout status` succeeds:
- `kubectl exec deployment/<agent> -- curl -fs http://localhost:<port>/health`
- `kubectl exec deployment/<agent> -- curl -fs http://localhost:<port>/.well-known/agent.json | jq -e '.name'`
Failures abort with a non-zero exit code so CI can block.
## `scripts/deploy-local.sh`
For dev loop on a workstation without cluster access.
- Builds the image locally with `docker` or `podman`.
- Reads `.env.local` (sibling of `.env.local.example`).
- Runs the container with `-p <port>:<port>` and `--env-file`.
Useful for iterating on prompt changes without touching the cluster.
## CI integration
The platform's GitHub Actions / Gitea Actions runners invoke `scripts/deploy.sh --tag $GITHUB_SHA` (or similar) when an agent's directory changes on the deploy branch. Pull requests trigger a build-only run that pushes a `:<sha>` tag without applying manifests.
## Rolling back
There is no automated rollback. To roll back:
```bash
kubectl -n agents set image deployment/<agent-name> \
<agent-name>=bstagecjotdevacr.azurecr.io/<agent-name>:<previous-version>
kubectl -n agents rollout status deployment/<agent-name> --timeout=180s
```
If the rollback ships a different `service.version`, also revert `OTEL_RESOURCE_ATTRIBUTES` in the ConfigMap or the dashboards will misattribute. Prefer a forward-fix (new patch version) over a rollback whenever possible.
## Re-registration after redeploy
The agent re-registers with the gateway on every pod start. No manual step is required after a rollout. If the gateway pod is also restarting at the same time, expect 1–2 `WARNING`s in the agent log before registration succeeds — see [registration.md](registration.md#failure-modes--resolution).

24
docs/deviations.md Normal file
View File

@@ -0,0 +1,24 @@
# Known deviations
This file tracks where existing agents in `cjot-backstage-az/agents/` deviate from the [SPEC](../SPEC.md). Each row is a tracked debt; deviations should converge over time.
| Agent | Deviation | Spec clause | Notes |
|---|---|---|---|
| `agent-registry` | Uses FastAPI, not Starlette | §3.2 | Grandfathered — registry pre-dates the spec. |
| `support-intake-agent` | Uses FastAPI, not Starlette | §3.2 | Grandfathered. |
| `agent-factory` | Uses `agent-factory-sa` ServiceAccount | §9.3 | Has its own Key Vault binding (`agent-factory-kv-spc`) for build-time credentials. |
| `sonarqube-mcp` | Uses `sonarqube-mcp-kv-spc` SecretProviderClass | §9 (Mandatory cross-references) | Domain-specific Key Vault for SonarQube tokens. |
| `golden-path` | Listens on port 8000 (not 8080) | §3.5 (allowed) | Port 8000 is permitted; flagged here only for awareness. |
| `modernization-factory-v2` | Runs in namespace `modernization-factory` | §9.2 | Namespace separation by domain; `service.namespace` and `AGENT_SELF_URL` are correctly aligned. |
| `documentor-agent` | Skeleton only (no `app/` shipped) | All | Placeholder; not yet a deployable agent. |
## How to add a row
When you knowingly merge an agent that violates a spec clause, add a row here with:
- The agent's directory name.
- A one-line description of the deviation.
- The clause number(s) in `SPEC.md`.
- A short rationale or follow-up plan.
When you fix a deviation, remove the row in the same PR that closes it.

198
docs/kubernetes.md Normal file
View File

@@ -0,0 +1,198 @@
# Kubernetes deployment shape
> Normative requirements: [SPEC.md §9](../SPEC.md#9-kubernetes-manifests-must)
Every agent ships at minimum two YAML files under `k8s/`:
- `configmap.yaml` — non-secret config (URLs, log level, port, etc.).
- `deployment.yaml` — `Deployment` + `Service` (concatenated with `---`).
The reference template under [`../template/k8s/`](../template/k8s/) is a copy-and-fill skeleton.
## ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: <agent-name>-config
namespace: agents
data:
# Required — see docs/registration.md
REGISTRY_URL: "http://agent-gateway.agents.svc.cluster.local"
AGENT_SELF_URL: "http://<agent-name>.agents.svc.cluster.local"
# Standard knobs
LOG_LEVEL: "INFO"
PORT: "8000"
# Agent-specific (LLM, cache, feature flags, etc.)
AZURE_OPENAI_DEPLOYMENT: "gpt-4o"
AZURE_OPENAI_API_VERSION: "2024-08-01-preview"
```
The agent's container **MUST** consume the ConfigMap via `envFrom: configMapRef`. Inline `env:` is reserved for OTel vars and `secretKeyRef` references.
## Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: <agent-name>
namespace: agents
labels:
app: <agent-name>
component: agent
spec:
replicas: 1
selector:
matchLabels:
app: <agent-name>
template:
metadata:
labels:
app: <agent-name>
azure.workload.identity/use: "true"
annotations:
instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
instrumentation.opentelemetry.io/container-names: "<agent-name>"
spec:
serviceAccountName: agents-sa
containers:
- name: <agent-name>
image: bstagecjotdevacr.azurecr.io/<agent-name>:latest
imagePullPolicy: Always
ports:
- name: http
containerPort: 8000
protocol: TCP
envFrom:
- configMapRef:
name: <agent-name>-config
env:
# ─── OTel (mandatory) ───────────────────────────────────────────
- name: OTEL_SERVICE_NAME
value: "<agent-name>"
- name: OTEL_RESOURCE_ATTRIBUTES
value: "service.namespace=agents,service.version=1.0.0,deployment.environment=prod,agent.type=<role>"
- name: OTEL_LOGS_EXPORTER
value: "otlp"
- name: OTEL_METRICS_EXPORTER
value: "otlp"
- name: OTEL_METRIC_EXPORT_INTERVAL
value: "15000"
- name: OTEL_SEMCONV_STABILITY_OPT_IN
value: "http"
# ─── Secrets from agents-kv-sync (Key Vault sync) ───────────────
- name: AZURE_OPENAI_ENDPOINT
valueFrom:
secretKeyRef:
name: agents-kv-sync
key: azure-openai-endpoint
- name: AZURE_OPENAI_API_KEY
valueFrom:
secretKeyRef:
name: agents-kv-sync
key: azure-openai-api-key
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /health
port: http
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: agents-kv-spc
---
apiVersion: v1
kind: Service
metadata:
name: <agent-name>
namespace: agents
labels:
app: <agent-name>
spec:
selector:
app: <agent-name>
ports:
- name: http
port: 80
targetPort: 8000
protocol: TCP
type: ClusterIP
```
## Mandatory cross-references
These names are platform conventions and **MUST** be used as-written:
| Name | Kind | Purpose |
|---|---|---|
| `agents` | Namespace | All agents live here unless they have a strong domain reason for their own namespace. |
| `agents-sa` | ServiceAccount | Workload-identity bound to the agents Key Vault. |
| `agents-kv-sync` | Secret | Synced from Key Vault by the SecretProviderClass. Source of `secretKeyRef` values. |
| `agents-kv-spc` | SecretProviderClass | Mounts Key Vault into `/mnt/secrets-store` and rotates the synced Secret. |
| `monitoring/otel-instrumentation` | Instrumentation CR | OTel SDK init container injection. |
## Probes
Tune the timing per agent — these are sane defaults:
| Probe | Purpose | `initialDelaySeconds` | `periodSeconds` | `failureThreshold` |
|---|---|---|---|---|
| `liveness` | Restart hung pod. | 20 | 15 | 3 |
| `readiness` | Hold traffic until manifest is serving. | 10 | 10 | 3 |
Agents with heavy startup work (large LangGraph compile, prompt caches, model warm-ups) **SHOULD** raise `initialDelaySeconds` rather than reduce `failureThreshold`.
## Per-namespace agents
Some agents (e.g. `modernization-factory-v2`) run in their own namespace. When they do:
- The `serviceAccountName` **MUST** be created in that namespace and bound to the same workload identity (`agents-sa` is conventional, even when the namespace differs).
- `OTEL_RESOURCE_ATTRIBUTES`'s `service.namespace` **MUST** match the namespace.
- `AGENT_SELF_URL` **MUST** point to the cross-namespace FQDN (`http://<agent>.<namespace>.svc.cluster.local`); this is allow-listed by the gateway as long as the host ends in `.svc.cluster.local`.
## Resource sizing reference
Observed in production:
| Agent class | CPU req / lim | Mem req / lim |
|---|---|---|
| Stateless small (e.g. `support-intake-agent`) | 100m / 250m | 256Mi / 512Mi |
| Standard LLM agent (`policy-transformer`, `decomposer`) | 250m / 500m | 512Mi / 1Gi |
| Heavy workflow (`golden-path`, `modernization-factory-v1`) | 500m–1 / 1–2 | 1Gi–2Gi / 2Gi–4Gi |
When in doubt start with the *Standard* row and revisit after one week of production traffic.

138
docs/manifest.md Normal file
View File

@@ -0,0 +1,138 @@
# Agent manifest — `/.well-known/agent.json`
> Normative requirements: [SPEC.md §6](../SPEC.md#6-agent-manifest-must)
The agent-gateway discovers agents by fetching `GET {AGENT_SELF_URL}/.well-known/agent.json` after a successful `POST /agents/register-url` (see [registration.md](registration.md)).
## Schema
```jsonc
{
"name": "Policy Transformer", // human-readable, used as dedup key
"version": "1.0.0", // semver, MUST match .image-version
"description": "...", // 1-3 sentences
"url": "/", // base path, almost always "/"
"skills": [ // see "Skills" below
{
"id": "policy_generation", // snake_case, stable identifier
"name": "Policy Generation", // human-readable
"description": "...", // 1-2 sentences
"tags": ["policy", "generate"], // free-form filters
"examples": [ // example natural-language prompts
"Generate a Kyverno policy that requires labels"
]
}
],
"capabilities": {
"streaming": false, // SSE/WebSocket support
"conversational": false, // multi-turn chat support
"direct_api": true // synchronous request/response
}
}
```
## Skills
Skills declare what the agent can do. They are surfaced to:
- The Backstage agent catalogue page,
- The agent-gateway's discovery API (so other agents can route by capability),
- LLM tool registries (so the platform-orchestrator can pick a skill at runtime).
### Defining skills with `a2a-sdk`
Most agents use the `a2a-sdk`'s `AgentSkill` type:
```python
from a2a.types import AgentSkill
POLICY_TRANSFORMER_SKILLS = [
AgentSkill(
id="policy_generation",
name="Policy Generation",
description=(
"Generate production-ready Kyverno, Azure Policy, or "
"OPA/Gatekeeper policies from a natural-language description."
),
tags=["policy", "generate", "kyverno", "azure", "gatekeeper", "opa"],
examples=[
"Generate a Kyverno policy that requires all pods to have app and owner labels",
],
),
# ...
]
```
### Defensive serialisation
`AgentSkill` is currently exposed as a **protobuf `Message`** in `a2a-sdk` runtime images, even though older type stubs suggest pydantic. Calling `s.model_dump()` blindly raises `AttributeError: model_dump` and the manifest endpoint returns 500 — which then causes the gateway's `register-url` call to fail with `HTTP 400 — Failed to fetch agent card ... HTTP 500`.
The agent **MUST** serialise skills with a defensive helper:
```python
def _skill_dict(s) -> dict:
if hasattr(s, "model_dump"):
return s.model_dump()
if hasattr(s, "DESCRIPTOR"):
try:
from google.protobuf.json_format import MessageToDict
return MessageToDict(s, preserving_proto_field_name=True)
except Exception:
pass
if hasattr(s, "dict") and callable(getattr(s, "dict")):
try:
return s.dict()
except Exception:
pass
return {
"id": getattr(s, "id", None),
"name": getattr(s, "name", None),
"description": getattr(s, "description", ""),
"tags": list(getattr(s, "tags", []) or []),
"examples": list(getattr(s, "examples", []) or []),
}
```
The canonical copy lives in [`../template/app/agent.py`](../template/app/agent.py).
## The route handler
```python
async def agent_manifest(request: Request) -> JSONResponse:
"""GET /.well-known/agent.json — discovery manifest consumed by agent-gateway."""
return JSONResponse({
"name": AGENT_CONFIG["name"],
"version": AGENT_CONFIG["version"],
"description": AGENT_CONFIG["description"],
"url": "/",
"skills": [_skill_dict(s) for s in AGENT_SKILLS],
"capabilities": AGENT_CONFIG["capabilities"],
})
```
## Capability flags
| Flag | Meaning | Set to `true` when… |
|---|---|---|
| `streaming` | The agent supports server-sent events on at least one route. | You expose `text/event-stream` responses. |
| `conversational` | Multi-turn chat with session state. | You hold `session_id` → state in memory or Redis. |
| `direct_api` | Synchronous request → response. | The default for most agents. |
These are advisory — the gateway uses them to pick UI affordances, not to gate routing.
## Naming and dedup
The gateway normalises `name` to lower-hyphen-case before storing the registration row. `Policy Transformer` and `policy-transformer` collide; pick one form and stick to it. The registration row's `name` becomes the path segment in `/proxy/agent-gateway/{agent_name}/{path}`.
If two agents register with the same normalised name, the latest registration wins. Versioned agents (e.g. `Modernization Factory` vs `Modernization Factory v2`) **MUST** use distinct `name` strings.
## Validation
A simple smoke test you can run inside the pod:
```bash
kubectl exec deployment/<agent> -c <container> -- \
curl -fs http://localhost:<port>/.well-known/agent.json | jq -e '.name and .version and (.skills|type=="array")'
```
Exit code 0 means the manifest is structurally valid. The full validation lives in the gateway and runs automatically on every `register-url` call.

209
docs/observability.md Normal file
View File

@@ -0,0 +1,209 @@
# Observability — OpenTelemetry wiring
> Normative requirements: [SPEC.md §7](../SPEC.md#7-opentelemetry-wiring-must)
The platform runs the OpenTelemetry Operator. Every agent gets the SDK injected as an init container; the agent code itself does **not** install or configure SDK exporters. This document explains what is provided for free, what the agent must wire by hand, and how the signals flow to the cluster collectors.
## Pipeline
```mermaid
flowchart LR
A[Agent pod] -->|OTLP gRPC| T1[Tier-1 collector]
T1 -->|metrics| P[Prometheus RemoteWrite]
T1 -->|logs| L[Loki]
T1 -->|traces| TP[Tempo]
T1 -->|spanmetrics| P
```
- **Tier-1 collector** runs as a DaemonSet, decorates spans/logs with `k8sattributes` (pod, namespace, node), and forwards to tier-2.
- **Tier-2 collector** generates spanmetrics from traces, exports OTLP traces to Tempo, OTLP logs to Loki, and Prometheus remote-writes metrics.
- The agent only needs OTLP gRPC reachability to `tier-1` — everything else is the operator's job.
## What the operator gives you for free
The `Instrumentation` CR `monitoring/otel-instrumentation` injects:
- **Tracer/Meter/Logger providers** with OTLP exporters configured.
- **Auto-instrumented HTTP server** (`opentelemetry-instrumentation-asgi` or `-fastapi`).
- **Auto-instrumented outbound HTTP** (`urllib3`, `httpx`, `requests`).
- **Logger provider** with `BatchLogRecordProcessor` — but **does not** attach a `LoggingHandler` to Python's stdlib root logger. That is the agent's job.
## What the agent must wire by hand
### 1. The pod template annotations
```yaml
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
instrumentation.opentelemetry.io/container-names: "<container-name>"
```
`container-names` **MUST** match the container `name`. If you have multiple containers (sidecar, etc.), only list the agent container.
### 2. The standard env block
These six env vars are mandatory:
```yaml
- name: OTEL_SERVICE_NAME
value: "<agent-name>"
- name: OTEL_RESOURCE_ATTRIBUTES
value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
- name: OTEL_LOGS_EXPORTER
value: "otlp"
- name: OTEL_METRICS_EXPORTER
value: "otlp"
- name: OTEL_METRIC_EXPORT_INTERVAL
value: "15000"
- name: OTEL_SEMCONV_STABILITY_OPT_IN
value: "http"
```
#### `OTEL_SERVICE_NAME`
The unique service identity. Used as the `service.name` resource attribute on every span/metric/log. **MUST** match the agent's lower-hyphen identifier.
#### `OTEL_RESOURCE_ATTRIBUTES`
Comma-separated `key=value` pairs merged into the resource. Required keys:
- `service.namespace` — the Kubernetes namespace.
- `service.version` — semver string. **MUST** match the contents of the agent's `.image-version` file.
- `deployment.environment` — `prod` / `dev` / `local`.
- `agent.type` — stable lower-hyphen identifier (`policy-transformer`, `golden-path`, `registry`, etc.). This is the primary metric/log filter in dashboards.
#### `OTEL_METRIC_EXPORT_INTERVAL`
`15000` ms (15 s) is platform-wide. Faster than this generates noise; slower hides incidents.
#### `OTEL_SEMCONV_STABILITY_OPT_IN`
`http` opts the SDK into the **stable** HTTP semantic conventions:
- `http.server.request.duration` (histogram)
- `http.response.status_code` (attribute)
Without this flag the SDK emits the legacy `http.server.duration` + `http.status_code`, which the platform dashboards no longer query.
### 3. The logging bridge
The OTel Logger provider exists but isn't connected to stdlib `logging`. The agent **MUST** ship `app/logging_setup.py` and call `configure_otlp_log_handler()` once at module import time in `app/agent.py`.
The canonical implementation is in [`../template/app/logging_setup.py`](../template/app/logging_setup.py). Copy it verbatim — it is intentionally short and idempotent:
```python
def configure_otlp_log_handler(level: int = logging.INFO) -> None:
try:
from opentelemetry._logs import get_logger_provider
from opentelemetry.sdk._logs import LoggingHandler
except ImportError:
return
provider = get_logger_provider()
if provider is None or type(provider).__name__ == "NoOpLoggerProvider":
return
root = logging.getLogger()
if any(isinstance(h, LoggingHandler) for h in root.handlers):
return
root.addHandler(LoggingHandler(level=level, logger_provider=provider))
if root.level == logging.NOTSET or root.level > level:
root.setLevel(level)
```
The function is a no-op outside the cluster (the SDK packages aren't installed), so the same call site works locally.
### 4. Custom metrics
Custom metrics live in `app/metrics.py`. The shape:
```python
from opentelemetry import metrics
_meter = metrics.get_meter("<agent-name>", "<X.Y.Z>")
work_done = _meter.create_counter(
"agent.<type>.runs",
unit="1",
description="Work units processed by outcome.",
)
work_duration = _meter.create_histogram(
"agent.<type>.duration",
unit="s",
description="End-to-end work latency.",
)
```
Naming rules (mandatory):
- Prefix all metric names with `agent.<type>.` where `<type>` matches `agent.type` in `OTEL_RESOURCE_ATTRIBUTES`.
- Counters: noun (`runs`, `tokens`, `errors`).
- Histograms: noun + `.duration` (seconds).
- Use `unit="1"` for counts, `unit="s"` for seconds, `unit="By"` for bytes.
Recording from a route handler:
```python
from .metrics import work_done, work_duration
async def handler(request):
started = time.perf_counter()
try:
outcome = await do_the_thing(...)
work_done.add(1, {"outcome": "success"})
return JSONResponse(outcome)
except ValueError as exc:
work_done.add(1, {"outcome": "bad_request"})
raise
except Exception:
work_done.add(1, {"outcome": "error"})
raise
finally:
work_duration.record(time.perf_counter() - started)
```
Prefer **inline metric calls at every return path** over outer `try/finally` wrappers — it avoids indentation churn and reads cleaner. Use `try/finally` only for histograms that always need to record regardless of outcome.
Do **not** create `MeterProvider` instances or call `set_meter_provider()` — the operator owns the global provider.
### 5. LangChain / LLM instrumentation
Agents that call an LLM via LangChain **MUST** wire OpenInference:
```python
from openinference.instrumentation.langchain import LangChainInstrumentor
LangChainInstrumentor().instrument()
```
Call this **once**, before any `AzureChatOpenAI(...)` (or other chat-model) construction. It emits OpenInference spans with token counts, prompt/response payloads, and tool calls — visible in Tempo and surfacable as LLM dashboards.
Multiple `instrument()` calls are safe but log a warning. If you compose modules across `__main__` and `app.agent`, expect to see the warning once per import path; this is harmless.
## Verifying signals end-to-end
After deploying, check each signal flows:
```bash
# Logs — should appear in Loki within ~30s
kubectl logs deployment/<agent> -c <agent-name> | head
# Metrics — query Prometheus for the meter name
curl -s "$PROM/api/v1/query?query=agent_<type>_runs_total" | jq
# Traces — search Tempo for service.name
curl -s "$TEMPO/api/search?tags=service.name=<agent-name>" | jq
# LLM token usage (if instrumented)
curl -s "$PROM/api/v1/query?query=gen_ai_client_token_usage_sum" | jq
```
## Common pitfalls
| Pitfall | Symptom | Fix |
|---|---|---|
| Forgot the logging bridge | Logs appear in `kubectl logs` but not Loki. | Add `configure_otlp_log_handler()` in `app/agent.py`. |
| `OTEL_SEMCONV_STABILITY_OPT_IN` missing | Dashboards show no HTTP server latency. | Add the env var; redeploy. |
| `OTEL_RESOURCE_ATTRIBUTES` `service.version` drifts from `.image-version` | Dashboards group by version are wrong. | Bump both together; consider templating in `deployment.yaml`. |
| `LangChainInstrumentor()` called after LLM init | Spans missing token counts. | Move `.instrument()` to before `AzureChatOpenAI(...)`. |
| Custom metric names not prefixed `agent.<type>.` | Metric collides with another agent's. | Rename and bump minor version. |

71
docs/registration.md Normal file
View File

@@ -0,0 +1,71 @@
# Self-registration with the agent-gateway
> Normative requirements: [SPEC.md §5](../SPEC.md#5-self-registration-with-the-gateway-must)
Every agent self-registers with the agent-gateway on startup. The gateway is the single Service in front of the registry catalogue and the proxy through which other agents (and Backstage) reach this agent's endpoints.
## Topology
```mermaid
sequenceDiagram
participant A as Agent pod
participant G as agent-gateway
Note over A: lifespan startup
A->>A: asyncio.sleep(2s)<br/>(let uvicorn bind)
A->>G: POST /agents/register-url<br/>{"endpoint": AGENT_SELF_URL}
G->>A: GET {AGENT_SELF_URL}/.well-known/agent.json
A-->>G: 200 OK + manifest
G-->>A: 201 Created (registration row)
```
## Configuration
Two environment variables drive the flow. Both **MUST** be set in the agent's ConfigMap.
| Var | Example | Purpose |
|---|---|---|
| `REGISTRY_URL` | `http://agent-gateway.agents.svc.cluster.local` | The gateway's base URL. |
| `AGENT_SELF_URL` | `http://policy-transformer.agents.svc.cluster.local` | The URL the gateway should call back on. |
Notes:
- The registry validates `AGENT_SELF_URL` against an SSRF allow-list. The host **MUST** end in `.svc.cluster.local` or `.kyndemo.live` and use `http` or `https`. Anything else is rejected with `400 Bad Request`.
- Cross-namespace `AGENT_SELF_URL` values are fine as long as the URL is resolvable in cluster DNS.
- The trailing path on `AGENT_SELF_URL` **MUST** be `/` (or empty); the gateway appends `/.well-known/agent.json` itself.
## The registration call
The canonical implementation lives in [`template/app/agent.py`](../template/app/agent.py). Key invariants:
- Initial sleep of 2 seconds **before** the first attempt — uvicorn needs time to bind the listen socket and the manifest route.
- 3 attempts total, 5 seconds between attempts, 10-second timeout per request.
- HTTP 200 *or* 201 are both treated as success.
- Logging:
- `INFO` once on success.
- `WARNING` per failed attempt with `HTTP <code>` + body excerpt.
- `ERROR` once on final failure (after the third attempt).
- The task is created via `asyncio.create_task` inside the Starlette `lifespan`, so it does **not** block uvicorn from accepting traffic. A failed registration **MUST NOT** crash the agent.
## What the gateway does next
After the agent's `POST /agents/register-url` succeeds, the gateway performs:
1. `GET {endpoint}/.well-known/agent.json` (hardcoded path).
2. Parses the manifest into an `AgentRegistration` row keyed by the manifest's `name`, normalised to lower-hyphen-case.
3. Registers the agent in the catalogue.
4. Routes inbound traffic from `/proxy/agent-gateway/{agent_name}/{path}` to `{endpoint}/{path}`.
Therefore the manifest at `AGENT_SELF_URL` **MUST** be reachable and **MUST** match the schema in [docs/manifest.md](manifest.md).
## Failure modes & resolution
| Symptom | Cause | Fix |
|---|---|---|
| `HTTP 400 — Failed to fetch agent card from .../.well-known/agent.json: HTTP 404` | The agent has no manifest route. | Add `Route("/.well-known/agent.json", agent_manifest, methods=["GET"])`. |
| `HTTP 400 — ... HTTP 500` | The manifest route raises (e.g. `AgentSkill.model_dump` `AttributeError`). | Use the defensive `_skill_dict()` helper from the template. |
| `HTTP 400 — endpoint URL not allowed` | `AGENT_SELF_URL` host does not end in `.svc.cluster.local` or `.kyndemo.live`. | Fix the ConfigMap. |
| Three `WARNING`s + final `ERROR` but agent serves traffic fine | Gateway pod was restarting during agent startup. | Restart the agent pod once the gateway is `Ready`. Re-registration on next pod restart will recover. |
## Local development
When running outside the cluster, leave `REGISTRY_URL` unset (or set it to `""`) — the agent will log `REGISTRY_URL not set — skipping self-registration` and continue.