# Platform Agent Harness Specification

**Version:** 1.0.0
**Status:** Draft → ratification pending
**Applies to:** All Python agents deployed under `cjot-backstage-az/agents/` and any new agent on the platform.

The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).

---

## 1. Scope

This spec defines the *harness* — the cross-cutting plumbing every agent shares — so that domain logic is the only thing an agent author has to write. It does **not** prescribe how an agent's business logic, prompts, or workflow internals are structured.

In scope:

- HTTP framework, entry point, and required routes
- Self-registration with the agent-gateway / agent-registry
- The `/.well-known/agent.json` manifest contract
- OpenTelemetry wiring (logs, metrics, traces, LangChain instrumentation)
- Container image shape and build provenance
- Kubernetes deployment shape (namespace, RBAC, secrets, probes, OTel injection)
- Deployment scripts and the `.image-version` cadence

Out of scope:

- LangGraph workflow shape (recommended but not mandated — see §11.2)
- A2A SDK integration (recommended for multi-agent coordination — see §11.3)
- Domain-specific concerns (templates, git ops, multi-cloud auth, etc.)

---

## 2. Vocabulary

| Term | Meaning |
|---|---|
| **Agent** | A Python HTTP service that exposes domain skills, registered with the agent-gateway. |
| **agent-registry** / **agent-gateway** | The single Service (`agent-gateway.agents.svc.cluster.local`) that holds the catalogue of running agents and proxies external traffic to them. |
| **Manifest** | The JSON document served at `/.well-known/agent.json` describing the agent's identity, version, skills, and capabilities. |
| **Harness** | The non-domain plumbing every agent shares (routes, registration, OTel, k8s shape). |
| **Operator** | The OpenTelemetry Operator that injects the Python auto-instrumentation init container. |

---

## 3. Runtime requirements (MUST)

1. The agent **MUST** be a Python ≥ 3.12 application.
2. The agent **MUST** expose its HTTP server via [Starlette](https://www.starlette.io/). FastAPI is permitted only for legacy services (e.g. `agent-registry`, `support-intake-agent`); new agents **MUST** use Starlette.
3. The ASGI application **MUST** be importable as `app.agent:app`.
4. The container **MUST** run `uvicorn app.agent:app --host 0.0.0.0 --port <PORT>` as PID 1.
5. The agent **MUST** listen on TCP port `8000` *or* `8080` inside the container. Port `8000` is preferred for new agents.

---

## 4. Required HTTP routes (MUST)

Every agent **MUST** register the following routes:

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/health` | Liveness/readiness probe. Returns `200 OK` with a JSON body when the process is up. |
| `GET` | `/.well-known/agent.json` | A2A discovery manifest — see §6 and [docs/manifest.md](docs/manifest.md). |

Additional routes (the agent's domain endpoints) **MAY** be registered freely.

`/health` **MUST** return JSON of at least:

```json
{
  "status": "healthy",
  "service": "<human-readable agent name>"
}
```

A `version` field is **RECOMMENDED**.

---

## 5. Self-registration with the gateway (MUST)

Every agent **MUST** self-register with the agent-gateway on startup. The flow is described in detail in [docs/registration.md](docs/registration.md). In short:

1. The agent **MUST** read these env vars:
   - `REGISTRY_URL` — the gateway's base URL (default: `http://agent-gateway.agents.svc.cluster.local`).
   - `AGENT_SELF_URL` — the URL the gateway should call back on (default: `http://<agent-name>.agents.svc.cluster.local`).
2. On startup (inside the Starlette `lifespan` context manager), the agent **MUST** spawn a background task that POSTs `{"endpoint": AGENT_SELF_URL}` to `{REGISTRY_URL}/agents/register-url`.
3. The registration call **MUST** retry at least 3 times with a 5-second backoff and a 2-second initial delay (to let uvicorn finish binding).
4. Registration failure **MUST NOT** prevent the agent from serving traffic — it is logged at `ERROR` and retried only via pod restart.
5. If `REGISTRY_URL` is unset or empty, the agent **MUST** skip registration and log at `INFO`.

The gateway will then `GET {AGENT_SELF_URL}/.well-known/agent.json` to read the manifest. `AGENT_SELF_URL` **MUST** therefore be reachable from the gateway pod (i.e. resolve via cluster DNS) and serve a valid manifest.

---

## 6. Agent manifest (MUST)

`GET /.well-known/agent.json` **MUST** return a JSON document with at minimum:

```json
{
  "name": "Policy Transformer",
  "version": "1.0.0",
  "description": "...",
  "url": "/",
  "skills": [
    {
      "id": "policy_generation",
      "name": "Policy Generation",
      "description": "...",
      "tags": ["policy", "generate"],
      "examples": ["Generate a Kyverno policy that ..."]
    }
  ],
  "capabilities": {
    "streaming": false,
    "conversational": false,
    "direct_api": true
  }
}
```

`skills` **MUST** be a JSON array. If the agent uses the `a2a-sdk` `AgentSkill` type, it **MUST** serialise each skill defensively because `AgentSkill` is exposed as a protobuf `Message` in current `a2a-sdk` releases (not pydantic). Use a helper that handles `model_dump`, protobuf `MessageToDict`, and attribute fallback — see the canonical implementation in [`template/app/agent.py`](template/app/agent.py) (`_skill_dict`).

Full schema and examples: [docs/manifest.md](docs/manifest.md).

---

## 7. OpenTelemetry wiring (MUST)

The full design is in [docs/observability.md](docs/observability.md). Mandatory bullets:

### 7.1 Operator-driven SDK injection

The pod template **MUST** carry these annotations:

```yaml
annotations:
  instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
  instrumentation.opentelemetry.io/container-names: "<container-name>"
```

The `container-names` value **MUST** match the agent's container `name`.

### 7.2 Standard OTel env vars

The pod **MUST** set all six env vars below, in the agent container:

```yaml
env:
  - name: OTEL_SERVICE_NAME
    value: "<agent-name>"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
  - name: OTEL_LOGS_EXPORTER
    value: "otlp"
  - name: OTEL_METRICS_EXPORTER
    value: "otlp"
  - name: OTEL_METRIC_EXPORT_INTERVAL
    value: "15000"
  - name: OTEL_SEMCONV_STABILITY_OPT_IN
    value: "http"
```

Notes:

- `service.namespace` **MUST** be the Kubernetes namespace the agent runs in (typically `agents`).
- `service.version` **MUST** match the contents of `.image-version`.
- `agent.type` **MUST** be a stable, lower-hyphen identifier (e.g. `policy-transformer`, `golden-path`, `registry`). It is the primary metric/log label used in dashboards.
- `OTEL_SEMCONV_STABILITY_OPT_IN=http` is required so the SDK emits stable HTTP semantic conventions (`http.server.request.duration`, `http.response.status_code`).

### 7.3 Logging bridge

The agent **MUST** ship `app/logging_setup.py` with a `configure_otlp_log_handler(level)` function that attaches a `LoggingHandler` to the root logger using the operator-provided `LoggerProvider`. The implementation **MUST** be a no-op when:

- The OTel SDK packages are not importable, or
- `get_logger_provider()` returns `None` or a `NoOpLoggerProvider`, or
- A `LoggingHandler` is already attached.

The canonical implementation is committed in [`template/app/logging_setup.py`](template/app/logging_setup.py); copy it verbatim.

The agent **MUST** call `configure_otlp_log_handler()` once at module import time in `app/agent.py`.

### 7.4 Custom metrics

The agent **MUST** ship `app/metrics.py` that creates a meter via `metrics.get_meter("<agent-name>", "<X.Y.Z>")` and declares all custom instruments at module scope.

Naming **MUST** follow `agent.<agent_type>.<verb_or_noun>[.duration|.errors]`, with:

- `unit="1"` for counters,
- `unit="s"` for histograms timing seconds.

The agent **MUST NOT** call `MeterProvider` setters at runtime; the operator owns the global provider.

### 7.5 LLM instrumentation

If the agent calls a LangChain-style LLM, the agent **MUST** call `LangChainInstrumentor().instrument()` **before** the LLM client is constructed. This emits OpenInference spans with token counts and prompts/responses.

### 7.6 Tracing

Tracing of HTTP server, outbound HTTP, and FastAPI/Starlette routes is auto-instrumented by the operator. The agent **SHOULD NOT** install per-instrumentor wheels manually unless extending the default set.

---

## 8. Container image (MUST)

See [`template/Dockerfile`](template/Dockerfile) for the canonical shape. Required properties:

1. Base image **MUST** be `python:3.12-slim` (or `3.13-slim` once promoted by platform).
2. Image **MUST** install `requirements.txt` at build time; runtime `pip install` is forbidden.
3. Image **MUST NOT** run as `root`. Create and switch to a non-root user (UID 1001).
4. Image **MUST** expose only the chosen container port (8000 or 8080).
5. Image **SHOULD** declare a `HEALTHCHECK` calling `GET /health`.
6. Image **MUST** be tagged with both `:latest` and `:<version>` from `.image-version` and pushed to `bstagecjotdevacr.azurecr.io/<agent-name>`.

---

## 9. Kubernetes manifests (MUST)

See [docs/kubernetes.md](docs/kubernetes.md) for the full template. Required properties:

1. The agent **MUST** ship `k8s/configmap.yaml` and `k8s/deployment.yaml` (and the `Service` may be inlined in `deployment.yaml`).
2. Namespace **MUST** be `agents` unless the agent has a strong domain reason for its own namespace (e.g. `modernization-factory-v2` runs in `modernization-factory`); in that case, `service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` **MUST** match.
3. ServiceAccount **MUST** be `agents-sa` (workload-identity bound).
4. The pod **MUST** mount the shared `agents-kv-spc` SecretProviderClass at `/mnt/secrets-store` and consume secrets via `secretKeyRef` from the synced `agents-kv-sync` secret.
5. The deployment **MUST** declare both `livenessProbe` and `readinessProbe` against `httpGet: { path: /health, port: http }`. Recommended timing: liveness `initialDelaySeconds: 20, periodSeconds: 15`; readiness `initialDelaySeconds: 10, periodSeconds: 10`. Init delays **MUST** account for OTel SDK warm-up + LangGraph compile time.
6. The deployment **MUST** carry the OTel inject annotations from §7.1.
7. The Service **MUST** expose port `80` mapped to the container port (named `http`), `type: ClusterIP`.
8. Resource requests/limits **MUST** be declared explicitly (no defaults).

---

## 10. Build & deployment (MUST)

See [docs/deployment.md](docs/deployment.md) for the full flow. Required properties:

1. The agent **MUST** ship a `.image-version` file containing a single line of semver (`X.Y.Z`).
2. The agent **MUST** ship `scripts/deploy.sh` that:
   - Auto-bumps the patch version of `.image-version` when no `--tag` is given,
   - Runs `az acr build --registry bstagecjotdevacr --image <agent>:<tag> --image <agent>:latest .`,
   - Applies `k8s/configmap.yaml` then `k8s/deployment.yaml`,
   - Waits for `kubectl rollout status` (recommended ≥ 180s timeout).
3. The agent **SHOULD** ship `scripts/full-deploy.sh` (preflight + build + apply + post-deploy `/health` smoke) and `scripts/deploy-local.sh` (docker/podman + `.env.local`).
4. Bumping `.image-version` **MUST** be paired with a code change in the same commit; image-version-only commits are reserved for redeploy reruns.

---

## 11. Recommended structure (SHOULD)

### 11.1 `app/` layout

```
app/
├── __init__.py
├── agent.py            # Starlette entry point, routes, lifespan
├── logging_setup.py    # OTel log bridge (copy verbatim from template)
├── metrics.py          # Meter + custom instruments
├── skills.py           # AGENT_SKILLS list + AGENT_CONFIG dict
└── ...                 # workflows/, nodes/, tools/, states/, prompts/
```

### 11.2 LangGraph

Agents that drive a multi-step LLM workflow **SHOULD** use LangGraph and place graph definitions under `app/workflows/`, state TypedDicts under `app/states.py` (or `app/states/`), and node handlers under `app/nodes/`.

### 11.3 A2A SDK

Agents that participate in multi-agent coordination **SHOULD** integrate the `a2a-sdk` and mount its routes alongside the agent's domain routes. New agents **MUST** still serve `/.well-known/agent.json` themselves (do not rely on the SDK's manifest because it differs from the registry's expected shape).

### 11.4 Local dev

The agent **SHOULD** ship `.env.local.example` enumerating every env var read at runtime, with safe defaults committed and secrets blank.

---

## 12. Conformance checklist

Use this list to audit any agent (existing or new):

### Code

- [ ] Python ≥ 3.12 (§3.1)
- [ ] Starlette app at `app.agent:app` (§3.2–3.3)
- [ ] `GET /health` returns `{"status": "healthy", ...}` (§4)
- [ ] `GET /.well-known/agent.json` returns the manifest defined in §6
- [ ] Manifest serialisation handles pydantic + protobuf `AgentSkill` (§6)
- [ ] `_register_with_registry()` POSTs to `{REGISTRY_URL}/agents/register-url` with 3×5s retry budget (§5)
- [ ] Registration runs inside the Starlette `lifespan` and does not block startup (§5)
- [ ] `app/logging_setup.py` matches the canonical implementation and is invoked at import time (§7.3)
- [ ] `app/metrics.py` creates the meter via `metrics.get_meter(name, version)` (§7.4)
- [ ] `LangChainInstrumentor().instrument()` is called before any LLM client construction (§7.5)

### Image

- [ ] `Dockerfile` uses `python:3.12-slim`, runs as non-root UID 1001, declares `HEALTHCHECK` (§8)
- [ ] `requirements.txt` includes the harness dependency floor (see [`template/requirements.txt`](template/requirements.txt))
- [ ] `.image-version` file present and matches `OTEL_RESOURCE_ATTRIBUTES` `service.version` (§10)

### Kubernetes

- [ ] Namespace is `agents` (or matches `service.namespace`) (§9.2)
- [ ] ServiceAccount is `agents-sa` (§9.3)
- [ ] OTel inject annotations present and `container-names` matches container name (§7.1)
- [ ] All six standard OTel env vars present (§7.2)
- [ ] `agents-kv-spc` SecretProviderClass mounted at `/mnt/secrets-store` (§9.4)
- [ ] Liveness + readiness probes hit `/health` (§9.5)
- [ ] Service exposes port 80 → container port via named `http` (§9.7)
- [ ] Resource requests/limits declared (§9.8)

### Deployment

- [ ] `scripts/deploy.sh` exists and auto-bumps `.image-version` (§10.2)
- [ ] `scripts/full-deploy.sh` and `scripts/deploy-local.sh` exist (§10.3 — SHOULD)
- [ ] `.env.local.example` present (§11.4 — SHOULD)

---

## 13. Versioning of this spec

The spec follows semver:

- **MAJOR** — breaking change to a `MUST` clause (e.g. registration endpoint changes shape).
- **MINOR** — additive `MUST` or `SHOULD` clause.
- **PATCH** — clarifications, examples, typo fixes.

See [CHANGELOG.md](CHANGELOG.md). Agents **SHOULD** record the spec version they target in their `README.md`.