feat: initial harness spec v1.0.0 + reference template

Adds the normative contract every platform agent must conform to, plus a working reference template under template/ that ticks every box out of the box. Spec is derived from the production agents shipped in cjot-backstage-az/agents/ on branch sandbox/jonathan: - agent-registry, agent-factory, decomposer, discovery-agent, golden-path, modernization-factory, modernization-factory-v2, policy-transformer, scaffold-agent, support-intake-agent, the-watcher Contents: - SPEC.md normative contract (13 sections + conformance checklist) - CHANGELOG.md spec versioning (1.0.0) - docs/ registration.md self-registration with the agent-gateway observability.md OTel logs/metrics/traces wiring manifest.md /.well-known/agent.json schema + AgentSkill serialisation (pydantic vs protobuf) kubernetes.md k8s deployment shape, mandatory cross-refs, per-namespace agents, resource sizing deployment.md .image-version, ACR build, deploy scripts, rollback deviations.md tracked debts against the spec - template/ .env.local.example, .gitignore, .image-version (0.1.0), Dockerfile (python:3.12-slim, non-root UID 1001, HEALTHCHECK), requirements.txt (harness floor + optional LLM stack), app/agent.py (Starlette entry, lifespan + self-registration, defensive _skill_dict for pydantic vs protobuf), app/logging_setup.py (canonical OTel log bridge — copy verbatim), app/metrics.py (meter + example counter/histogram), app/skills.py (AGENT_CONFIG + AgentSkill list), k8s/configmap.yaml + deployment.yaml (Deployment + Service, OTel annotations + 6-var env block, agents-sa + agents-kv-spc bindings), scripts/deploy.sh (auto-bump + az acr build + apply + rollout), scripts/full-deploy.sh (preflight + deploy + post-deploy smoke), scripts/deploy-local.sh (docker/podman + .env.local)
2026-06-09 15:54:18 +01:00
parent c87dd3fa55
commit e6f10ba8c9
25 changed files with 1933 additions and 1 deletions
--- a/SPEC.md
+++ b/SPEC.md
@@ -0,0 +1,325 @@
+# Platform Agent Harness Specification
+
+**Version:** 1.0.0
+**Status:** Draft → ratification pending
+**Applies to:** All Python agents deployed under `cjot-backstage-az/agents/` and any new agent on the platform.
+
+The key words **MUST**, **MUST NOT**, **SHOULD**, **SHOULD NOT**, and **MAY** in this document are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
+
+---
+
+## 1. Scope
+
+This spec defines the *harness* — the cross-cutting plumbing every agent shares — so that domain logic is the only thing an agent author has to write. It does **not** prescribe how an agent's business logic, prompts, or workflow internals are structured.
+
+In scope:
+
+- HTTP framework, entry point, and required routes
+- Self-registration with the agent-gateway / agent-registry
+- The `/.well-known/agent.json` manifest contract
+- OpenTelemetry wiring (logs, metrics, traces, LangChain instrumentation)
+- Container image shape and build provenance
+- Kubernetes deployment shape (namespace, RBAC, secrets, probes, OTel injection)
+- Deployment scripts and the `.image-version` cadence
+
+Out of scope:
+
+- LangGraph workflow shape (recommended but not mandated — see §11.2)
+- A2A SDK integration (recommended for multi-agent coordination — see §11.3)
+- Domain-specific concerns (templates, git ops, multi-cloud auth, etc.)
+
+---
+
+## 2. Vocabulary
+
+| Term | Meaning |
+|---|---|
+| **Agent** | A Python HTTP service that exposes domain skills, registered with the agent-gateway. |
+| **agent-registry** / **agent-gateway** | The single Service (`agent-gateway.agents.svc.cluster.local`) that holds the catalogue of running agents and proxies external traffic to them. |
+| **Manifest** | The JSON document served at `/.well-known/agent.json` describing the agent's identity, version, skills, and capabilities. |
+| **Harness** | The non-domain plumbing every agent shares (routes, registration, OTel, k8s shape). |
+| **Operator** | The OpenTelemetry Operator that injects the Python auto-instrumentation init container. |
+
+---
+
+## 3. Runtime requirements (MUST)
+
+1. The agent **MUST** be a Python ≥ 3.12 application.
+2. The agent **MUST** expose its HTTP server via [Starlette](https://www.starlette.io/). FastAPI is permitted only for legacy services (e.g. `agent-registry`, `support-intake-agent`); new agents **MUST** use Starlette.
+3. The ASGI application **MUST** be importable as `app.agent:app`.
+4. The container **MUST** run `uvicorn app.agent:app --host 0.0.0.0 --port <PORT>` as PID 1.
+5. The agent **MUST** listen on TCP port `8000` *or* `8080` inside the container. Port `8000` is preferred for new agents.
+
+---
+
+## 4. Required HTTP routes (MUST)
+
+Every agent **MUST** register the following routes:
+
+| Method | Path | Purpose |
+|---|---|---|
+| `GET` | `/health` | Liveness/readiness probe. Returns `200 OK` with a JSON body when the process is up. |
+| `GET` | `/.well-known/agent.json` | A2A discovery manifest — see §6 and [docs/manifest.md](docs/manifest.md). |
+
+Additional routes (the agent's domain endpoints) **MAY** be registered freely.
+
+`/health` **MUST** return JSON of at least:
+
+```json
+{
+  "status": "healthy",
+  "service": "<human-readable agent name>"
+}
+```
+
+A `version` field is **RECOMMENDED**.
+
+---
+
+## 5. Self-registration with the gateway (MUST)
+
+Every agent **MUST** self-register with the agent-gateway on startup. The flow is described in detail in [docs/registration.md](docs/registration.md). In short:
+
+1. The agent **MUST** read these env vars:
+   - `REGISTRY_URL` — the gateway's base URL (default: `http://agent-gateway.agents.svc.cluster.local`).
+   - `AGENT_SELF_URL` — the URL the gateway should call back on (default: `http://<agent-name>.agents.svc.cluster.local`).
+2. On startup (inside the Starlette `lifespan` context manager), the agent **MUST** spawn a background task that POSTs `{"endpoint": AGENT_SELF_URL}` to `{REGISTRY_URL}/agents/register-url`.
+3. The registration call **MUST** retry at least 3 times with a 5-second backoff and a 2-second initial delay (to let uvicorn finish binding).
+4. Registration failure **MUST NOT** prevent the agent from serving traffic — it is logged at `ERROR` and retried only via pod restart.
+5. If `REGISTRY_URL` is unset or empty, the agent **MUST** skip registration and log at `INFO`.
+
+The gateway will then `GET {AGENT_SELF_URL}/.well-known/agent.json` to read the manifest. `AGENT_SELF_URL` **MUST** therefore be reachable from the gateway pod (i.e. resolve via cluster DNS) and serve a valid manifest.
+
+---
+
+## 6. Agent manifest (MUST)
+
+`GET /.well-known/agent.json` **MUST** return a JSON document with at minimum:
+
+```json
+{
+  "name": "Policy Transformer",
+  "version": "1.0.0",
+  "description": "...",
+  "url": "/",
+  "skills": [
+    {
+      "id": "policy_generation",
+      "name": "Policy Generation",
+      "description": "...",
+      "tags": ["policy", "generate"],
+      "examples": ["Generate a Kyverno policy that ..."]
+    }
+  ],
+  "capabilities": {
+    "streaming": false,
+    "conversational": false,
+    "direct_api": true
+  }
+}
+```
+
+`skills` **MUST** be a JSON array. If the agent uses the `a2a-sdk` `AgentSkill` type, it **MUST** serialise each skill defensively because `AgentSkill` is exposed as a protobuf `Message` in current `a2a-sdk` releases (not pydantic). Use a helper that handles `model_dump`, protobuf `MessageToDict`, and attribute fallback — see the canonical implementation in [`template/app/agent.py`](template/app/agent.py) (`_skill_dict`).
+
+Full schema and examples: [docs/manifest.md](docs/manifest.md).
+
+---
+
+## 7. OpenTelemetry wiring (MUST)
+
+The full design is in [docs/observability.md](docs/observability.md). Mandatory bullets:
+
+### 7.1 Operator-driven SDK injection
+
+The pod template **MUST** carry these annotations:
+
+```yaml
+annotations:
+  instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
+  instrumentation.opentelemetry.io/container-names: "<container-name>"
+```
+
+The `container-names` value **MUST** match the agent's container `name`.
+
+### 7.2 Standard OTel env vars
+
+The pod **MUST** set all six env vars below, in the agent container:
+
+```yaml
+env:
+  - name: OTEL_SERVICE_NAME
+    value: "<agent-name>"
+  - name: OTEL_RESOURCE_ATTRIBUTES
+    value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
+  - name: OTEL_LOGS_EXPORTER
+    value: "otlp"
+  - name: OTEL_METRICS_EXPORTER
+    value: "otlp"
+  - name: OTEL_METRIC_EXPORT_INTERVAL
+    value: "15000"
+  - name: OTEL_SEMCONV_STABILITY_OPT_IN
+    value: "http"
+```
+
+Notes:
+
+- `service.namespace` **MUST** be the Kubernetes namespace the agent runs in (typically `agents`).
+- `service.version` **MUST** match the contents of `.image-version`.
+- `agent.type` **MUST** be a stable, lower-hyphen identifier (e.g. `policy-transformer`, `golden-path`, `registry`). It is the primary metric/log label used in dashboards.
+- `OTEL_SEMCONV_STABILITY_OPT_IN=http` is required so the SDK emits stable HTTP semantic conventions (`http.server.request.duration`, `http.response.status_code`).
+
+### 7.3 Logging bridge
+
+The agent **MUST** ship `app/logging_setup.py` with a `configure_otlp_log_handler(level)` function that attaches a `LoggingHandler` to the root logger using the operator-provided `LoggerProvider`. The implementation **MUST** be a no-op when:
+
+- The OTel SDK packages are not importable, or
+- `get_logger_provider()` returns `None` or a `NoOpLoggerProvider`, or
+- A `LoggingHandler` is already attached.
+
+The canonical implementation is committed in [`template/app/logging_setup.py`](template/app/logging_setup.py); copy it verbatim.
+
+The agent **MUST** call `configure_otlp_log_handler()` once at module import time in `app/agent.py`.
+
+### 7.4 Custom metrics
+
+The agent **MUST** ship `app/metrics.py` that creates a meter via `metrics.get_meter("<agent-name>", "<X.Y.Z>")` and declares all custom instruments at module scope.
+
+Naming **MUST** follow `agent.<agent_type>.<verb_or_noun>[.duration|.errors]`, with:
+
+- `unit="1"` for counters,
+- `unit="s"` for histograms timing seconds.
+
+The agent **MUST NOT** call `MeterProvider` setters at runtime; the operator owns the global provider.
+
+### 7.5 LLM instrumentation
+
+If the agent calls a LangChain-style LLM, the agent **MUST** call `LangChainInstrumentor().instrument()` **before** the LLM client is constructed. This emits OpenInference spans with token counts and prompts/responses.
+
+### 7.6 Tracing
+
+Tracing of HTTP server, outbound HTTP, and FastAPI/Starlette routes is auto-instrumented by the operator. The agent **SHOULD NOT** install per-instrumentor wheels manually unless extending the default set.
+
+---
+
+## 8. Container image (MUST)
+
+See [`template/Dockerfile`](template/Dockerfile) for the canonical shape. Required properties:
+
+1. Base image **MUST** be `python:3.12-slim` (or `3.13-slim` once promoted by platform).
+2. Image **MUST** install `requirements.txt` at build time; runtime `pip install` is forbidden.
+3. Image **MUST NOT** run as `root`. Create and switch to a non-root user (UID 1001).
+4. Image **MUST** expose only the chosen container port (8000 or 8080).
+5. Image **SHOULD** declare a `HEALTHCHECK` calling `GET /health`.
+6. Image **MUST** be tagged with both `:latest` and `:<version>` from `.image-version` and pushed to `bstagecjotdevacr.azurecr.io/<agent-name>`.
+
+---
+
+## 9. Kubernetes manifests (MUST)
+
+See [docs/kubernetes.md](docs/kubernetes.md) for the full template. Required properties:
+
+1. The agent **MUST** ship `k8s/configmap.yaml` and `k8s/deployment.yaml` (and the `Service` may be inlined in `deployment.yaml`).
+2. Namespace **MUST** be `agents` unless the agent has a strong domain reason for its own namespace (e.g. `modernization-factory-v2` runs in `modernization-factory`); in that case, `service.namespace` in `OTEL_RESOURCE_ATTRIBUTES` **MUST** match.
+3. ServiceAccount **MUST** be `agents-sa` (workload-identity bound).
+4. The pod **MUST** mount the shared `agents-kv-spc` SecretProviderClass at `/mnt/secrets-store` and consume secrets via `secretKeyRef` from the synced `agents-kv-sync` secret.
+5. The deployment **MUST** declare both `livenessProbe` and `readinessProbe` against `httpGet: { path: /health, port: http }`. Recommended timing: liveness `initialDelaySeconds: 20, periodSeconds: 15`; readiness `initialDelaySeconds: 10, periodSeconds: 10`. Init delays **MUST** account for OTel SDK warm-up + LangGraph compile time.
+6. The deployment **MUST** carry the OTel inject annotations from §7.1.
+7. The Service **MUST** expose port `80` mapped to the container port (named `http`), `type: ClusterIP`.
+8. Resource requests/limits **MUST** be declared explicitly (no defaults).
+
+---
+
+## 10. Build & deployment (MUST)
+
+See [docs/deployment.md](docs/deployment.md) for the full flow. Required properties:
+
+1. The agent **MUST** ship a `.image-version` file containing a single line of semver (`X.Y.Z`).
+2. The agent **MUST** ship `scripts/deploy.sh` that:
+   - Auto-bumps the patch version of `.image-version` when no `--tag` is given,
+   - Runs `az acr build --registry bstagecjotdevacr --image <agent>:<tag> --image <agent>:latest .`,
+   - Applies `k8s/configmap.yaml` then `k8s/deployment.yaml`,
+   - Waits for `kubectl rollout status` (recommended ≥ 180s timeout).
+3. The agent **SHOULD** ship `scripts/full-deploy.sh` (preflight + build + apply + post-deploy `/health` smoke) and `scripts/deploy-local.sh` (docker/podman + `.env.local`).
+4. Bumping `.image-version` **MUST** be paired with a code change in the same commit; image-version-only commits are reserved for redeploy reruns.
+
+---
+
+## 11. Recommended structure (SHOULD)
+
+### 11.1 `app/` layout
+
+```
+app/
+├── __init__.py
+├── agent.py            # Starlette entry point, routes, lifespan
+├── logging_setup.py    # OTel log bridge (copy verbatim from template)
+├── metrics.py          # Meter + custom instruments
+├── skills.py           # AGENT_SKILLS list + AGENT_CONFIG dict
+└── ...                 # workflows/, nodes/, tools/, states/, prompts/
+```
+
+### 11.2 LangGraph
+
+Agents that drive a multi-step LLM workflow **SHOULD** use LangGraph and place graph definitions under `app/workflows/`, state TypedDicts under `app/states.py` (or `app/states/`), and node handlers under `app/nodes/`.
+
+### 11.3 A2A SDK
+
+Agents that participate in multi-agent coordination **SHOULD** integrate the `a2a-sdk` and mount its routes alongside the agent's domain routes. New agents **MUST** still serve `/.well-known/agent.json` themselves (do not rely on the SDK's manifest because it differs from the registry's expected shape).
+
+### 11.4 Local dev
+
+The agent **SHOULD** ship `.env.local.example` enumerating every env var read at runtime, with safe defaults committed and secrets blank.
+
+---
+
+## 12. Conformance checklist
+
+Use this list to audit any agent (existing or new):
+
+### Code
+
+- [ ] Python ≥ 3.12 (§3.1)
+- [ ] Starlette app at `app.agent:app` (§3.2–3.3)
+- [ ] `GET /health` returns `{"status": "healthy", ...}` (§4)
+- [ ] `GET /.well-known/agent.json` returns the manifest defined in §6
+- [ ] Manifest serialisation handles pydantic + protobuf `AgentSkill` (§6)
+- [ ] `_register_with_registry()` POSTs to `{REGISTRY_URL}/agents/register-url` with 3×5s retry budget (§5)
+- [ ] Registration runs inside the Starlette `lifespan` and does not block startup (§5)
+- [ ] `app/logging_setup.py` matches the canonical implementation and is invoked at import time (§7.3)
+- [ ] `app/metrics.py` creates the meter via `metrics.get_meter(name, version)` (§7.4)
+- [ ] `LangChainInstrumentor().instrument()` is called before any LLM client construction (§7.5)
+
+### Image
+
+- [ ] `Dockerfile` uses `python:3.12-slim`, runs as non-root UID 1001, declares `HEALTHCHECK` (§8)
+- [ ] `requirements.txt` includes the harness dependency floor (see [`template/requirements.txt`](template/requirements.txt))
+- [ ] `.image-version` file present and matches `OTEL_RESOURCE_ATTRIBUTES` `service.version` (§10)
+
+### Kubernetes
+
+- [ ] Namespace is `agents` (or matches `service.namespace`) (§9.2)
+- [ ] ServiceAccount is `agents-sa` (§9.3)
+- [ ] OTel inject annotations present and `container-names` matches container name (§7.1)
+- [ ] All six standard OTel env vars present (§7.2)
+- [ ] `agents-kv-spc` SecretProviderClass mounted at `/mnt/secrets-store` (§9.4)
+- [ ] Liveness + readiness probes hit `/health` (§9.5)
+- [ ] Service exposes port 80 → container port via named `http` (§9.7)
+- [ ] Resource requests/limits declared (§9.8)
+
+### Deployment
+
+- [ ] `scripts/deploy.sh` exists and auto-bumps `.image-version` (§10.2)
+- [ ] `scripts/full-deploy.sh` and `scripts/deploy-local.sh` exist (§10.3 — SHOULD)
+- [ ] `.env.local.example` present (§11.4 — SHOULD)
+
+---
+
+## 13. Versioning of this spec
+
+The spec follows semver:
+
+- **MAJOR** — breaking change to a `MUST` clause (e.g. registration endpoint changes shape).
+- **MINOR** — additive `MUST` or `SHOULD` clause.
+- **PATCH** — clarifications, examples, typo fixes.
+
+See [CHANGELOG.md](CHANGELOG.md). Agents **SHOULD** record the spec version they target in their `README.md`.