agent-harness-spec/SPEC.md

Platform Agent Harness Specification

Version: 1.0.0 Status: Draft → ratification pending Applies to: All Python agents deployed under cjot-backstage-az/agents/ and any new agent on the platform.

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY in this document are to be interpreted as described in RFC 2119.

---

1. Scope

This spec defines the harness — the cross-cutting plumbing every agent shares — so that domain logic is the only thing an agent author has to write. It does not prescribe how an agent's business logic, prompts, or workflow internals are structured.

In scope:

• HTTP framework, entry point, and required routes
• Self-registration with the agent-gateway / agent-registry
• The /.well-known/agent.json manifest contract
• OpenTelemetry wiring (logs, metrics, traces, LangChain instrumentation)
• Container image shape and build provenance
• Kubernetes deployment shape (namespace, RBAC, secrets, probes, OTel injection)
• Deployment scripts and the .image-version cadence

Out of scope:

• LangGraph workflow shape (recommended but not mandated — see §11.2)
• A2A SDK integration (recommended for multi-agent coordination — see §11.3)
• Domain-specific concerns (templates, git ops, multi-cloud auth, etc.)

---

2. Vocabulary

Term	Meaning
Agent	A Python HTTP service that exposes domain skills, registered with the agent-gateway.
agent-registry / agent-gateway	The single Service (agent-gateway.agents.svc.cluster.local) that holds the catalogue of running agents and proxies external traffic to them.
Manifest	The JSON document served at /.well-known/agent.json describing the agent's identity, version, skills, and capabilities.
Harness	The non-domain plumbing every agent shares (routes, registration, OTel, k8s shape).
Operator	The OpenTelemetry Operator that injects the Python auto-instrumentation init container.

---

3. Runtime requirements (MUST)

1. The agent MUST be a Python ≥ 3.12 application.
2. The agent MUST expose its HTTP server via Starlette. FastAPI is permitted only for legacy services (e.g. agent-registry, support-intake-agent); new agents MUST use Starlette.
3. The ASGI application MUST be importable as app.agent:app.
4. The container MUST run uvicorn app.agent:app --host 0.0.0.0 --port <PORT> as PID 1.
5. The agent MUST listen on TCP port 8000 or 8080 inside the container. Port 8000 is preferred for new agents.

---

4. Required HTTP routes (MUST)

Every agent MUST register the following routes:

Method	Path	Purpose
GET	/health	Liveness/readiness probe. Returns 200 OK with a JSON body when the process is up.
GET	/.well-known/agent.json	A2A discovery manifest — see §6 and docs/manifest.md.

Additional routes (the agent's domain endpoints) MAY be registered freely.

/health MUST return JSON of at least:

{
  "status": "healthy",
  "service": "<human-readable agent name>"
}

A version field is RECOMMENDED.

---

5. Self-registration with the gateway (MUST)

Every agent MUST self-register with the agent-gateway on startup. The flow is described in detail in docs/registration.md. In short:

1. The agent MUST read these env vars:
   • REGISTRY_URL — the gateway's base URL (default: http://agent-gateway.agents.svc.cluster.local).
   • AGENT_SELF_URL — the URL the gateway should call back on (default: http://<agent-name>.agents.svc.cluster.local).
2. On startup (inside the Starlette lifespan context manager), the agent MUST spawn a background task that POSTs {"endpoint": AGENT_SELF_URL} to {REGISTRY_URL}/agents/register-url.
3. The registration call MUST retry at least 3 times with a 5-second backoff and a 2-second initial delay (to let uvicorn finish binding).
4. Registration failure MUST NOT prevent the agent from serving traffic — it is logged at ERROR and retried only via pod restart.
5. If REGISTRY_URL is unset or empty, the agent MUST skip registration and log at INFO.

The gateway will then GET {AGENT_SELF_URL}/.well-known/agent.json to read the manifest. AGENT_SELF_URL MUST therefore be reachable from the gateway pod (i.e. resolve via cluster DNS) and serve a valid manifest.

---

6. Agent manifest (MUST)

GET /.well-known/agent.json MUST return a JSON document with at minimum:

{
  "name": "Policy Transformer",
  "version": "1.0.0",
  "description": "...",
  "url": "/",
  "skills": [
    {
      "id": "policy_generation",
      "name": "Policy Generation",
      "description": "...",
      "tags": ["policy", "generate"],
      "examples": ["Generate a Kyverno policy that ..."]
    }
  ],
  "capabilities": {
    "streaming": false,
    "conversational": false,
    "direct_api": true
  }
}

skills MUST be a JSON array. If the agent uses the a2a-sdk AgentSkill type, it MUST serialise each skill defensively because AgentSkill is exposed as a protobuf Message in current a2a-sdk releases (not pydantic). Use a helper that handles model_dump, protobuf MessageToDict, and attribute fallback — see the canonical implementation in template/app/agent.py (_skill_dict).

Full schema and examples: docs/manifest.md.

---

7. OpenTelemetry wiring (MUST)

The full design is in docs/observability.md. Mandatory bullets:

7.1 Operator-driven SDK injection

The pod template MUST carry these annotations:

annotations:
  instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
  instrumentation.opentelemetry.io/container-names: "<container-name>"

The container-names value MUST match the agent's container name.

7.2 Standard OTel env vars

The pod MUST set all six env vars below, in the agent container:

env:
  - name: OTEL_SERVICE_NAME
    value: "<agent-name>"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
  - name: OTEL_LOGS_EXPORTER
    value: "otlp"
  - name: OTEL_METRICS_EXPORTER
    value: "otlp"
  - name: OTEL_METRIC_EXPORT_INTERVAL
    value: "15000"
  - name: OTEL_SEMCONV_STABILITY_OPT_IN
    value: "http"

Notes:

• service.namespace MUST be the Kubernetes namespace the agent runs in (typically agents).
• service.version MUST match the contents of .image-version.
• agent.type MUST be a stable, lower-hyphen identifier (e.g. policy-transformer, golden-path, registry). It is the primary metric/log label used in dashboards.
• OTEL_SEMCONV_STABILITY_OPT_IN=http is required so the SDK emits stable HTTP semantic conventions (http.server.request.duration, http.response.status_code).

7.3 Logging bridge

The agent MUST ship app/logging_setup.py with a configure_otlp_log_handler(level) function that attaches a LoggingHandler to the root logger using the operator-provided LoggerProvider. The implementation MUST be a no-op when:

• The OTel SDK packages are not importable, or
• get_logger_provider() returns None or a NoOpLoggerProvider, or
• A LoggingHandler is already attached.

The canonical implementation is committed in template/app/logging_setup.py; copy it verbatim.

The agent MUST call configure_otlp_log_handler() once at module import time in app/agent.py.

7.4 Custom metrics

The agent MUST ship app/metrics.py that creates a meter via metrics.get_meter("<agent-name>", "<X.Y.Z>") and declares all custom instruments at module scope.

Naming MUST follow agent.<agent_type>.<verb_or_noun>[.duration|.errors], with:

• unit="1" for counters,
• unit="s" for histograms timing seconds.

The agent MUST NOT call MeterProvider setters at runtime; the operator owns the global provider.

7.5 LLM instrumentation

If the agent calls a LangChain-style LLM, the agent MUST call LangChainInstrumentor().instrument() before the LLM client is constructed. This emits OpenInference spans with token counts and prompts/responses.

7.6 Tracing

Tracing of HTTP server, outbound HTTP, and FastAPI/Starlette routes is auto-instrumented by the operator. The agent SHOULD NOT install per-instrumentor wheels manually unless extending the default set.

---

8. Container image (MUST)

See template/Dockerfile for the canonical shape. Required properties:

1. Base image MUST be python:3.12-slim (or 3.13-slim once promoted by platform).
2. Image MUST install requirements.txt at build time; runtime pip install is forbidden.
3. Image MUST NOT run as root. Create and switch to a non-root user (UID 1001).
4. Image MUST expose only the chosen container port (8000 or 8080).
5. Image SHOULD declare a HEALTHCHECK calling GET /health.
6. Image MUST be tagged with both :latest and :<version> from .image-version and pushed to bstagecjotdevacr.azurecr.io/<agent-name>.

---

9. Kubernetes manifests (MUST)

See docs/kubernetes.md for the full template. Required properties:

1. The agent MUST ship k8s/configmap.yaml and k8s/deployment.yaml (and the Service may be inlined in deployment.yaml).
2. Namespace MUST be agents unless the agent has a strong domain reason for its own namespace (e.g. modernization-factory-v2 runs in modernization-factory); in that case, service.namespace in OTEL_RESOURCE_ATTRIBUTES MUST match.
3. ServiceAccount MUST be agents-sa (workload-identity bound).
4. The pod MUST mount the shared agents-kv-spc SecretProviderClass at /mnt/secrets-store and consume secrets via secretKeyRef from the synced agents-kv-sync secret.
5. The deployment MUST declare both livenessProbe and readinessProbe against httpGet: { path: /health, port: http }. Recommended timing: liveness initialDelaySeconds: 20, periodSeconds: 15; readiness initialDelaySeconds: 10, periodSeconds: 10. Init delays MUST account for OTel SDK warm-up + LangGraph compile time.
6. The deployment MUST carry the OTel inject annotations from §7.1.
7. The Service MUST expose port 80 mapped to the container port (named http), type: ClusterIP.
8. Resource requests/limits MUST be declared explicitly (no defaults).

---

10. Build & deployment (MUST)

See docs/deployment.md for the full flow. Required properties:

1. The agent MUST ship a .image-version file containing a single line of semver (X.Y.Z).
2. The agent MUST ship scripts/deploy.sh that:
   • Auto-bumps the patch version of .image-version when no --tag is given,
   • Runs az acr build --registry bstagecjotdevacr --image <agent>:<tag> --image <agent>:latest .,
   • Applies k8s/configmap.yaml then k8s/deployment.yaml,
   • Waits for kubectl rollout status (recommended ≥ 180s timeout).
3. The agent SHOULD ship scripts/full-deploy.sh (preflight + build + apply + post-deploy /health smoke) and scripts/deploy-local.sh (docker/podman + .env.local).
4. Bumping .image-version MUST be paired with a code change in the same commit; image-version-only commits are reserved for redeploy reruns.

---

11. Recommended structure (SHOULD)

11.1 app/ layout

app/
├── __init__.py
├── agent.py            # Starlette entry point, routes, lifespan
├── logging_setup.py    # OTel log bridge (copy verbatim from template)
├── metrics.py          # Meter + custom instruments
├── skills.py           # AGENT_SKILLS list + AGENT_CONFIG dict
└── ...                 # workflows/, nodes/, tools/, states/, prompts/

11.2 LangGraph

Agents that drive a multi-step LLM workflow SHOULD use LangGraph and place graph definitions under app/workflows/, state TypedDicts under app/states.py (or app/states/), and node handlers under app/nodes/.

11.3 A2A SDK

Agents that participate in multi-agent coordination SHOULD integrate the a2a-sdk and mount its routes alongside the agent's domain routes. New agents MUST still serve /.well-known/agent.json themselves (do not rely on the SDK's manifest because it differs from the registry's expected shape).

11.4 Local dev

The agent SHOULD ship .env.local.example enumerating every env var read at runtime, with safe defaults committed and secrets blank.

---

12. Conformance checklist

Use this list to audit any agent (existing or new):

Code

• Python ≥ 3.12 (§3.1)
• Starlette app at app.agent:app (§3.2–3.3)
• GET /health returns {"status": "healthy", ...} (§4)
• GET /.well-known/agent.json returns the manifest defined in §6
• Manifest serialisation handles pydantic + protobuf AgentSkill (§6)
• _register_with_registry() POSTs to {REGISTRY_URL}/agents/register-url with 3×5s retry budget (§5)
• Registration runs inside the Starlette lifespan and does not block startup (§5)
• app/logging_setup.py matches the canonical implementation and is invoked at import time (§7.3)
• app/metrics.py creates the meter via metrics.get_meter(name, version) (§7.4)
• LangChainInstrumentor().instrument() is called before any LLM client construction (§7.5)

Image

• Dockerfile uses python:3.12-slim, runs as non-root UID 1001, declares HEALTHCHECK (§8)
• requirements.txt includes the harness dependency floor (see template/requirements.txt)
• .image-version file present and matches OTEL_RESOURCE_ATTRIBUTES service.version (§10)

Kubernetes

• Namespace is agents (or matches service.namespace) (§9.2)
• ServiceAccount is agents-sa (§9.3)
• OTel inject annotations present and container-names matches container name (§7.1)
• All six standard OTel env vars present (§7.2)
• agents-kv-spc SecretProviderClass mounted at /mnt/secrets-store (§9.4)
• Liveness + readiness probes hit /health (§9.5)
• Service exposes port 80 → container port via named http (§9.7)
• Resource requests/limits declared (§9.8)

Deployment

• scripts/deploy.sh exists and auto-bumps .image-version (§10.2)
• scripts/full-deploy.sh and scripts/deploy-local.sh exist (§10.3 — SHOULD)
• .env.local.example present (§11.4 — SHOULD)

---

13. Versioning of this spec

The spec follows semver:

• MAJOR — breaking change to a MUST clause (e.g. registration endpoint changes shape).
• MINOR — additive MUST or SHOULD clause.
• PATCH — clarifications, examples, typo fixes.

See CHANGELOG.md. Agents SHOULD record the spec version they target in their README.md.