feat: initial harness spec v1.0.0 + reference template
Adds the normative contract every platform agent must conform to, plus a
working reference template under template/ that ticks every box out of
the box.
Spec is derived from the production agents shipped in
cjot-backstage-az/agents/ on branch sandbox/jonathan:
- agent-registry, agent-factory, decomposer, discovery-agent,
golden-path, modernization-factory, modernization-factory-v2,
policy-transformer, scaffold-agent, support-intake-agent, the-watcher
Contents:
- SPEC.md normative contract (13 sections + conformance checklist)
- CHANGELOG.md spec versioning (1.0.0)
- docs/
registration.md self-registration with the agent-gateway
observability.md OTel logs/metrics/traces wiring
manifest.md /.well-known/agent.json schema + AgentSkill
serialisation (pydantic vs protobuf)
kubernetes.md k8s deployment shape, mandatory cross-refs,
per-namespace agents, resource sizing
deployment.md .image-version, ACR build, deploy scripts, rollback
deviations.md tracked debts against the spec
- template/
.env.local.example, .gitignore, .image-version (0.1.0),
Dockerfile (python:3.12-slim, non-root UID 1001, HEALTHCHECK),
requirements.txt (harness floor + optional LLM stack),
app/agent.py (Starlette entry, lifespan + self-registration,
defensive _skill_dict for pydantic vs protobuf),
app/logging_setup.py (canonical OTel log bridge — copy verbatim),
app/metrics.py (meter + example counter/histogram),
app/skills.py (AGENT_CONFIG + AgentSkill list),
k8s/configmap.yaml + deployment.yaml (Deployment + Service,
OTel annotations + 6-var env block,
agents-sa + agents-kv-spc bindings),
scripts/deploy.sh (auto-bump + az acr build + apply + rollout),
scripts/full-deploy.sh (preflight + deploy + post-deploy smoke),
scripts/deploy-local.sh (docker/podman + .env.local)
This commit is contained in:
209
docs/observability.md
Normal file
209
docs/observability.md
Normal file
@@ -0,0 +1,209 @@
|
||||
# Observability — OpenTelemetry wiring
|
||||
|
||||
> Normative requirements: [SPEC.md §7](../SPEC.md#7-opentelemetry-wiring-must)
|
||||
|
||||
The platform runs the OpenTelemetry Operator. Every agent gets the SDK injected as an init container; the agent code itself does **not** install or configure SDK exporters. This document explains what is provided for free, what the agent must wire by hand, and how the signals flow to the cluster collectors.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Agent pod] -->|OTLP gRPC| T1[Tier-1 collector]
|
||||
T1 -->|metrics| P[Prometheus RemoteWrite]
|
||||
T1 -->|logs| L[Loki]
|
||||
T1 -->|traces| TP[Tempo]
|
||||
T1 -->|spanmetrics| P
|
||||
```
|
||||
|
||||
- **Tier-1 collector** runs as a DaemonSet, decorates spans/logs with `k8sattributes` (pod, namespace, node), and forwards to tier-2.
|
||||
- **Tier-2 collector** generates spanmetrics from traces, exports OTLP traces to Tempo, OTLP logs to Loki, and Prometheus remote-writes metrics.
|
||||
- The agent only needs OTLP gRPC reachability to `tier-1` — everything else is the operator's job.
|
||||
|
||||
## What the operator gives you for free
|
||||
|
||||
The `Instrumentation` CR `monitoring/otel-instrumentation` injects:
|
||||
|
||||
- **Tracer/Meter/Logger providers** with OTLP exporters configured.
|
||||
- **Auto-instrumented HTTP server** (`opentelemetry-instrumentation-asgi` or `-fastapi`).
|
||||
- **Auto-instrumented outbound HTTP** (`urllib3`, `httpx`, `requests`).
|
||||
- **Logger provider** with `BatchLogRecordProcessor` — but **does not** attach a `LoggingHandler` to Python's stdlib root logger. That is the agent's job.
|
||||
|
||||
## What the agent must wire by hand
|
||||
|
||||
### 1. The pod template annotations
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
annotations:
|
||||
instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
|
||||
instrumentation.opentelemetry.io/container-names: "<container-name>"
|
||||
```
|
||||
|
||||
`container-names` **MUST** match the container `name`. If you have multiple containers (sidecar, etc.), only list the agent container.
|
||||
|
||||
### 2. The standard env block
|
||||
|
||||
These six env vars are mandatory:
|
||||
|
||||
```yaml
|
||||
- name: OTEL_SERVICE_NAME
|
||||
value: "<agent-name>"
|
||||
- name: OTEL_RESOURCE_ATTRIBUTES
|
||||
value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
|
||||
- name: OTEL_LOGS_EXPORTER
|
||||
value: "otlp"
|
||||
- name: OTEL_METRICS_EXPORTER
|
||||
value: "otlp"
|
||||
- name: OTEL_METRIC_EXPORT_INTERVAL
|
||||
value: "15000"
|
||||
- name: OTEL_SEMCONV_STABILITY_OPT_IN
|
||||
value: "http"
|
||||
```
|
||||
|
||||
#### `OTEL_SERVICE_NAME`
|
||||
|
||||
The unique service identity. Used as the `service.name` resource attribute on every span/metric/log. **MUST** match the agent's lower-hyphen identifier.
|
||||
|
||||
#### `OTEL_RESOURCE_ATTRIBUTES`
|
||||
|
||||
Comma-separated `key=value` pairs merged into the resource. Required keys:
|
||||
|
||||
- `service.namespace` — the Kubernetes namespace.
|
||||
- `service.version` — semver string. **MUST** match the contents of the agent's `.image-version` file.
|
||||
- `deployment.environment` — `prod` / `dev` / `local`.
|
||||
- `agent.type` — stable lower-hyphen identifier (`policy-transformer`, `golden-path`, `registry`, etc.). This is the primary metric/log filter in dashboards.
|
||||
|
||||
#### `OTEL_METRIC_EXPORT_INTERVAL`
|
||||
|
||||
`15000` ms (15 s) is platform-wide. Faster than this generates noise; slower hides incidents.
|
||||
|
||||
#### `OTEL_SEMCONV_STABILITY_OPT_IN`
|
||||
|
||||
`http` opts the SDK into the **stable** HTTP semantic conventions:
|
||||
|
||||
- `http.server.request.duration` (histogram)
|
||||
- `http.response.status_code` (attribute)
|
||||
|
||||
Without this flag the SDK emits the legacy `http.server.duration` + `http.status_code`, which the platform dashboards no longer query.
|
||||
|
||||
### 3. The logging bridge
|
||||
|
||||
The OTel Logger provider exists but isn't connected to stdlib `logging`. The agent **MUST** ship `app/logging_setup.py` and call `configure_otlp_log_handler()` once at module import time in `app/agent.py`.
|
||||
|
||||
The canonical implementation is in [`../template/app/logging_setup.py`](../template/app/logging_setup.py). Copy it verbatim — it is intentionally short and idempotent:
|
||||
|
||||
```python
|
||||
def configure_otlp_log_handler(level: int = logging.INFO) -> None:
|
||||
try:
|
||||
from opentelemetry._logs import get_logger_provider
|
||||
from opentelemetry.sdk._logs import LoggingHandler
|
||||
except ImportError:
|
||||
return
|
||||
provider = get_logger_provider()
|
||||
if provider is None or type(provider).__name__ == "NoOpLoggerProvider":
|
||||
return
|
||||
root = logging.getLogger()
|
||||
if any(isinstance(h, LoggingHandler) for h in root.handlers):
|
||||
return
|
||||
root.addHandler(LoggingHandler(level=level, logger_provider=provider))
|
||||
if root.level == logging.NOTSET or root.level > level:
|
||||
root.setLevel(level)
|
||||
```
|
||||
|
||||
The function is a no-op outside the cluster (the SDK packages aren't installed), so the same call site works locally.
|
||||
|
||||
### 4. Custom metrics
|
||||
|
||||
Custom metrics live in `app/metrics.py`. The shape:
|
||||
|
||||
```python
|
||||
from opentelemetry import metrics
|
||||
|
||||
_meter = metrics.get_meter("<agent-name>", "<X.Y.Z>")
|
||||
|
||||
work_done = _meter.create_counter(
|
||||
"agent.<type>.runs",
|
||||
unit="1",
|
||||
description="Work units processed by outcome.",
|
||||
)
|
||||
|
||||
work_duration = _meter.create_histogram(
|
||||
"agent.<type>.duration",
|
||||
unit="s",
|
||||
description="End-to-end work latency.",
|
||||
)
|
||||
```
|
||||
|
||||
Naming rules (mandatory):
|
||||
|
||||
- Prefix all metric names with `agent.<type>.` where `<type>` matches `agent.type` in `OTEL_RESOURCE_ATTRIBUTES`.
|
||||
- Counters: noun (`runs`, `tokens`, `errors`).
|
||||
- Histograms: noun + `.duration` (seconds).
|
||||
- Use `unit="1"` for counts, `unit="s"` for seconds, `unit="By"` for bytes.
|
||||
|
||||
Recording from a route handler:
|
||||
|
||||
```python
|
||||
from .metrics import work_done, work_duration
|
||||
|
||||
async def handler(request):
|
||||
started = time.perf_counter()
|
||||
try:
|
||||
outcome = await do_the_thing(...)
|
||||
work_done.add(1, {"outcome": "success"})
|
||||
return JSONResponse(outcome)
|
||||
except ValueError as exc:
|
||||
work_done.add(1, {"outcome": "bad_request"})
|
||||
raise
|
||||
except Exception:
|
||||
work_done.add(1, {"outcome": "error"})
|
||||
raise
|
||||
finally:
|
||||
work_duration.record(time.perf_counter() - started)
|
||||
```
|
||||
|
||||
Prefer **inline metric calls at every return path** over outer `try/finally` wrappers — it avoids indentation churn and reads cleaner. Use `try/finally` only for histograms that always need to record regardless of outcome.
|
||||
|
||||
Do **not** create `MeterProvider` instances or call `set_meter_provider()` — the operator owns the global provider.
|
||||
|
||||
### 5. LangChain / LLM instrumentation
|
||||
|
||||
Agents that call an LLM via LangChain **MUST** wire OpenInference:
|
||||
|
||||
```python
|
||||
from openinference.instrumentation.langchain import LangChainInstrumentor
|
||||
|
||||
LangChainInstrumentor().instrument()
|
||||
```
|
||||
|
||||
Call this **once**, before any `AzureChatOpenAI(...)` (or other chat-model) construction. It emits OpenInference spans with token counts, prompt/response payloads, and tool calls — visible in Tempo and surfacable as LLM dashboards.
|
||||
|
||||
Multiple `instrument()` calls are safe but log a warning. If you compose modules across `__main__` and `app.agent`, expect to see the warning once per import path; this is harmless.
|
||||
|
||||
## Verifying signals end-to-end
|
||||
|
||||
After deploying, check each signal flows:
|
||||
|
||||
```bash
|
||||
# Logs — should appear in Loki within ~30s
|
||||
kubectl logs deployment/<agent> -c <agent-name> | head
|
||||
|
||||
# Metrics — query Prometheus for the meter name
|
||||
curl -s "$PROM/api/v1/query?query=agent_<type>_runs_total" | jq
|
||||
|
||||
# Traces — search Tempo for service.name
|
||||
curl -s "$TEMPO/api/search?tags=service.name=<agent-name>" | jq
|
||||
|
||||
# LLM token usage (if instrumented)
|
||||
curl -s "$PROM/api/v1/query?query=gen_ai_client_token_usage_sum" | jq
|
||||
```
|
||||
|
||||
## Common pitfalls
|
||||
|
||||
| Pitfall | Symptom | Fix |
|
||||
|---|---|---|
|
||||
| Forgot the logging bridge | Logs appear in `kubectl logs` but not Loki. | Add `configure_otlp_log_handler()` in `app/agent.py`. |
|
||||
| `OTEL_SEMCONV_STABILITY_OPT_IN` missing | Dashboards show no HTTP server latency. | Add the env var; redeploy. |
|
||||
| `OTEL_RESOURCE_ATTRIBUTES` `service.version` drifts from `.image-version` | Dashboards group by version are wrong. | Bump both together; consider templating in `deployment.yaml`. |
|
||||
| `LangChainInstrumentor()` called after LLM init | Spans missing token counts. | Move `.instrument()` to before `AzureChatOpenAI(...)`. |
|
||||
| Custom metric names not prefixed `agent.<type>.` | Metric collides with another agent's. | Rename and bump minor version. |
|
||||
Reference in New Issue
Block a user