agent-harness-spec/docs/observability.md

# Observability — OpenTelemetry wiring

> Normative requirements: [SPEC.md §7](../SPEC.md#7-opentelemetry-wiring-must)

The platform runs the OpenTelemetry Operator. Every agent gets the SDK injected as an init container; the agent code itself does **not** install or configure SDK exporters. This document explains what is provided for free, what the agent must wire by hand, and how the signals flow to the cluster collectors.

## Pipeline

```mermaid
flowchart LR
    A[Agent pod] -->|OTLP gRPC| T1[Tier-1 collector]
    T1 -->|metrics| P[Prometheus RemoteWrite]
    T1 -->|logs| L[Loki]
    T1 -->|traces| TP[Tempo]
    T1 -->|spanmetrics| P
```

- **Tier-1 collector** runs as a DaemonSet, decorates spans/logs with `k8sattributes` (pod, namespace, node), and forwards to tier-2.
- **Tier-2 collector** generates spanmetrics from traces, exports OTLP traces to Tempo, OTLP logs to Loki, and Prometheus remote-writes metrics.
- The agent only needs OTLP gRPC reachability to `tier-1` — everything else is the operator's job.

## What the operator gives you for free

The `Instrumentation` CR `monitoring/otel-instrumentation` injects:

- **Tracer/Meter/Logger providers** with OTLP exporters configured.
- **Auto-instrumented HTTP server** (`opentelemetry-instrumentation-asgi` or `-fastapi`).
- **Auto-instrumented outbound HTTP** (`urllib3`, `httpx`, `requests`).
- **Logger provider** with `BatchLogRecordProcessor` — but **does not** attach a `LoggingHandler` to Python's stdlib root logger. That is the agent's job.

## What the agent must wire by hand

### 1. The pod template annotations

```yaml
metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
    instrumentation.opentelemetry.io/container-names: "<container-name>"
```

`container-names` **MUST** match the container `name`. If you have multiple containers (sidecar, etc.), only list the agent container.

### 2. The standard env block

These six env vars are mandatory:

```yaml
- name: OTEL_SERVICE_NAME
  value: "<agent-name>"
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
- name: OTEL_LOGS_EXPORTER
  value: "otlp"
- name: OTEL_METRICS_EXPORTER
  value: "otlp"
- name: OTEL_METRIC_EXPORT_INTERVAL
  value: "15000"
- name: OTEL_SEMCONV_STABILITY_OPT_IN
  value: "http"
```

#### `OTEL_SERVICE_NAME`

The unique service identity. Used as the `service.name` resource attribute on every span/metric/log. **MUST** match the agent's lower-hyphen identifier.

#### `OTEL_RESOURCE_ATTRIBUTES`

Comma-separated `key=value` pairs merged into the resource. Required keys:

- `service.namespace` — the Kubernetes namespace.
- `service.version` — semver string. **MUST** match the contents of the agent's `.image-version` file.
- `deployment.environment` — `prod` / `dev` / `local`.
- `agent.type` — stable lower-hyphen identifier (`policy-transformer`, `golden-path`, `registry`, etc.). This is the primary metric/log filter in dashboards.

#### `OTEL_METRIC_EXPORT_INTERVAL`

`15000` ms (15 s) is platform-wide. Faster than this generates noise; slower hides incidents.

#### `OTEL_SEMCONV_STABILITY_OPT_IN`

`http` opts the SDK into the **stable** HTTP semantic conventions:

- `http.server.request.duration` (histogram)
- `http.response.status_code` (attribute)

Without this flag the SDK emits the legacy `http.server.duration` + `http.status_code`, which the platform dashboards no longer query.

### 3. The logging bridge

The OTel Logger provider exists but isn't connected to stdlib `logging`. The agent **MUST** ship `app/logging_setup.py` and call `configure_otlp_log_handler()` once at module import time in `app/agent.py`.

The canonical implementation is in [`../template/app/logging_setup.py`](../template/app/logging_setup.py). Copy it verbatim — it is intentionally short and idempotent:

```python
def configure_otlp_log_handler(level: int = logging.INFO) -> None:
    try:
        from opentelemetry._logs import get_logger_provider
        from opentelemetry.sdk._logs import LoggingHandler
    except ImportError:
        return
    provider = get_logger_provider()
    if provider is None or type(provider).__name__ == "NoOpLoggerProvider":
        return
    root = logging.getLogger()
    if any(isinstance(h, LoggingHandler) for h in root.handlers):
        return
    root.addHandler(LoggingHandler(level=level, logger_provider=provider))
    if root.level == logging.NOTSET or root.level > level:
        root.setLevel(level)
```

The function is a no-op outside the cluster (the SDK packages aren't installed), so the same call site works locally.

### 4. Custom metrics

Custom metrics live in `app/metrics.py`. The shape:

```python
from opentelemetry import metrics

_meter = metrics.get_meter("<agent-name>", "<X.Y.Z>")

work_done = _meter.create_counter(
    "agent.<type>.runs",
    unit="1",
    description="Work units processed by outcome.",
)

work_duration = _meter.create_histogram(
    "agent.<type>.duration",
    unit="s",
    description="End-to-end work latency.",
)
```

Naming rules (mandatory):

- Prefix all metric names with `agent.<type>.` where `<type>` matches `agent.type` in `OTEL_RESOURCE_ATTRIBUTES`.
- Counters: noun (`runs`, `tokens`, `errors`).
- Histograms: noun + `.duration` (seconds).
- Use `unit="1"` for counts, `unit="s"` for seconds, `unit="By"` for bytes.

Recording from a route handler:

```python
from .metrics import work_done, work_duration

async def handler(request):
    started = time.perf_counter()
    try:
        outcome = await do_the_thing(...)
        work_done.add(1, {"outcome": "success"})
        return JSONResponse(outcome)
    except ValueError as exc:
        work_done.add(1, {"outcome": "bad_request"})
        raise
    except Exception:
        work_done.add(1, {"outcome": "error"})
        raise
    finally:
        work_duration.record(time.perf_counter() - started)
```

Prefer **inline metric calls at every return path** over outer `try/finally` wrappers — it avoids indentation churn and reads cleaner. Use `try/finally` only for histograms that always need to record regardless of outcome.

Do **not** create `MeterProvider` instances or call `set_meter_provider()` — the operator owns the global provider.

### 5. LangChain / LLM instrumentation

Agents that call an LLM via LangChain **MUST** wire OpenInference:

```python
from openinference.instrumentation.langchain import LangChainInstrumentor

LangChainInstrumentor().instrument()
```

Call this **once**, before any `AzureChatOpenAI(...)` (or other chat-model) construction. It emits OpenInference spans with token counts, prompt/response payloads, and tool calls — visible in Tempo and surfacable as LLM dashboards.

Multiple `instrument()` calls are safe but log a warning. If you compose modules across `__main__` and `app.agent`, expect to see the warning once per import path; this is harmless.

## Verifying signals end-to-end

After deploying, check each signal flows:

```bash
# Logs — should appear in Loki within ~30s
kubectl logs deployment/<agent> -c <agent-name> | head

# Metrics — query Prometheus for the meter name
curl -s "$PROM/api/v1/query?query=agent_<type>_runs_total" | jq

# Traces — search Tempo for service.name
curl -s "$TEMPO/api/search?tags=service.name=<agent-name>" | jq

# LLM token usage (if instrumented)
curl -s "$PROM/api/v1/query?query=gen_ai_client_token_usage_sum" | jq
```

## Common pitfalls

| Pitfall | Symptom | Fix |
|---|---|---|
| Forgot the logging bridge | Logs appear in `kubectl logs` but not Loki. | Add `configure_otlp_log_handler()` in `app/agent.py`. |
| `OTEL_SEMCONV_STABILITY_OPT_IN` missing | Dashboards show no HTTP server latency. | Add the env var; redeploy. |
| `OTEL_RESOURCE_ATTRIBUTES` `service.version` drifts from `.image-version` | Dashboards group by version are wrong. | Bump both together; consider templating in `deployment.yaml`. |
| `LangChainInstrumentor()` called after LLM init | Spans missing token counts. | Move `.instrument()` to before `AzureChatOpenAI(...)`. |
| Custom metric names not prefixed `agent.<type>.` | Metric collides with another agent's. | Rename and bump minor version. |