agent-harness-spec/docs/observability.md

Observability — OpenTelemetry wiring

Normative requirements: SPEC.md §7

The platform runs the OpenTelemetry Operator. Every agent gets the SDK injected as an init container; the agent code itself does not install or configure SDK exporters. This document explains what is provided for free, what the agent must wire by hand, and how the signals flow to the cluster collectors.

Pipeline

flowchart LR
    A[Agent pod] -->|OTLP gRPC| T1[Tier-1 collector]
    T1 -->|metrics| P[Prometheus RemoteWrite]
    T1 -->|logs| L[Loki]
    T1 -->|traces| TP[Tempo]
    T1 -->|spanmetrics| P

• Tier-1 collector runs as a DaemonSet, decorates spans/logs with k8sattributes (pod, namespace, node), and forwards to tier-2.
• Tier-2 collector generates spanmetrics from traces, exports OTLP traces to Tempo, OTLP logs to Loki, and Prometheus remote-writes metrics.
• The agent only needs OTLP gRPC reachability to tier-1 — everything else is the operator's job.

What the operator gives you for free

The Instrumentation CR monitoring/otel-instrumentation injects:

• Tracer/Meter/Logger providers with OTLP exporters configured.
• Auto-instrumented HTTP server (opentelemetry-instrumentation-asgi or -fastapi).
• Auto-instrumented outbound HTTP (urllib3, httpx, requests).
• Logger provider with BatchLogRecordProcessor — but does not attach a LoggingHandler to Python's stdlib root logger. That is the agent's job.

What the agent must wire by hand

1. The pod template annotations

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
    instrumentation.opentelemetry.io/container-names: "<container-name>"

container-names MUST match the container name. If you have multiple containers (sidecar, etc.), only list the agent container.

2. The standard env block

These six env vars are mandatory:

- name: OTEL_SERVICE_NAME
  value: "<agent-name>"
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "service.namespace=agents,service.version=<X.Y.Z>,deployment.environment=<env>,agent.type=<role>"
- name: OTEL_LOGS_EXPORTER
  value: "otlp"
- name: OTEL_METRICS_EXPORTER
  value: "otlp"
- name: OTEL_METRIC_EXPORT_INTERVAL
  value: "15000"
- name: OTEL_SEMCONV_STABILITY_OPT_IN
  value: "http"

OTEL_SERVICE_NAME

The unique service identity. Used as the service.name resource attribute on every span/metric/log. MUST match the agent's lower-hyphen identifier.

OTEL_RESOURCE_ATTRIBUTES

Comma-separated key=value pairs merged into the resource. Required keys:

• service.namespace — the Kubernetes namespace.
• service.version — semver string. MUST match the contents of the agent's .image-version file.
• deployment.environment — prod / dev / local.
• agent.type — stable lower-hyphen identifier (policy-transformer, golden-path, registry, etc.). This is the primary metric/log filter in dashboards.

OTEL_METRIC_EXPORT_INTERVAL

15000 ms (15 s) is platform-wide. Faster than this generates noise; slower hides incidents.

OTEL_SEMCONV_STABILITY_OPT_IN

http opts the SDK into the stable HTTP semantic conventions:

• http.server.request.duration (histogram)
• http.response.status_code (attribute)

Without this flag the SDK emits the legacy http.server.duration + http.status_code, which the platform dashboards no longer query.

3. The logging bridge

The OTel Logger provider exists but isn't connected to stdlib logging. The agent MUST ship app/logging_setup.py and call configure_otlp_log_handler() once at module import time in app/agent.py.

The canonical implementation is in ../template/app/logging_setup.py. Copy it verbatim — it is intentionally short and idempotent:

def configure_otlp_log_handler(level: int = logging.INFO) -> None:
    try:
        from opentelemetry._logs import get_logger_provider
        from opentelemetry.sdk._logs import LoggingHandler
    except ImportError:
        return
    provider = get_logger_provider()
    if provider is None or type(provider).__name__ == "NoOpLoggerProvider":
        return
    root = logging.getLogger()
    if any(isinstance(h, LoggingHandler) for h in root.handlers):
        return
    root.addHandler(LoggingHandler(level=level, logger_provider=provider))
    if root.level == logging.NOTSET or root.level > level:
        root.setLevel(level)

The function is a no-op outside the cluster (the SDK packages aren't installed), so the same call site works locally.

4. Custom metrics

Custom metrics live in app/metrics.py. The shape:

from opentelemetry import metrics

_meter = metrics.get_meter("<agent-name>", "<X.Y.Z>")

work_done = _meter.create_counter(
    "agent.<type>.runs",
    unit="1",
    description="Work units processed by outcome.",
)

work_duration = _meter.create_histogram(
    "agent.<type>.duration",
    unit="s",
    description="End-to-end work latency.",
)

Naming rules (mandatory):

• Prefix all metric names with agent.<type>. where <type> matches agent.type in OTEL_RESOURCE_ATTRIBUTES.
• Counters: noun (runs, tokens, errors).
• Histograms: noun + .duration (seconds).
• Use unit="1" for counts, unit="s" for seconds, unit="By" for bytes.

Recording from a route handler:

from .metrics import work_done, work_duration

async def handler(request):
    started = time.perf_counter()
    try:
        outcome = await do_the_thing(...)
        work_done.add(1, {"outcome": "success"})
        return JSONResponse(outcome)
    except ValueError as exc:
        work_done.add(1, {"outcome": "bad_request"})
        raise
    except Exception:
        work_done.add(1, {"outcome": "error"})
        raise
    finally:
        work_duration.record(time.perf_counter() - started)

Prefer inline metric calls at every return path over outer try/finally wrappers — it avoids indentation churn and reads cleaner. Use try/finally only for histograms that always need to record regardless of outcome.

Do not create MeterProvider instances or call set_meter_provider() — the operator owns the global provider.

5. LangChain / LLM instrumentation

Agents that call an LLM via LangChain MUST wire OpenInference:

from openinference.instrumentation.langchain import LangChainInstrumentor

LangChainInstrumentor().instrument()

Call this once, before any AzureChatOpenAI(...) (or other chat-model) construction. It emits OpenInference spans with token counts, prompt/response payloads, and tool calls — visible in Tempo and surfacable as LLM dashboards.

Multiple instrument() calls are safe but log a warning. If you compose modules across __main__ and app.agent, expect to see the warning once per import path; this is harmless.

Verifying signals end-to-end

After deploying, check each signal flows:

# Logs — should appear in Loki within ~30s
kubectl logs deployment/<agent> -c <agent-name> | head

# Metrics — query Prometheus for the meter name
curl -s "$PROM/api/v1/query?query=agent_<type>_runs_total" | jq

# Traces — search Tempo for service.name
curl -s "$TEMPO/api/search?tags=service.name=<agent-name>" | jq

# LLM token usage (if instrumented)
curl -s "$PROM/api/v1/query?query=gen_ai_client_token_usage_sum" | jq

Common pitfalls

Pitfall	Symptom	Fix
Forgot the logging bridge	Logs appear in kubectl logs but not Loki.	Add configure_otlp_log_handler() in app/agent.py.
OTEL_SEMCONV_STABILITY_OPT_IN missing	Dashboards show no HTTP server latency.	Add the env var; redeploy.
OTEL_RESOURCE_ATTRIBUTES service.version drifts from .image-version	Dashboards group by version are wrong.	Bump both together; consider templating in deployment.yaml.
LangChainInstrumentor() called after LLM init	Spans missing token counts.	Move .instrument() to before AzureChatOpenAI(...).
Custom metric names not prefixed agent.<type>.	Metric collides with another agent's.	Rename and bump minor version.