agent-harness-spec/docs/kubernetes.md

# Kubernetes deployment shape

> Normative requirements: [SPEC.md §9](../SPEC.md#9-kubernetes-manifests-must)

Every agent ships at minimum two YAML files under `k8s/`:

- `configmap.yaml` — non-secret config (URLs, log level, port, etc.).
- `deployment.yaml` — `Deployment` + `Service` (concatenated with `---`).

The reference template under [`../template/k8s/`](../template/k8s/) is a copy-and-fill skeleton.

## ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: <agent-name>-config
  namespace: agents
data:
  # Required — see docs/registration.md
  REGISTRY_URL: "http://agent-gateway.agents.svc.cluster.local"
  AGENT_SELF_URL: "http://<agent-name>.agents.svc.cluster.local"

  # Standard knobs
  LOG_LEVEL: "INFO"
  PORT: "8000"

  # Agent-specific (LLM, cache, feature flags, etc.)
  AZURE_OPENAI_DEPLOYMENT: "gpt-4o"
  AZURE_OPENAI_API_VERSION: "2024-08-01-preview"
```

The agent's container **MUST** consume the ConfigMap via `envFrom: configMapRef`. Inline `env:` is reserved for OTel vars and `secretKeyRef` references.

## Deployment

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: <agent-name>
  namespace: agents
  labels:
    app: <agent-name>
    component: agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <agent-name>
  template:
    metadata:
      labels:
        app: <agent-name>
        azure.workload.identity/use: "true"
      annotations:
        instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
        instrumentation.opentelemetry.io/container-names: "<agent-name>"
    spec:
      serviceAccountName: agents-sa
      containers:
        - name: <agent-name>
          image: bstagecjotdevacr.azurecr.io/<agent-name>:latest
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          envFrom:
            - configMapRef:
                name: <agent-name>-config
          env:
            # ─── OTel (mandatory) ───────────────────────────────────────────
            - name: OTEL_SERVICE_NAME
              value: "<agent-name>"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "service.namespace=agents,service.version=1.0.0,deployment.environment=prod,agent.type=<role>"
            - name: OTEL_LOGS_EXPORTER
              value: "otlp"
            - name: OTEL_METRICS_EXPORTER
              value: "otlp"
            - name: OTEL_METRIC_EXPORT_INTERVAL
              value: "15000"
            - name: OTEL_SEMCONV_STABILITY_OPT_IN
              value: "http"

            # ─── Secrets from agents-kv-sync (Key Vault sync) ───────────────
            - name: AZURE_OPENAI_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: agents-kv-sync
                  key: azure-openai-endpoint
            - name: AZURE_OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: agents-kv-sync
                  key: azure-openai-api-key

          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "500m"

          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 20
            periodSeconds: 15
            timeoutSeconds: 5
            failureThreshold: 3

          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true

      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: agents-kv-spc
---
apiVersion: v1
kind: Service
metadata:
  name: <agent-name>
  namespace: agents
  labels:
    app: <agent-name>
spec:
  selector:
    app: <agent-name>
  ports:
    - name: http
      port: 80
      targetPort: 8000
      protocol: TCP
  type: ClusterIP
```

## Mandatory cross-references

These names are platform conventions and **MUST** be used as-written:

| Name | Kind | Purpose |
|---|---|---|
| `agents` | Namespace | All agents live here unless they have a strong domain reason for their own namespace. |
| `agents-sa` | ServiceAccount | Workload-identity bound to the agents Key Vault. |
| `agents-kv-sync` | Secret | Synced from Key Vault by the SecretProviderClass. Source of `secretKeyRef` values. |
| `agents-kv-spc` | SecretProviderClass | Mounts Key Vault into `/mnt/secrets-store` and rotates the synced Secret. |
| `monitoring/otel-instrumentation` | Instrumentation CR | OTel SDK init container injection. |

## Probes

Tune the timing per agent — these are sane defaults:

| Probe | Purpose | `initialDelaySeconds` | `periodSeconds` | `failureThreshold` |
|---|---|---|---|---|
| `liveness` | Restart hung pod. | 20 | 15 | 3 |
| `readiness` | Hold traffic until manifest is serving. | 10 | 10 | 3 |

Agents with heavy startup work (large LangGraph compile, prompt caches, model warm-ups) **SHOULD** raise `initialDelaySeconds` rather than reduce `failureThreshold`.

## Per-namespace agents

Some agents (e.g. `modernization-factory-v2`) run in their own namespace. When they do:

- The `serviceAccountName` **MUST** be created in that namespace and bound to the same workload identity (`agents-sa` is conventional, even when the namespace differs).
- `OTEL_RESOURCE_ATTRIBUTES`'s `service.namespace` **MUST** match the namespace.
- `AGENT_SELF_URL` **MUST** point to the cross-namespace FQDN (`http://<agent>.<namespace>.svc.cluster.local`); this is allow-listed by the gateway as long as the host ends in `.svc.cluster.local`.

## Resource sizing reference

Observed in production:

| Agent class | CPU req / lim | Mem req / lim |
|---|---|---|
| Stateless small (e.g. `support-intake-agent`) | 100m / 250m | 256Mi / 512Mi |
| Standard LLM agent (`policy-transformer`, `decomposer`) | 250m / 500m | 512Mi / 1Gi |
| Heavy workflow (`golden-path`, `modernization-factory-v1`) | 500m–1 / 1–2 | 1Gi–2Gi / 2Gi–4Gi |

When in doubt start with the *Standard* row and revisit after one week of production traffic.