agent-harness-spec/docs/kubernetes.md

Kubernetes deployment shape

Normative requirements: SPEC.md §9

Every agent ships at minimum two YAML files under k8s/:

• configmap.yaml — non-secret config (URLs, log level, port, etc.).
• deployment.yaml — Deployment + Service (concatenated with ---).

The reference template under ../template/k8s/ is a copy-and-fill skeleton.

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: <agent-name>-config
  namespace: agents
data:
  # Required — see docs/registration.md
  REGISTRY_URL: "http://agent-gateway.agents.svc.cluster.local"
  AGENT_SELF_URL: "http://<agent-name>.agents.svc.cluster.local"

  # Standard knobs
  LOG_LEVEL: "INFO"
  PORT: "8000"

  # Agent-specific (LLM, cache, feature flags, etc.)
  AZURE_OPENAI_DEPLOYMENT: "gpt-4o"
  AZURE_OPENAI_API_VERSION: "2024-08-01-preview"

The agent's container MUST consume the ConfigMap via envFrom: configMapRef. Inline env: is reserved for OTel vars and secretKeyRef references.

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <agent-name>
  namespace: agents
  labels:
    app: <agent-name>
    component: agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: <agent-name>
  template:
    metadata:
      labels:
        app: <agent-name>
        azure.workload.identity/use: "true"
      annotations:
        instrumentation.opentelemetry.io/inject-python: "monitoring/otel-instrumentation"
        instrumentation.opentelemetry.io/container-names: "<agent-name>"
    spec:
      serviceAccountName: agents-sa
      containers:
        - name: <agent-name>
          image: bstagecjotdevacr.azurecr.io/<agent-name>:latest
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          envFrom:
            - configMapRef:
                name: <agent-name>-config
          env:
            # ─── OTel (mandatory) ───────────────────────────────────────────
            - name: OTEL_SERVICE_NAME
              value: "<agent-name>"
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "service.namespace=agents,service.version=1.0.0,deployment.environment=prod,agent.type=<role>"
            - name: OTEL_LOGS_EXPORTER
              value: "otlp"
            - name: OTEL_METRICS_EXPORTER
              value: "otlp"
            - name: OTEL_METRIC_EXPORT_INTERVAL
              value: "15000"
            - name: OTEL_SEMCONV_STABILITY_OPT_IN
              value: "http"

            # ─── Secrets from agents-kv-sync (Key Vault sync) ───────────────
            - name: AZURE_OPENAI_ENDPOINT
              valueFrom:
                secretKeyRef:
                  name: agents-kv-sync
                  key: azure-openai-endpoint
            - name: AZURE_OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: agents-kv-sync
                  key: azure-openai-api-key

          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "1Gi"
              cpu: "500m"

          livenessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 20
            periodSeconds: 15
            timeoutSeconds: 5
            failureThreshold: 3

          readinessProbe:
            httpGet:
              path: /health
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true

      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: agents-kv-spc
---
apiVersion: v1
kind: Service
metadata:
  name: <agent-name>
  namespace: agents
  labels:
    app: <agent-name>
spec:
  selector:
    app: <agent-name>
  ports:
    - name: http
      port: 80
      targetPort: 8000
      protocol: TCP
  type: ClusterIP

Mandatory cross-references

These names are platform conventions and MUST be used as-written:

Name	Kind	Purpose
agents	Namespace	All agents live here unless they have a strong domain reason for their own namespace.
agents-sa	ServiceAccount	Workload-identity bound to the agents Key Vault.
agents-kv-sync	Secret	Synced from Key Vault by the SecretProviderClass. Source of secretKeyRef values.
agents-kv-spc	SecretProviderClass	Mounts Key Vault into /mnt/secrets-store and rotates the synced Secret.
monitoring/otel-instrumentation	Instrumentation CR	OTel SDK init container injection.

Probes

Tune the timing per agent — these are sane defaults:

Probe	Purpose	initialDelaySeconds	periodSeconds	failureThreshold
liveness	Restart hung pod.	20	15	3
readiness	Hold traffic until manifest is serving.	10	10	3

Agents with heavy startup work (large LangGraph compile, prompt caches, model warm-ups) SHOULD raise initialDelaySeconds rather than reduce failureThreshold.

Per-namespace agents

Some agents (e.g. modernization-factory-v2) run in their own namespace. When they do:

• The serviceAccountName MUST be created in that namespace and bound to the same workload identity (agents-sa is conventional, even when the namespace differs).
• OTEL_RESOURCE_ATTRIBUTES's service.namespace MUST match the namespace.
• AGENT_SELF_URL MUST point to the cross-namespace FQDN (http://<agent>.<namespace>.svc.cluster.local); this is allow-listed by the gateway as long as the host ends in .svc.cluster.local.

Resource sizing reference

Observed in production:

Agent class	CPU req / lim	Mem req / lim
Stateless small (e.g. support-intake-agent)	100m / 250m	256Mi / 512Mi
Standard LLM agent (policy-transformer, decomposer)	250m / 500m	512Mi / 1Gi
Heavy workflow (golden-path, modernization-factory-v1)	500m–1 / 1–2	1Gi–2Gi / 2Gi–4Gi

When in doubt start with the Standard row and revisit after one week of production traffic.