Files
cp1-shakedown/docs/implementation-plan.md
Jonathan Boniface c0e4710eb7 Shakedown implementation plan
Rate, experiment on and build use cases for KAB workflows, starting with
ALZ-F02. LangChain Deep Agents with Claude Sonnet 5.5 (Azure) as the
agent and judge; deployed to crucible-kab-dev-aks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 14:55:07 +00:00

23 KiB

Shakedown: implementation plan

Shakedown is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs, runs experiments on them, and builds use cases for them. On a ship, the shakedown cruise proves the vessel before it enters service. This app does the same for every workflow before its outputs reach an Engagement Lead.

Date 2026-11-09 (revised the same day with the product owner's decisions, §8)
Status Draft. Nothing is built yet
Repository agents/cp1-shakedown on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in cp1-agents
Runs on The crucible-kab-dev-aks cluster
Deep agent LangChain Deep Agents (deepagents, on LangGraph), driven by Claude Sonnet 5.5 on the Kyndryl Azure resource
Judge Claude Sonnet 5.5, a different model family from the gpt-5.x agents under test
First workflow under test ALZ-F02 Assessment & Disposition (wf_7b6cae08-…, individual pattern)
Then ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded before its first live run

1. Why: the work Shakedown replaces

The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004 to 015, commit 66c3274). Almost every step of that work was done by hand:

Manual activity in the session Evidence What Shakedown does instead
Refreshing the 5-minute KAB token about 40 "kab token refreshed" prompts You sign in once through KAB's Keycloak; the server-side session renews the token itself
Starting a run, polling it, pulling steps.json kab_run_workflow.py start/status/outputs A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage
Keeping rollback copies of the workflow before each run 11 workflow-rollback-wf_*.json files in reports/cp1 A snapshot of the workflow and the agents is taken automatically for every run
Checking report figures against the facts PLAN.md improvement 4: "checked by hand" in runs 014 and 015 Every report figure is checked in code against the facts the run produced
Spotting confidence drift between runs APP-001 scored 0.50 or 0.60 on the same package (improvement 1) Stability experiments: the same input N times, with the variance shown for each field
Spotting ambiguous rules "a compute rate" read two ways (improvement 2) The drift report links each divergent field to the prompt rule the agent cited
Finding the costliest agent security posture agent at 520k to 540k tokens (improvement 5) Cost and latency per step and per run, against the case's limits:
Judging whether an output is useful "it doesn't break down what the application is within the report" A business-value rubric scored on every output, not only a schema check
Writing checks for a case Meridian checks still todo (F02 PLAN step 10) The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run

The acceptance test for the whole product: given F02 runs 010 to 015, Shakedown finds the APP-001 confidence drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report, without anyone pointing it at them.

2. What Shakedown does

Four capabilities, each usable through the UI and by the deep agent (§4):

  1. Rate. Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
    • Checks: the existing case format (case.yaml checks: with contains, item.match, absent, equals, severities), run by a generic case runner that works for any workflow, not only F01.
    • Invariants in code: the run's figures trace to its facts, the package digests recompute, ids resolve across steps, every evidence_ref names a real execution, and token and time limits: are met.
    • Business value: a rubric scored by Claude Sonnet with quoted evidence (§5).
  2. Experiment. Change one variable and measure the effect against a baseline:
    • Stability: the same input N times (default 5), with the spread for each output field.
    • Step replay: re-send one step's recorded input straight to an agent over A2A (message/send on http://<agent>.agents.svc.cluster.local:8080). This needs no full 12-step workflow run, and no upstream agent tokens are spent.
    • Variants: prompt, model, LITELLM_MAX_TOKENS or a code hook changed in a shadow deployment of the agent, with the same replay inputs sent to the baseline and to the variant.
  3. Build use cases. Cases are drafted from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). Synthetic estates go in alz-discovery-sample, which is the one server that may be extended for eval data.
  4. Align. Check every workflow hand-off, F01 package → F02 intake → F03 intake:
    • Statically: the producer's output.json against the consumer's input.json, and the fields each consumer's prompt reads.
    • Live: a real F01 package run through F02's intake.
    • Traceability: each ADO acceptance criterion has at least one check, and each check names its criterion.

3. Repository: use cases and experiments, apart from the agents

The new gitea repository holds Shakedown's code, the definitions of new use cases and experiments, and their conclusions. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each experiment's record links to its runs by execution id.

cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/                     # FastAPI API + UI
├── bosun/                   # the deep agent
│   ├── agent.py             #   create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│   ├── subagents/           #   one module per sub-agent: prompt + tool list
│   └── tools/               #   KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
├── shakedown_lib/           # case runner, invariants, rubric scoring, KAB client, ledger
├── use-cases/<workstream>/<workflow>/<case-id>/
│   ├── case.yaml            #   checks, value bars, limits; frozen (hashed) before the first run
│   ├── context.yaml         #   run context (workflow_inputs, customer, plan, context): no run_id, no credentials
│   └── uploads/, README.md
├── rubrics/<workflow>.yaml  # business-value rubric for each workflow (§5)
├── experiments/EXP-NNN-<slug>/
│   ├── experiment.yaml      #   hypothesis, one variable, baseline, variants, N, metric, stop rule
│   └── results.md           #   conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
├── variants/<agent>/<variant>/   # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag
└── deploy/                  # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
  • Existing cases stay where they are. F01's 7 workflow cases, F02's 2, and each agent's eval/cases.yaml remain in cp1-agents. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its own cases with their source shown. Only new use cases and all experiments go into the Shakedown repository.
  • No change to cp1-agents. Shakedown imports the scripts it reuses (kab_run_workflow.py's Kab, load_context and new_ids; evaluate_workflow_run.py; collect_run_usage.py; the package agents' scorecard.py) from that pinned clone by path, instead of extracting a shared package from cp1-agents.
  • Reproducible results: every ledger run records the commit of cp1-agents and of Shakedown that it ran against.
  • Changes reach the repositories as pull requests. Bosun opens a gitea pull request for a new case, an experiment record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.

4. Architecture on the cluster

                       you (browser) ── Keycloak sign-in (realm kab) ──┐
                                                                       ▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│  shakedown-app  (FastAPI + UI, one replica)                                    │
│    ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│    ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger    │
│    │     └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret)     │
│    ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API     │
│    └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│
│  shadow agents  (variants/*, applied only with approval)                       │
└────────────────────────────────────────────────────────────────────────────────┘
      │ HTTPS: api.kab-dev…    │ A2A :8080: agents.svc   │ read-only RBAC: pods/log in `agents`
  • Sign-in and tokens: Shakedown is an OIDC client of KAB's Keycloak (id.kab-dev.kyndemo.live, realm kab). The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. This replaces kab_token.py for everything Shakedown does.
  • Reading the cluster: a ServiceAccount with read-only RBAC (get/list on pods and pods/log in agents) replaces collect_run_usage.py's kubeconfig. It gets no secrets access.
  • The UI is exposed through an ingress from Phase 0, with Keycloak sign-in in front of it. There's no port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
  • Shadow agents: these are applied to the shakedown namespace, beside Shakedown and apart from the live agents in agents. They reach the gateways through the same deploy/ profile values. Shakedown's ServiceAccount cannot apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' rule that each cluster change is approved.
  • Model: Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, which doesn't serve it. The key is in Secret shakedown-secrets, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
  • Tracing: LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to the ledger instead.
  • Images and manifests: built with buildah, in parallel, and pushed to cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/<component>:<version>. Manifests are rendered like cp1-agents', and every apply needs your approval.
  • The ledger records each run as a folder: context.yaml (with generated:), workflow.json, agents.json (image, version, model, max tokens and temperature for each step's agent), steps.json, usage.json, scores.json and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first data load.

5. The deep agent: "Bosun"

The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that changes state.

Built with LangChain Deep Agents. create_deep_agent provides the planning (to-do) tool, a file workspace, sub-agents with their own prompts and tools, and interrupt_on, which pauses before a named tool runs. Shakedown uses them as follows:

Deep Agents feature Used for
Planning tool Navigator's experiment plan, shown live in the UI
File workspace (backend) A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied
Sub-agents The six roles below, each with only the tools its job needs, which keeps the main agent's context small
interrupt_on Approval in the UI before start_run, start_replay_batch, open_pull_request and render_variant
LangGraph checkpointer A long experiment survives a pod restart, and its whole trail can be audited afterwards
Sub-agent Job Tools Writes
Navigator (planner) Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule ledger read, repo read experiment.yaml draft
Lookout (run operator) Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you KAB, replay, cluster read Ledger entries
Surveyor (judge) Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring case runner, rubric, ledger read scores.json
Shipwright (fixer) Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes repo read, gitea PR Pull request to cp1-agents, never merged by Bosun
Chartmaker (case author) Drafts cases and checks from ADO acceptance criteria and worked examples, and adversarial variants of them ADO read, repo read, gitea PR Pull request to Shakedown
Purser (cost) Tokens, time and cost per step; flags steps over limits: and the cheapest variant that keeps the score ledger read Cost section of the scores

Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores.

How a loop runs: a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today.

Guardrails, enforced in code (tool permissions, interrupt_on, RBAC), not only in prompts:

  • A HITL review (continue/stop/rerun) is a person's decision. Bosun has no tool that answers one.
  • Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
  • No changes to ADO work items. Bosun's ADO tools are read-only.
  • No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
  • Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as data in every sub-agent's context, and a planted-injection case runs against Bosun itself.
  • Checks are frozen before the run. A case's hash is recorded when a run starts, and a check edited afterwards can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README).
  • The patterns stay apart: a finding on individual-agents/ never proposes a change to sub-agents/ or to outcome-pattern/. Solution architecture and Discovery are never linked.
  • A model change for an agent under test is proposed, never made.

6. Business value: rating quality, not volume

Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric (rubrics/<workflow>.yaml), starting from the six dimensions below. Each score carries a short quote from the output and a reason.

Dimension Question Fails when
Decision-ready Can the reader make the decision this step exists for? It lists data and leaves the conclusion to the reader
Traceable Does every claim and figure trace to a fact and an evidence_ref? A figure appears that no fact supports (checked in code first)
Owned and actionable Does each finding name an owner and a next action? "Investigate" with no owner (extends the F01 scorecard's with_owner_share and with_action_share)
Explained Are the subjects described in business terms (which application it is, who uses it, why it matters)? Ids without descriptions: the APP-001 report gap
Consumable downstream Does the next workflow get every field it needs, filled or declared missing? Fields filled with placeholders (extends f02_fields.filled_or_declared)
Signal over volume Is the output proportionate: no restated inputs, no duplicates, no padding? Raw records passed through; the same finding written three times

The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a case can fail on value as well as on checks, through its value: bars.

Calibrating the judge: before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015). Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its model version or the rubric changes.

7. Delivery phases

Each phase ends with a check that proves it. The phase's status is kept in cp1-shakedown/.agents/PLAN.md once the repository exists.

# Phase Delivers Verified by
0 Foundations Repository scaffold in cp1-shakedown; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; shakedown namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; first deploy Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets
1 Ledger and rating Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI The azure-crucible-kab-dev-assessment checks score runs 014 and 015; a report figure changed in a copy of a run is caught
2 Bosun v1, read-only Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" The acceptance test in §1 passes on runs 010 to 015
3 Experiments Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; experiments/ records by pull request 5 replays of F02 s5 on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread
4 Alignment and value Static contract checks F01 → F02 → F03; live hand-off test; acceptance-criteria traceability from ADO; F02 rubric and judge calibration Every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs
5 Use-case factory Chartmaker and Shipwright; drafting of adversarial cases; a patch pull request and ADR draft for each confirmed finding F03's cases and checks are merged in Shakedown before F03's first live run; F03 is onboarded with no new Shakedown code
6 Scale out F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows A new workflow is onboarded by adding config only: its export path, cases and rubric

Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back.

8. Seed experiments (first experiments/ entries)

These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it.

  1. EXP-001 P2 counting unit (improvement 1): stability on s5 with the current and the proposed wording; the metric is the spread of APP-001's confidence.
  2. EXP-002 P1 to P3 in code (improvement 3): a variant hook; the metric is the spread (target 0) and the change in the other checks.
  3. EXP-003 "Compute rate" definition (improvement 2): replay with the clarified 7R rule; the metric is a consistent Rehost precondition across runs.
  4. EXP-004 Security posture cost (improvement 5): LITELLM_MAX_TOKENS tiers, and gpt-4.1 against gpt-5.6-sol; the metrics are tokens and the value score. A model change remains your call.
  5. EXP-005 Report figures (improvement 4): turns the hand check into a permanent invariant.
  6. Temperature, as context for all of them: every agent runs at LITELLM_TEMPERATURE: "1", probably because the gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming it can be tuned away.

9. Decisions

Decided (2026-11-09):

Decision Choice
Deep agent framework LangChain Deep Agents
Bosun's model and the judge Claude Sonnet 5.5 for both
Model route The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API
Repository agents/cp1-shakedown on gitea, for use cases and experiments, apart from the agents and workflows
Existing cases Stay in cp1-agents and are read by path; only new use cases and all experiments go to Shakedown
Hosting The crucible cluster
Shadow agents The shakedown namespace
UI access Ingress with Keycloak sign-in from Phase 0 (no port-forward stage)

Still open:

  1. Keycloak client and hostname (platform ask, blocks Phase 0): the KAB platform team registers a cp1-shakedown OIDC client in realm kab, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS certificate under kyndemo.live.
  2. Gitea bot account: a gitea token that Bosun uses to open pull requests on cp1-shakedown and cp1-agents, held in a Secret, with rights limited to opening pull requests.
  3. Variant agent keys (to verify with KAB): whether a variant in shakedown may reuse the agent's MCP_AGENT_API_KEY Secret (copied by a person) or needs its own KAB registration.
  4. Rubric ownership: who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the rubric represents.

10. Out of scope

  • Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
  • Writing to ADO work items.
  • Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
  • Moving the existing cases out of cp1-agents.