Shakedown implementation plan

Rate, experiment on and build use cases for KAB workflows, starting with
ALZ-F02. LangChain Deep Agents with Claude Sonnet 5.5 (Azure) as the
agent and judge; deployed to crucible-kab-dev-aks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-09 14:55:07 +00:00
commit c0e4710eb7

284
docs/implementation-plan.md Normal file
View File

@@ -0,0 +1,284 @@
# Shakedown: implementation plan
**Shakedown** is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs,
runs experiments on them, and builds use cases for them. On a ship, the *shakedown cruise* proves the vessel before it
enters service. This app does the same for every workflow before its outputs reach an Engagement Lead.
| | |
|---|---|
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §8) |
| Status | Draft. Nothing is built yet |
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
| Runs on | The `crucible-kab-dev-aks` cluster |
| Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource |
| Judge | **Claude Sonnet 5.5**, a different model family from the gpt-5.x agents under test |
| First workflow under test | ALZ-F02 Assessment & Disposition (`wf_7b6cae08-…`, individual pattern) |
| Then | ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded *before* its first live run |
---
## 1. Why: the work Shakedown replaces
The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004
to 015, commit `66c3274`). Almost every step of that work was done by hand:
| Manual activity in the session | Evidence | What Shakedown does instead |
|---|---|---|
| Refreshing the 5-minute KAB token | about 40 "kab token refreshed" prompts | You sign in once through KAB's Keycloak; the server-side session renews the token itself |
| Starting a run, polling it, pulling `steps.json` | `kab_run_workflow.py start/status/outputs` | A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage |
| Keeping rollback copies of the workflow before each run | 11 `workflow-rollback-wf_*.json` files in `reports/cp1` | A snapshot of the workflow and the agents is taken automatically for every run |
| Checking report figures against the facts | PLAN.md improvement 4: "checked by hand" in runs 014 and 015 | Every report figure is checked in code against the facts the run produced |
| Spotting confidence drift between runs | APP-001 scored 0.50 or 0.60 on the same package (improvement 1) | Stability experiments: the same input N times, with the variance shown for each field |
| Spotting ambiguous rules | "a compute rate" read two ways (improvement 2) | The drift report links each divergent field to the prompt rule the agent cited |
| Finding the costliest agent | security posture agent at 520k to 540k tokens (improvement 5) | Cost and latency per step and per run, against the case's `limits:` |
| Judging whether an output is *useful* | "it doesn't break down what the application is within the report" | A business-value rubric scored on every output, not only a schema check |
| Writing checks for a case | Meridian checks still `todo` (F02 PLAN step 10) | The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run |
**The acceptance test for the whole product:** given F02 runs 010 to 015, Shakedown finds the APP-001 confidence
drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report,
without anyone pointing it at them.
## 2. What Shakedown does
Four capabilities, each usable through the UI and by the deep agent (§4):
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
severities), run by a generic case runner that works for any workflow, not only F01.
- *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across
steps, every `evidence_ref` names a real execution, and token and time `limits:` are met.
- *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§5).
2. **Experiment.** Change one variable and measure the effect against a baseline:
- *Stability:* the same input N times (default 5), with the spread for each output field.
- *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on
`http://<agent>.agents.svc.cluster.local:8080`). This needs no full 12-step workflow run, and no upstream agent
tokens are spent.
- *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with
the same replay inputs sent to the baseline and to the variant.
3. **Build use cases.** Cases are drafted from ADO stories (acceptance criteria and worked examples), from earlier
failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate).
Synthetic estates go in `alz-discovery-sample`, which is the one server that may be extended for eval data.
4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake:
- *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's
prompt reads.
- *Live:* a real F01 package run through F02's intake.
- *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion.
## 3. Repository: use cases and experiments, apart from the agents
The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their
**conclusions**. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each
experiment's record links to its runs by execution id.
```
cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/ # FastAPI API + UI
├── bosun/ # the deep agent
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│ ├── subagents/ # one module per sub-agent: prompt + tool list
│ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger
├── use-cases/<workstream>/<workflow>/<case-id>/
│ ├── case.yaml # checks, value bars, limits; frozen (hashed) before the first run
│ ├── context.yaml # run context (workflow_inputs, customer, plan, context): no run_id, no credentials
│ └── uploads/, README.md
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§5)
├── experiments/EXP-NNN-<slug>/
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
│ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
├── variants/<agent>/<variant>/ # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag
└── deploy/ # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
```
- **Existing cases stay where they are.** F01's 7 workflow cases, F02's 2, and each agent's `eval/cases.yaml` remain
in `cp1-agents`. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its
own cases with their source shown. Only **new** use cases and **all** experiments go into the Shakedown repository.
- **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`,
`load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents'
`scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents.
- **Reproducible results:** every ledger run records the commit of cp1-agents and of Shakedown that it ran against.
- **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment
record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.
## 4. Architecture on the cluster
```
you (browser) ── Keycloak sign-in (realm kab) ──┐
▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│ shakedown-app (FastAPI + UI, one replica) │
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
│ └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│
│ shadow agents (variants/*, applied only with approval) │
└────────────────────────────────────────────────────────────────────────────────┘
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
```
- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`).
The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the
ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you.
This replaces `kab_token.py` for everything Shakedown does.
- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`)
replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access.
- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no
port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents
in `agents`. They
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents'
rule that each cluster change is approved.
- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path
from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
- **Tracing:** LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to
the ledger instead.
- **Images and manifests:** built with buildah, in parallel, and pushed to
`cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/<component>:<version>`. Manifests are rendered like cp1-agents', and
every `apply` needs your approval.
- **The ledger** records each run as a folder: `context.yaml` (with `generated:`), `workflow.json`, `agents.json`
(image, version, model, max tokens and temperature for each step's agent), `steps.json`, `usage.json`,
`scores.json` and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first
data load.
## 5. The deep agent: "Bosun"
The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a
hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that
changes state.
**Built with LangChain Deep Agents.** `create_deep_agent` provides the planning (to-do) tool, a file workspace,
sub-agents with their own prompts and tools, and `interrupt_on`, which pauses before a named tool runs. Shakedown uses
them as follows:
| Deep Agents feature | Used for |
|---|---|
| Planning tool | Navigator's experiment plan, shown live in the UI |
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` |
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
| Sub-agent | Job | Tools | Writes |
|---|---|---|---|
| **Navigator** (planner) | Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule | ledger read, repo read | `experiment.yaml` draft |
| **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries |
| **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` |
| **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun |
| **Chartmaker** (case author) | Drafts cases and checks from ADO acceptance criteria and worked examples, and adversarial variants of them | ADO read, repo read, gitea PR | Pull request to Shakedown |
| **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores |
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
the judge is visible in its scores.
**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and
Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You
review, merge, deploy and record the ADR, as today.
**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:**
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one.
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
- No changes to ADO work items. Bosun's ADO tools are read-only.
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as
data in every sub-agent's context, and a planted-injection case runs against Bosun itself.
- **Checks are frozen before the run.** A case's hash is recorded when a run starts, and a check edited afterwards
can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README).
- The patterns stay apart: a finding on `individual-agents/` never proposes a change to `sub-agents/` or to
`outcome-pattern/`. Solution architecture and Discovery are never linked.
- A model change for an agent under test is proposed, never made.
## 6. Business value: rating quality, not volume
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric
(`rubrics/<workflow>.yaml`), starting from the six dimensions below. Each score carries a short quote from the output
and a reason.
| Dimension | Question | Fails when |
|---|---|---|
| **Decision-ready** | Can the reader make the decision this step exists for? | It lists data and leaves the conclusion to the reader |
| **Traceable** | Does every claim and figure trace to a fact and an `evidence_ref`? | A figure appears that no fact supports (checked in code first) |
| **Owned and actionable** | Does each finding name an owner and a next action? | "Investigate" with no owner (extends the F01 scorecard's `with_owner_share` and `with_action_share`) |
| **Explained** | Are the subjects described in business terms (which application it is, who uses it, why it matters)? | Ids without descriptions: the APP-001 report gap |
| **Consumable downstream** | Does the next workflow get every field it needs, filled or declared missing? | Fields filled with placeholders (extends `f02_fields.filled_or_declared`) |
| **Signal over volume** | Is the output proportionate: no restated inputs, no duplicates, no padding? | Raw records passed through; the same finding written three times |
The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a
case can fail on value as well as on checks, through its `value:` bars.
**Calibrating the judge:** before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015).
Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its
model version or the rubric changes.
## 7. Delivery phases
Each phase ends with a check that proves it. The phase's status is kept in `cp1-shakedown/.agents/PLAN.md` once the
repository exists.
| # | Phase | Delivers | Verified by |
|---|---|---|---|
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
| 4 | **Alignment and value** | Static contract checks F01 → F02 → F03; live hand-off test; acceptance-criteria traceability from ADO; F02 rubric and judge calibration | Every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs |
| 5 | **Use-case factory** | Chartmaker and Shipwright; drafting of adversarial cases; a patch pull request and ADR draft for each confirmed finding | F03's cases and checks are merged in Shakedown **before** F03's first live run; F03 is onboarded with no new Shakedown code |
| 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric |
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
the experimentation loop pays back.
## 8. Seed experiments (first `experiments/` entries)
These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it.
1. **EXP-001 P2 counting unit** (improvement 1): stability on `s5` with the current and the proposed wording; the
metric is the spread of APP-001's confidence.
2. **EXP-002 P1 to P3 in code** (improvement 3): a variant hook; the metric is the spread (target 0) and the change
in the other checks.
3. **EXP-003 "Compute rate" definition** (improvement 2): replay with the clarified 7R rule; the metric is a
consistent Rehost precondition across runs.
4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against
`gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call.
5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant.
6. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the
gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming
it can be tuned away.
## 9. Decisions
**Decided (2026-11-09):**
| Decision | Choice |
|---|---|
| Deep agent framework | LangChain Deep Agents |
| Bosun's model and the judge | Claude Sonnet 5.5 for both |
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows |
| Existing cases | Stay in cp1-agents and are read by path; only new use cases and all experiments go to Shakedown |
| Hosting | The crucible cluster |
| Shadow agents | The `shakedown` namespace |
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
**Still open:**
1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown`
OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS
certificate under `kyndemo.live`.
2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`,
held in a Secret, with rights limited to opening pull requests.
3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's
`MCP_AGENT_API_KEY` Secret (copied by a person) or needs its own KAB registration.
4. **Rubric ownership:** who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the
rubric represents.
## 10. Out of scope
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
- Writing to ADO work items.
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
- Moving the existing cases out of cp1-agents.