Workflow registry by config; capabilities built read-only from ADO Epics, starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07); links as hand-off contracts checked statically, live, for drift and for use; chain-level checks from the Epic's AC1 to AC8; ripple variations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
43 KiB
Shakedown: implementation plan
Shakedown is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs, runs experiments on them, and builds use cases for them. On a ship, the shakedown cruise proves the vessel before it enters service. This app does the same for every workflow before its outputs reach an Engagement Lead.
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
| Status | Draft. Nothing is built yet |
| Repository | agents/cp1-shakedown on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in cp1-agents |
| Runs on | The crucible-kab-dev-aks cluster |
| Deep agent | LangChain Deep Agents (deepagents, on LangGraph), driven by Claude Sonnet 5.5 on the Kyndryl Azure resource |
| Judge | Claude Sonnet 5.5, a different model family from the gpt-5.x agents under test |
| First workflow under test | ALZ-F02 Assessment & Disposition (wf_7b6cae08-…, individual pattern) |
| Then | ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded before its first live run |
1. Why: the work Shakedown replaces
The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004
to 015, commit 66c3274). Almost every step of that work was done by hand:
| Manual activity in the session | Evidence | What Shakedown does instead |
|---|---|---|
| Refreshing the 5-minute KAB token | about 40 "kab token refreshed" prompts | You sign in once through KAB's Keycloak; the server-side session renews the token itself |
Starting a run, polling it, pulling steps.json |
kab_run_workflow.py start/status/outputs |
A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage |
| Keeping rollback copies of the workflow before each run | 11 workflow-rollback-wf_*.json files in reports/cp1 |
A snapshot of the workflow and the agents is taken automatically for every run |
| Checking report figures against the facts | PLAN.md improvement 4: "checked by hand" in runs 014 and 015 | Every report figure is checked in code against the facts the run produced |
| Spotting confidence drift between runs | APP-001 scored 0.50 or 0.60 on the same package (improvement 1) | Stability experiments: the same input N times, with the variance shown for each field |
| Spotting ambiguous rules | "a compute rate" read two ways (improvement 2) | The drift report links each divergent field to the prompt rule the agent cited |
| Finding the costliest agent | security posture agent at 520k to 540k tokens (improvement 5) | Cost and latency per step and per run, against the case's limits: |
| Judging whether an output is useful | "it doesn't break down what the application is within the report" | A business-value rubric scored on every output, not only a schema check |
| Writing checks for a case | Meridian checks still todo (F02 PLAN step 10) |
The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run |
The acceptance test for the whole product: given F02 runs 010 to 015, Shakedown finds the APP-001 confidence drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report, without anyone pointing it at them.
2. What Shakedown does
Shakedown does four things, each usable through the UI and by the deep agent (§5):
- Rate. Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
- Checks: the existing case format (
case.yamlchecks:withcontains,item.match,absent,equals, severities), run by a generic case runner that works for any workflow, not only F01. - Invariants in code: the run's figures trace to its facts, the package digests recompute, ids resolve across
steps, every
evidence_refnames a real execution, and token and timelimits:are met. - Business value: a rubric scored by Claude Sonnet with quoted evidence (§6).
- Checks: the existing case format (
- Experiment. Change one variable and measure the effect against a baseline:
- Stability: the same input N times (default 5), with the spread for each output field.
- Step replay: re-send one step's recorded input straight to an agent over A2A (
message/sendonhttp://<agent>.agents.svc.cluster.local:8080). This needs no full 12-step workflow run, and no upstream agent tokens are spent. - Variants: prompt, model,
LITELLM_MAX_TOKENSor a code hook changed in a shadow deployment of the agent, with the same replay inputs sent to the baseline and to the variant.
- Build use cases and their variations. A use case is a business scenario fed to a workflow (§3a): its run
context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new
ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial
patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it
derives variations: one business input changed, with the direction the output should move. Synthetic estates go
in
alz-discovery-sample, which is the one server that may be extended for eval data. - Align. Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03
intake:
- Statically: the producer's
output.jsonagainst the consumer'sinput.json, and the fields each consumer's prompt reads. - Live: a voyage, where a real upstream package runs through the downstream intake.
- Across the chain: the Epic's acceptance criteria checked over a whole voyage, not one run.
- Traceability: each ADO acceptance criterion has at least one check, and each check names its criterion.
- Statically: the producer's
Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that make up one ADO capability are linked into a chain (§3c).
3. Repository: use cases and experiments, apart from the agents
The new gitea repository holds Shakedown's code, the definitions of new use cases and experiments, and their conclusions. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each experiment's record links to its runs by execution id.
cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/ # FastAPI API + UI
├── bosun/ # the deep agent
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│ ├── subagents/ # one module per sub-agent: prompt + tool list
│ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger
├── use-cases/<workflow>/<use-case-id>/ # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/
│ ├── run-context.yaml # the scenario, in the shape its workflow takes; no run_id, no credentials
│ ├── uploads/ # CMDB exports, rate cards …
│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run
│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a)
│ └── README.md # the business situation in a paragraph, and its upstream use case
├── workflows/<workflow>.yaml # registry: one file per workflow under test (§3c)
├── capabilities/<capability>/ # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440)
│ ├── capability.yaml # stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings
│ ├── links.yaml # hand-off contracts between stages
│ ├── expectations.yaml # chain-level checks from the Epic's acceptance criteria
│ └── voyages/<engagement-id>.yaml # one engagement's chain of use cases, and how ids flow between them
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6)
├── experiments/EXP-NNN-<slug>/
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
│ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
├── variants/<agent>/<variant>/ # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag
└── deploy/ # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
- New use cases start here, from F03. F03's
azure-crucible-kab-dev-strategy(written 2026-11-09, not yet committed to cp1-agents) is the first use case in this repository, and every later one is created here. The committed F01 cases (7) and F02 cases (2), and each agent'seval/cases.yaml, stay incp1-agents. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown. All experiments go into this repository. - No change to cp1-agents. Shakedown imports the scripts it reuses (
kab_run_workflow.py'sKab,load_contextandnew_ids;evaluate_workflow_run.py;collect_run_usage.py; the package agents'scorecard.py) from that pinned clone by path, instead of extracting a shared package from cp1-agents. - Reproducible results: every ledger run records the commit of cp1-agents and of Shakedown that it ran against.
- Changes reach the repositories as pull requests. Bosun opens a gitea pull request for a new case, an experiment record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.
3a. Use cases, voyages and variations
A use case is a business scenario, not a test script. Today's folders show its parts, spread unevenly across Features:
| Feature | Scenario | How it reaches the run | Expectations |
|---|---|---|---|
| F01 | context.yaml (customer, plan, scope, sources) and CMDB uploads for live cases; brief.txt for the sample cases |
Uploaded through kab_run_workflow.py --context, or --file |
case.yaml checks in 5 of 7; azure-live and gcp-live have none |
| F02 | rate-card.json; everything else is typed as --input flags (execution id, approval reference, artifact id) |
Run inputs, including rate_card_document; no context file, unlike F01 and F03 |
Written as prose in the README only |
| F03 | run-context.yaml (objectives, constraints, priorities, planning assumptions) |
The run input run_context_document, as text (F03 ADR 0002) |
None |
Shakedown standardises the folder, not the workflows:
run-context.yamlstays in its workflow's own format. A small adapter for each workflow knows how to deliver it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their context three ways is reported as an alignment finding (§2, Align) for you to decide on.expectations.yamlgathers what each Feature now keeps in different places (checks, value bars,limits:), and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned into checks by Chartmaker and reviewed by you.
A voyage chains one engagement's use cases across workflows. F01's azure-crucible-kab-dev-cmdb, F02's
azure-crucible-kab-dev-assessment and F03's azure-crucible-kab-dev-strategy all describe ENG-AZURE-KAB-DEV-002,
but the links are execution ids copied by hand (F02's README asks for discovery_execution_id and the package
artifact id; F03 has assessment_execution_id: exec_374d5d577077). A voyage file
(capabilities/<capability>/voyages/<engagement>.yaml) names the use case for each stage and which upstream output fills each
downstream input, for example assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run: latest_signed_off}. When a run starts, Shakedown resolves the reference into the uploaded copy under generated:, as
kab_run_workflow.py already does for run_id. The use case file is never edited. A voyage is also the live
alignment test: the downstream intake has to accept what the upstream run really produced.
A variation changes one business input and states how the output must respond. This is the most direct test of
business value: an output that doesn't move when the business situation changes is not reasoning about the business.
Each entry in variations.yaml has a change (a path in run-context.yaml and its new value) and one or more
expect relations:
| Relation | Meaning | Judged by |
|---|---|---|
direction |
A named output metric goes up or down against the base run | Code, from the metric's path |
bound |
A limit from the input is respected (no plan above budget_ceiling) |
Code |
unchanged |
Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) | Code |
rationale |
The output's reasoning cites the changed input | Claude Sonnet, with a quote |
Seed variations for F03's azure-crucible-kab-dev-strategy, drafted from its run context:
| Change | Expected |
|---|---|
priorities.cost 0.3 → 0.7 (risk 0.4 → 0.1) |
Total cost of the recommended strategy goes down; the rationale cites cost priority |
constraints.budget_ceiling.amount 40000 → 20000 |
No recommended plan exceeds 20,000 USD, or the output says none can, and why |
constraints.risk_appetite medium → low |
Refactor share goes down; risk mitigations go up |
planning_assumptions.team_capacity 2 → 1 |
The timeline gets longer; the disposition set is unchanged |
constraints.exit_date 2027-06-30 → 2027-03-31 |
Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't |
constraints.data_residency drop eastus2 |
No target in eastus2 |
Variations stay cheap. Only the stage whose input changed is run again. The upstream ids stay pinned to the base
run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation
passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget
for each batch, and start_replay_batch asks for your approval.
3b. The existing use cases
Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place
(§3), and F03's moves to this repository. Paths are relative to cp1-agents/individual-agents/.
ALZ-F01 Discovery: ALZ-F01: Discovery Workflow Agent/workflows/cases/
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|---|---|---|---|---|
azure-crucible-kab-dev-cmdb |
Live Azure, resource group crucible-kab-dev, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection |
ENG-AZURE-KAB-DEV-002 |
48 checks, value: bar, limits: |
First stage of the ENG-AZURE-KAB-DEV-002 voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner) |
azure-crucible-kab-dev |
The same group, no CMDB or ITSM: every gap declared, a complete package | ENG-AZURE-KAB-DEV-001 |
15 checks, value: bar, limits: |
The base its -cmdb sibling is a variation of: unchanged relations for the estate facts, direction for drift findings |
meridian-baseline |
Meridian Retail Group sample estate (provider: sample): six scope entries, one unscannable HSM, a planted prompt injection |
ENG-MERIDIAN-001 |
45 checks | First stage of the Meridian voyage; rubric scores (no value: bar yet) |
meridian-without-itsm |
Meridian with ITSM turned off on the run form | ENG-MERIDIAN-001 |
6 checks | Recast as a variation of meridian-baseline (include_itsm yes → no) with unchanged and direction relations |
snapshot-missing-sources |
A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source | any | 8 checks | Adversarial base for "declare the gap, don't invent it" |
azure-live |
Whole subscription sub-kyn-RITM0250380-653-dev, about 644 resources |
ENG-AZURE-LIVE-001 |
None | Expectations drafted by Chartmaker (scale, cost and coverage limits: first) |
gcp-live |
Whole live GCP project | ENG-GCP-LIVE-001 |
None | Expectations drafted by Chartmaker; the same expectations as azure-live where the providers should agree (hyperscaler servers stay separate, the workflow is the same) |
ALZ-F02 Assessment & Disposition: ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|---|---|---|---|---|
azure-crucible-kab-dev-assessment |
The Discovery package of an azure-crucible-kab-dev-cmdb run, and an illustrative Azure rate card |
ENG-AZURE-KAB-DEV-002 |
Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run | Prose turned into expectations.yaml (reviewed by you); a run-context.yaml gathering its --input flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent) |
meridian-assessment |
A Meridian Discovery run (RUN-008, critic pass) and the sample rate card |
ENG-MERIDIAN-001 |
Six expectations as prose in cases/README.md (no cloud, risk, partial score, dependents, sign-off, evidence); checks still todo in F02 PLAN step 10 |
Expectations written and frozen before its first scored run; second stage of the Meridian voyage |
ALZ-F03 Strategy Synthesis: ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|---|---|---|---|---|
azure-crucible-kab-dev-strategy |
Strategy for the KAB Platform on the signed-off Assessment exec_374d5d577077: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in eastus and eastus2, priorities and planning assumptions |
ENG-AZURE-KAB-DEV-002 |
None | Moves to cp1-shakedown/use-cases/alz-f03-strategy/; expectations; third stage of the voyage; the six seed variations (§3a) |
Each agent's eval/cases.yaml (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent
alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and
reports agents with no eval case.
Voyages to define first (in capabilities/cp1-agentic-landing-zone/voyages/):
ENG-AZURE-KAB-DEV-002: F01azure-crucible-kab-dev-cmdb→ F02azure-crucible-kab-dev-assessment→ F03azure-crucible-kab-dev-strategy. This is the only engagement with all three stages today.ENG-MERIDIAN-001: F01meridian-baseline→ F02meridian-assessment. There is no F03 stage yet; Chartmaker drafts one.
Gaps across the set (each one a first job for Chartmaker, reviewed by you):
- Expectations are missing for
azure-live,gcp-liveand F03, and are only prose for both F02 use cases. - F02 has no context file. Its scenario is spread across
--inputflags in a README, which goes against the convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on. Until then, Shakedown's F02 adapter turnsrun-context.yamlinto those inputs. - No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02 variants vary sources and data.
- Only
ENG-AZURE-KAB-DEV-002has a CMDB upload, and no use case has ITSM data from a file.
3c. Many workflows, linked into capabilities
Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a capability: an ADO Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three stages of one capability, the Epic 3214440 "CP1 – Agentic Landing Zone". Its own description sets the chain ("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them.
The workflow registry: workflows/<workflow>.yaml, one file for each workflow under test. Onboarding a workflow
means adding this file and its rubric, with no code. Each file holds:
- its key (
alz-f02-assessment), its ADO Feature (3214442), its workstream tag and its pattern (individual-agents,sub-agentsoroutcome-pattern); - its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so their history is kept and each ledger run records the one it used;
- its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as
evaluate_workflow_run.pyalready does; - its context adapter: an upload through
--context(F01),--inputflags (F02), or the run inputrun_context_document(F03); - its human gates (for example
disposition_signoff), what it produces (its package and the package's schema), and what it consumes.
A workflow needn't belong to a capability. The solution architecture agent (outcome-pattern/) can be registered and
rated on its own, and is never linked to Discovery.
The capability: capabilities/cp1-agentic-landing-zone/capability.yaml. Shakedown builds it from ADO, read-only:
the Epic, its Features, their Stories and acceptance criteria, each with its rev. The stage order, gates and
"mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves
each stage. When an item's rev changes, Shakedown flags the expectations written against the older wording.
| # | Stage | ADO Feature | Workflow binding | Human gate | State on 2026-11-09 |
|---|---|---|---|---|---|
| 1 | Discovery | 3214441 | alz-f01-discovery (individual). The sub-agent build is a second binding, never mixed with the first |
none | Deployed; live runs |
| 2 | Assessment | 3214442 | alz-f02-assessment |
disposition_signoff |
Deployed; runs 014 and 015 signed off |
| 3 | Strategy | 3214443 | alz-f03-strategy |
strategy_decision |
Built (6 agents, export of 9 steps); not deployed or imported |
| 4 | Business Case | 3214444 | none yet | assumption validation | Not built |
| 5 | Target Architecture | 3214445 | none yet | architect blueprint | Not built |
| 6 | Wave Planning | 3214446 | none yet | wave plan approval | Not built; tagged Review - Overlap |
| 7 | Landing Zone | 3214447 | none yet | lz_foundation_approval, tf_high_blast_approval |
Not built; the only stage that mutates |
The links: capabilities/cp1-agentic-landing-zone/links.yaml. Each link is a hand-off contract between two
stages. It names:
- the producer's step and artifact, and the schema of what passes;
- the consumer's intake step;
- the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids;
- the upstream gate that must be cleared first;
- the ADRs that define the link.
| Link | What passes | Consumer's check | Inputs it fills | Defined by |
|---|---|---|---|---|
| F01 → F02 | The Discovery package: a JSON package with a manifest and the value scorecard | F02 intake s0: one package, a sha256 for each part, complete, critic pass, the named run and engagement |
discovery_execution_id, discovery_approval_ref, artifact_ids |
F01 ADR 0020; F02 ADR 0008 |
| F02 → F03 | The Assessment package | F03 intake | assessment_execution_id, assessment_approval_ref, the package's artifact id |
F02 ADRs 0014, 0015; F03 ADR 0002 |
| F03 → F04 onwards | Not defined yet | Drafted when F04 is designed |
Shakedown checks each link four ways:
- Statically: the producer's package schema against the consumer's intake schema and the fields its prompts read, at the pinned commit.
- Live: a voyage.
- For drift: when either side's schema changes in cp1-agents, the link is marked stale until the static check passes again on the new commit.
- For use: a field that is produced but never consumed downstream is data passed on with no purpose, which is reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must be declared missing, never left out silently.
Chain-level expectations, from the Epic's acceptance criteria:
capabilities/cp1-agentic-landing-zone/expectations.yaml is checked over a voyage, not over one run.
| Epic AC | Checked over a voyage |
|---|---|
| AC1 End-to-end chain | Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference |
| AC2 Shared agentic rails | Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence |
| AC3 Declared equals executed | Each run's executed steps and tool calls match its workflow export |
| AC4 Critic verdict binds | No stage ends completed after a critic fail or an unparseable verdict |
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from agents.json), or the deliverable says it doesn't |
| AC6 One binding budget | One token ceiling binds each run, checked against usage.json |
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
Rules for linked workflows:
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is rated on its own and is never linked.
- A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can draft its use cases, link and expectations before the workflow is built, starting with F04.
- Ripple variations change an input at one stage and follow the effect down the chain. For example, adding the CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage downstream, so each one needs your approval.
Other capabilities onboard the same way: a capability.yaml built from the Epic, its workflows registered and its
links declared. CP4 Epics are next after the landing zone.
In the UI, the capability view is the home page. It shows the chain, each stage's state and binding, the health of each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the Epic and Feature acceptance criteria that have no check yet.
4. Architecture on the cluster
you (browser) ── Keycloak sign-in (realm kab) ──┐
▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│ shakedown-app (FastAPI + UI, one replica) │
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
│ └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│
│ shadow agents (variants/*, applied only with approval) │
└────────────────────────────────────────────────────────────────────────────────┘
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
- Sign-in and tokens: Shakedown is an OIDC client of KAB's Keycloak (
id.kab-dev.kyndemo.live, realmkab). The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. This replaceskab_token.pyfor everything Shakedown does. - Reading the cluster: a ServiceAccount with read-only RBAC (
get/liston pods andpods/loginagents) replacescollect_run_usage.py's kubeconfig. It gets nosecretsaccess. - The UI is exposed through an ingress from Phase 0, with Keycloak sign-in in front of it. There's no port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
- Shadow agents: these are applied to the
shakedownnamespace, beside Shakedown and apart from the live agents inagents. They reach the gateways through the samedeploy/profile values. Shakedown's ServiceAccount cannot apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' rule that each cluster change is approved. - Model: Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
which doesn't serve it. The key is in Secret
shakedown-secrets, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it. - Tracing: LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to the ledger instead.
- Images and manifests: built with buildah, in parallel, and pushed to
cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/<component>:<version>. Manifests are rendered like cp1-agents', and everyapplyneeds your approval. - The ledger records each run as a folder:
context.yaml(withgenerated:),workflow.json,agents.json(image, version, model, max tokens and temperature for each step's agent),steps.json,usage.json,scores.jsonand the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first data load.
5. The deep agent: "Bosun"
The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that changes state.
Built with LangChain Deep Agents. create_deep_agent provides the planning (to-do) tool, a file workspace,
sub-agents with their own prompts and tools, and interrupt_on, which pauses before a named tool runs. Shakedown uses
them as follows:
| Deep Agents feature | Used for |
|---|---|
| Planning tool | Navigator's experiment plan, shown live in the UI |
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
interrupt_on |
Approval in the UI before start_run, start_replay_batch, open_pull_request and render_variant |
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
| Sub-agent | Job | Tools | Writes |
|---|---|---|---|
| Navigator (planner) | Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule | ledger read, repo read | experiment.yaml draft |
| Lookout (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries |
| Surveyor (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | scores.json |
| Shipwright (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun |
| Chartmaker (use-case author) | Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions | ADO read, repo read, gitea PR | Pull request to Shakedown |
| Purser (cost) | Tokens, time and cost per step; flags steps over limits: and the cheapest variant that keeps the score |
ledger read | Cost section of the scores |
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores.
How a loop runs: a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today.
Guardrails, enforced in code (tool permissions, interrupt_on, RBAC), not only in prompts:
- A HITL review (
continue/stop/rerun) is a person's decision. Bosun has no tool that answers one. - Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
- No changes to ADO work items. Bosun's ADO tools are read-only.
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as data in every sub-agent's context, and a planted-injection case runs against Bosun itself.
- Checks are frozen before the run. A case's hash is recorded when a run starts, and a check edited afterwards can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README).
- The patterns stay apart: a finding on
individual-agents/never proposes a change tosub-agents/or tooutcome-pattern/. Solution architecture and Discovery are never linked. - A model change for an agent under test is proposed, never made.
6. Business value: rating quality, not volume
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric
(rubrics/<workflow>.yaml), starting from the six dimensions below. Each score carries a short quote from the output
and a reason.
| Dimension | Question | Fails when |
|---|---|---|
| Decision-ready | Can the reader make the decision this step exists for? | It lists data and leaves the conclusion to the reader |
| Traceable | Does every claim and figure trace to a fact and an evidence_ref? |
A figure appears that no fact supports (checked in code first) |
| Owned and actionable | Does each finding name an owner and a next action? | "Investigate" with no owner (extends the F01 scorecard's with_owner_share and with_action_share) |
| Explained | Are the subjects described in business terms (which application it is, who uses it, why it matters)? | Ids without descriptions: the APP-001 report gap |
| Consumable downstream | Does the next workflow get every field it needs, filled or declared missing? | Fields filled with placeholders (extends f02_fields.filled_or_declared) |
| Signal over volume | Is the output proportionate: no restated inputs, no duplicates, no padding? | Raw records passed through; the same finding written three times |
The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a
case can fail on value as well as on checks, through its value: bars.
Calibrating the judge: before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015). Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its model version or the rubric changes.
7. Delivery phases
Each phase ends with a check that proves it. The phase's status is kept in cp1-shakedown/.agents/PLAN.md once the
repository exists.
| # | Phase | Delivers | Verified by |
|---|---|---|---|
| 0 | Foundations | Repository scaffold in cp1-shakedown; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; shakedown namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's azure-crucible-kab-dev-strategy brought in as the first use case (agreed with whoever is running F03); first deploy |
Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | Ledger and rating | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The azure-crucible-kab-dev-assessment checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 2 | Bosun v1, read-only | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | The acceptance test in §1 passes on runs 010 to 015 |
| 3 | Experiments | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; experiments/ records by pull request |
5 replays of F02 s5 on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
| 4 | Capability, links, value and variations | cp1-agentic-landing-zone capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with direction, bound, unchanged and rationale relations |
The ENG-AZURE-KAB-DEV-002 voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
| 5 | Use-case factory | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown before F04's first live run; F04 is onboarded with no new Shakedown code |
| 6 | Scale out | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its capability.yaml and links.yaml |
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back.
8. Seed experiments (first experiments/ entries)
These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it.
- EXP-001 P2 counting unit (improvement 1): stability on
s5with the current and the proposed wording; the metric is the spread of APP-001's confidence. - EXP-002 P1 to P3 in code (improvement 3): a variant hook; the metric is the spread (target 0) and the change in the other checks.
- EXP-003 "Compute rate" definition (improvement 2): replay with the clarified 7R rule; the metric is a consistent Rehost precondition across runs.
- EXP-004 Security posture cost (improvement 5):
LITELLM_MAX_TOKENStiers, andgpt-4.1againstgpt-5.6-sol; the metrics are tokens and the value score. A model change remains your call. - EXP-005 Report figures (improvement 4): turns the hand check into a permanent invariant.
- EXP-006 F03 business sensitivity: the six seed variations of
azure-crucible-kab-dev-strategy(§3a); the metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment. - Temperature, as context for all of them: every agent runs at
LITELLM_TEMPERATURE: "1", probably because the gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming it can be tuned away.
9. Decisions
Decided (2026-11-09):
| Decision | Choice |
|---|---|
| Deep agent framework | LangChain Deep Agents |
| Bosun's model and the judge | Claude Sonnet 5.5 for both |
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
| Repository | agents/cp1-shakedown on gitea, for use cases and experiments, apart from the agents and workflows |
| Use cases | Scenario folders like F03's azure-crucible-kab-dev-strategy. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path |
| Many workflows and linking | Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07) |
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
| Hosting | The crucible cluster |
| Shadow agents | The shakedown namespace |
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
Still open:
- Keycloak client and hostname (platform ask, blocks Phase 0): the KAB platform team registers a
cp1-shakedownOIDC client in realmkab, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS certificate underkyndemo.live. - Gitea bot account: a gitea token that Bosun uses to open pull requests on
cp1-shakedownandcp1-agents, held in a Secret, with rights limited to opening pull requests. - Variant agent keys (to verify with KAB): whether a variant in
shakedownmay reuse the agent'sMCP_AGENT_API_KEYSecret (copied by a person) or needs its own KAB registration. - Rubric ownership: who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the rubric represents.
10. Out of scope
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
- Writing to ADO work items.
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
- Moving the existing cases out of cp1-agents.