Files
cp1-shakedown/docs/implementation-plan.md
Jonathan Boniface 98e3ec69b4 Plan: many workflows, linked into capabilities
Workflow registry by config; capabilities built read-only from ADO Epics,
starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07); links
as hand-off contracts checked statically, live, for drift and for use;
chain-level checks from the Epic's AC1 to AC8; ripple variations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 15:34:41 +00:00

43 KiB
Raw Blame History

Shakedown: implementation plan

Shakedown is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs, runs experiments on them, and builds use cases for them. On a ship, the shakedown cruise proves the vessel before it enters service. This app does the same for every workflow before its outputs reach an Engagement Lead.

Date 2026-11-09 (revised the same day with the product owner's decisions, §9)
Status Draft. Nothing is built yet
Repository agents/cp1-shakedown on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in cp1-agents
Runs on The crucible-kab-dev-aks cluster
Deep agent LangChain Deep Agents (deepagents, on LangGraph), driven by Claude Sonnet 5.5 on the Kyndryl Azure resource
Judge Claude Sonnet 5.5, a different model family from the gpt-5.x agents under test
First workflow under test ALZ-F02 Assessment & Disposition (wf_7b6cae08-…, individual pattern)
Then ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded before its first live run

1. Why: the work Shakedown replaces

The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004 to 015, commit 66c3274). Almost every step of that work was done by hand:

Manual activity in the session Evidence What Shakedown does instead
Refreshing the 5-minute KAB token about 40 "kab token refreshed" prompts You sign in once through KAB's Keycloak; the server-side session renews the token itself
Starting a run, polling it, pulling steps.json kab_run_workflow.py start/status/outputs A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage
Keeping rollback copies of the workflow before each run 11 workflow-rollback-wf_*.json files in reports/cp1 A snapshot of the workflow and the agents is taken automatically for every run
Checking report figures against the facts PLAN.md improvement 4: "checked by hand" in runs 014 and 015 Every report figure is checked in code against the facts the run produced
Spotting confidence drift between runs APP-001 scored 0.50 or 0.60 on the same package (improvement 1) Stability experiments: the same input N times, with the variance shown for each field
Spotting ambiguous rules "a compute rate" read two ways (improvement 2) The drift report links each divergent field to the prompt rule the agent cited
Finding the costliest agent security posture agent at 520k to 540k tokens (improvement 5) Cost and latency per step and per run, against the case's limits:
Judging whether an output is useful "it doesn't break down what the application is within the report" A business-value rubric scored on every output, not only a schema check
Writing checks for a case Meridian checks still todo (F02 PLAN step 10) The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run

The acceptance test for the whole product: given F02 runs 010 to 015, Shakedown finds the APP-001 confidence drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report, without anyone pointing it at them.

2. What Shakedown does

Shakedown does four things, each usable through the UI and by the deep agent (§5):

  1. Rate. Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
    • Checks: the existing case format (case.yaml checks: with contains, item.match, absent, equals, severities), run by a generic case runner that works for any workflow, not only F01.
    • Invariants in code: the run's figures trace to its facts, the package digests recompute, ids resolve across steps, every evidence_ref names a real execution, and token and time limits: are met.
    • Business value: a rubric scored by Claude Sonnet with quoted evidence (§6).
  2. Experiment. Change one variable and measure the effect against a baseline:
    • Stability: the same input N times (default 5), with the spread for each output field.
    • Step replay: re-send one step's recorded input straight to an agent over A2A (message/send on http://<agent>.agents.svc.cluster.local:8080). This needs no full 12-step workflow run, and no upstream agent tokens are spent.
    • Variants: prompt, model, LITELLM_MAX_TOKENS or a code hook changed in a shadow deployment of the agent, with the same replay inputs sent to the baseline and to the variant.
  3. Build use cases and their variations. A use case is a business scenario fed to a workflow (§3a): its run context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it derives variations: one business input changed, with the direction the output should move. Synthetic estates go in alz-discovery-sample, which is the one server that may be extended for eval data.
  4. Align. Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03 intake:
    • Statically: the producer's output.json against the consumer's input.json, and the fields each consumer's prompt reads.
    • Live: a voyage, where a real upstream package runs through the downstream intake.
    • Across the chain: the Epic's acceptance criteria checked over a whole voyage, not one run.
    • Traceability: each ADO acceptance criterion has at least one check, and each check names its criterion.

Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that make up one ADO capability are linked into a chain (§3c).

3. Repository: use cases and experiments, apart from the agents

The new gitea repository holds Shakedown's code, the definitions of new use cases and experiments, and their conclusions. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each experiment's record links to its runs by execution id.

cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/                     # FastAPI API + UI
├── bosun/                   # the deep agent
│   ├── agent.py             #   create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│   ├── subagents/           #   one module per sub-agent: prompt + tool list
│   └── tools/               #   KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
├── shakedown_lib/           # case runner, invariants, rubric scoring, KAB client, ledger
├── use-cases/<workflow>/<use-case-id>/      # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/
│   ├── run-context.yaml     #   the scenario, in the shape its workflow takes; no run_id, no credentials
│   ├── uploads/             #   CMDB exports, rate cards …
│   ├── expectations.yaml    #   checks, value bars, limits; frozen (hashed) before the first run
│   ├── variations.yaml      #   derived scenarios, each with its expected direction of change (§3a)
│   └── README.md            #   the business situation in a paragraph, and its upstream use case
├── workflows/<workflow>.yaml       # registry: one file per workflow under test (§3c)
├── capabilities/<capability>/     # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440)
│   ├── capability.yaml            #   stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings
│   ├── links.yaml                 #   hand-off contracts between stages
│   ├── expectations.yaml          #   chain-level checks from the Epic's acceptance criteria
│   └── voyages/<engagement-id>.yaml   # one engagement's chain of use cases, and how ids flow between them
├── rubrics/<workflow>.yaml  # business-value rubric for each workflow (§6)
├── experiments/EXP-NNN-<slug>/
│   ├── experiment.yaml      #   hypothesis, one variable, baseline, variants, N, metric, stop rule
│   └── results.md           #   conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
├── variants/<agent>/<variant>/   # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag
└── deploy/                  # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
  • New use cases start here, from F03. F03's azure-crucible-kab-dev-strategy (written 2026-11-09, not yet committed to cp1-agents) is the first use case in this repository, and every later one is created here. The committed F01 cases (7) and F02 cases (2), and each agent's eval/cases.yaml, stay in cp1-agents. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown. All experiments go into this repository.
  • No change to cp1-agents. Shakedown imports the scripts it reuses (kab_run_workflow.py's Kab, load_context and new_ids; evaluate_workflow_run.py; collect_run_usage.py; the package agents' scorecard.py) from that pinned clone by path, instead of extracting a shared package from cp1-agents.
  • Reproducible results: every ledger run records the commit of cp1-agents and of Shakedown that it ran against.
  • Changes reach the repositories as pull requests. Bosun opens a gitea pull request for a new case, an experiment record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.

3a. Use cases, voyages and variations

A use case is a business scenario, not a test script. Today's folders show its parts, spread unevenly across Features:

Feature Scenario How it reaches the run Expectations
F01 context.yaml (customer, plan, scope, sources) and CMDB uploads for live cases; brief.txt for the sample cases Uploaded through kab_run_workflow.py --context, or --file case.yaml checks in 5 of 7; azure-live and gcp-live have none
F02 rate-card.json; everything else is typed as --input flags (execution id, approval reference, artifact id) Run inputs, including rate_card_document; no context file, unlike F01 and F03 Written as prose in the README only
F03 run-context.yaml (objectives, constraints, priorities, planning assumptions) The run input run_context_document, as text (F03 ADR 0002) None

Shakedown standardises the folder, not the workflows:

  • run-context.yaml stays in its workflow's own format. A small adapter for each workflow knows how to deliver it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their context three ways is reported as an alignment finding (§2, Align) for you to decide on.
  • expectations.yaml gathers what each Feature now keeps in different places (checks, value bars, limits:), and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned into checks by Chartmaker and reviewed by you.

A voyage chains one engagement's use cases across workflows. F01's azure-crucible-kab-dev-cmdb, F02's azure-crucible-kab-dev-assessment and F03's azure-crucible-kab-dev-strategy all describe ENG-AZURE-KAB-DEV-002, but the links are execution ids copied by hand (F02's README asks for discovery_execution_id and the package artifact id; F03 has assessment_execution_id: exec_374d5d577077). A voyage file (capabilities/<capability>/voyages/<engagement>.yaml) names the use case for each stage and which upstream output fills each downstream input, for example assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run: latest_signed_off}. When a run starts, Shakedown resolves the reference into the uploaded copy under generated:, as kab_run_workflow.py already does for run_id. The use case file is never edited. A voyage is also the live alignment test: the downstream intake has to accept what the upstream run really produced.

A variation changes one business input and states how the output must respond. This is the most direct test of business value: an output that doesn't move when the business situation changes is not reasoning about the business. Each entry in variations.yaml has a change (a path in run-context.yaml and its new value) and one or more expect relations:

Relation Meaning Judged by
direction A named output metric goes up or down against the base run Code, from the metric's path
bound A limit from the input is respected (no plan above budget_ceiling) Code
unchanged Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) Code
rationale The output's reasoning cites the changed input Claude Sonnet, with a quote

Seed variations for F03's azure-crucible-kab-dev-strategy, drafted from its run context:

Change Expected
priorities.cost 0.3 → 0.7 (risk 0.4 → 0.1) Total cost of the recommended strategy goes down; the rationale cites cost priority
constraints.budget_ceiling.amount 40000 → 20000 No recommended plan exceeds 20,000 USD, or the output says none can, and why
constraints.risk_appetite medium → low Refactor share goes down; risk mitigations go up
planning_assumptions.team_capacity 2 → 1 The timeline gets longer; the disposition set is unchanged
constraints.exit_date 2027-06-30 → 2027-03-31 Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't
constraints.data_residency drop eastus2 No target in eastus2

Variations stay cheap. Only the stage whose input changed is run again. The upstream ids stay pinned to the base run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget for each batch, and start_replay_batch asks for your approval.

3b. The existing use cases

Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place (§3), and F03's moves to this repository. Paths are relative to cp1-agents/individual-agents/.

ALZ-F01 Discovery: ALZ-F01: Discovery Workflow Agent/workflows/cases/

Use case Scenario Engagement Expectations today What Shakedown adds
azure-crucible-kab-dev-cmdb Live Azure, resource group crucible-kab-dev, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection ENG-AZURE-KAB-DEV-002 48 checks, value: bar, limits: First stage of the ENG-AZURE-KAB-DEV-002 voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner)
azure-crucible-kab-dev The same group, no CMDB or ITSM: every gap declared, a complete package ENG-AZURE-KAB-DEV-001 15 checks, value: bar, limits: The base its -cmdb sibling is a variation of: unchanged relations for the estate facts, direction for drift findings
meridian-baseline Meridian Retail Group sample estate (provider: sample): six scope entries, one unscannable HSM, a planted prompt injection ENG-MERIDIAN-001 45 checks First stage of the Meridian voyage; rubric scores (no value: bar yet)
meridian-without-itsm Meridian with ITSM turned off on the run form ENG-MERIDIAN-001 6 checks Recast as a variation of meridian-baseline (include_itsm yes → no) with unchanged and direction relations
snapshot-missing-sources A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source any 8 checks Adversarial base for "declare the gap, don't invent it"
azure-live Whole subscription sub-kyn-RITM0250380-653-dev, about 644 resources ENG-AZURE-LIVE-001 None Expectations drafted by Chartmaker (scale, cost and coverage limits: first)
gcp-live Whole live GCP project ENG-GCP-LIVE-001 None Expectations drafted by Chartmaker; the same expectations as azure-live where the providers should agree (hyperscaler servers stay separate, the workflow is the same)

ALZ-F02 Assessment & Disposition: ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/

Use case Scenario Engagement Expectations today What Shakedown adds
azure-crucible-kab-dev-assessment The Discovery package of an azure-crucible-kab-dev-cmdb run, and an illustrative Azure rate card ENG-AZURE-KAB-DEV-002 Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run Prose turned into expectations.yaml (reviewed by you); a run-context.yaml gathering its --input flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent)
meridian-assessment A Meridian Discovery run (RUN-008, critic pass) and the sample rate card ENG-MERIDIAN-001 Six expectations as prose in cases/README.md (no cloud, risk, partial score, dependents, sign-off, evidence); checks still todo in F02 PLAN step 10 Expectations written and frozen before its first scored run; second stage of the Meridian voyage

ALZ-F03 Strategy Synthesis: ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/

Use case Scenario Engagement Expectations today What Shakedown adds
azure-crucible-kab-dev-strategy Strategy for the KAB Platform on the signed-off Assessment exec_374d5d577077: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in eastus and eastus2, priorities and planning assumptions ENG-AZURE-KAB-DEV-002 None Moves to cp1-shakedown/use-cases/alz-f03-strategy/; expectations; third stage of the voyage; the six seed variations (§3a)

Each agent's eval/cases.yaml (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and reports agents with no eval case.

Voyages to define first (in capabilities/cp1-agentic-landing-zone/voyages/):

  • ENG-AZURE-KAB-DEV-002: F01 azure-crucible-kab-dev-cmdb → F02 azure-crucible-kab-dev-assessment → F03 azure-crucible-kab-dev-strategy. This is the only engagement with all three stages today.
  • ENG-MERIDIAN-001: F01 meridian-baseline → F02 meridian-assessment. There is no F03 stage yet; Chartmaker drafts one.

Gaps across the set (each one a first job for Chartmaker, reviewed by you):

  • Expectations are missing for azure-live, gcp-live and F03, and are only prose for both F02 use cases.
  • F02 has no context file. Its scenario is spread across --input flags in a README, which goes against the convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on. Until then, Shakedown's F02 adapter turns run-context.yaml into those inputs.
  • No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02 variants vary sources and data.
  • Only ENG-AZURE-KAB-DEV-002 has a CMDB upload, and no use case has ITSM data from a file.

3c. Many workflows, linked into capabilities

Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a capability: an ADO Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three stages of one capability, the Epic 3214440 "CP1 – Agentic Landing Zone". Its own description sets the chain ("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them.

The workflow registry: workflows/<workflow>.yaml, one file for each workflow under test. Onboarding a workflow means adding this file and its rubric, with no code. Each file holds:

  • its key (alz-f02-assessment), its ADO Feature (3214442), its workstream tag and its pattern (individual-agents, sub-agents or outcome-pattern);
  • its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so their history is kept and each ledger run records the one it used;
  • its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as evaluate_workflow_run.py already does;
  • its context adapter: an upload through --context (F01), --input flags (F02), or the run input run_context_document (F03);
  • its human gates (for example disposition_signoff), what it produces (its package and the package's schema), and what it consumes.

A workflow needn't belong to a capability. The solution architecture agent (outcome-pattern/) can be registered and rated on its own, and is never linked to Discovery.

The capability: capabilities/cp1-agentic-landing-zone/capability.yaml. Shakedown builds it from ADO, read-only: the Epic, its Features, their Stories and acceptance criteria, each with its rev. The stage order, gates and "mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves each stage. When an item's rev changes, Shakedown flags the expectations written against the older wording.

# Stage ADO Feature Workflow binding Human gate State on 2026-11-09
1 Discovery 3214441 alz-f01-discovery (individual). The sub-agent build is a second binding, never mixed with the first none Deployed; live runs
2 Assessment 3214442 alz-f02-assessment disposition_signoff Deployed; runs 014 and 015 signed off
3 Strategy 3214443 alz-f03-strategy strategy_decision Built (6 agents, export of 9 steps); not deployed or imported
4 Business Case 3214444 none yet assumption validation Not built
5 Target Architecture 3214445 none yet architect blueprint Not built
6 Wave Planning 3214446 none yet wave plan approval Not built; tagged Review - Overlap
7 Landing Zone 3214447 none yet lz_foundation_approval, tf_high_blast_approval Not built; the only stage that mutates

The links: capabilities/cp1-agentic-landing-zone/links.yaml. Each link is a hand-off contract between two stages. It names:

  • the producer's step and artifact, and the schema of what passes;
  • the consumer's intake step;
  • the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids;
  • the upstream gate that must be cleared first;
  • the ADRs that define the link.
Link What passes Consumer's check Inputs it fills Defined by
F01 → F02 The Discovery package: a JSON package with a manifest and the value scorecard F02 intake s0: one package, a sha256 for each part, complete, critic pass, the named run and engagement discovery_execution_id, discovery_approval_ref, artifact_ids F01 ADR 0020; F02 ADR 0008
F02 → F03 The Assessment package F03 intake assessment_execution_id, assessment_approval_ref, the package's artifact id F02 ADRs 0014, 0015; F03 ADR 0002
F03 → F04 onwards Not defined yet Drafted when F04 is designed

Shakedown checks each link four ways:

  • Statically: the producer's package schema against the consumer's intake schema and the fields its prompts read, at the pinned commit.
  • Live: a voyage.
  • For drift: when either side's schema changes in cp1-agents, the link is marked stale until the static check passes again on the new commit.
  • For use: a field that is produced but never consumed downstream is data passed on with no purpose, which is reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must be declared missing, never left out silently.

Chain-level expectations, from the Epic's acceptance criteria: capabilities/cp1-agentic-landing-zone/expectations.yaml is checked over a voyage, not over one run.

Epic AC Checked over a voyage
AC1 End-to-end chain Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference
AC2 Shared agentic rails Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence
AC3 Declared equals executed Each run's executed steps and tool calls match its workflow export
AC4 Critic verdict binds No stage ends completed after a critic fail or an unparseable verdict
AC5 Independent critic Each stage's critic runs on a different model from its executors (from agents.json), or the deliverable says it doesn't
AC6 One binding budget One token ceiling binds each run, checked against usage.json
AC7 Truthful deliverables The provider, critic verdict, change count and run status each report states match the run record
AC8 Human decisions stay human Every gate decision in the ledger comes from a person's review action, never from an agent's output

Rules for linked workflows:

  • A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is rated on its own and is never linked.
  • A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can draft its use cases, link and expectations before the workflow is built, starting with F04.
  • Ripple variations change an input at one stage and follow the effect down the chain. For example, adding the CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage downstream, so each one needs your approval.

Other capabilities onboard the same way: a capability.yaml built from the Epic, its workflows registered and its links declared. CP4 Epics are next after the landing zone.

In the UI, the capability view is the home page. It shows the chain, each stage's state and binding, the health of each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the Epic and Feature acceptance criteria that have no check yet.

4. Architecture on the cluster

                       you (browser) ── Keycloak sign-in (realm kab) ──┐
                                                                       ▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│  shakedown-app  (FastAPI + UI, one replica)                                    │
│    ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│    ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger    │
│    │     └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret)     │
│    ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API     │
│    └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│
│  shadow agents  (variants/*, applied only with approval)                       │
└────────────────────────────────────────────────────────────────────────────────┘
      │ HTTPS: api.kab-dev…    │ A2A :8080: agents.svc   │ read-only RBAC: pods/log in `agents`
  • Sign-in and tokens: Shakedown is an OIDC client of KAB's Keycloak (id.kab-dev.kyndemo.live, realm kab). The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. This replaces kab_token.py for everything Shakedown does.
  • Reading the cluster: a ServiceAccount with read-only RBAC (get/list on pods and pods/log in agents) replaces collect_run_usage.py's kubeconfig. It gets no secrets access.
  • The UI is exposed through an ingress from Phase 0, with Keycloak sign-in in front of it. There's no port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
  • Shadow agents: these are applied to the shakedown namespace, beside Shakedown and apart from the live agents in agents. They reach the gateways through the same deploy/ profile values. Shakedown's ServiceAccount cannot apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' rule that each cluster change is approved.
  • Model: Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, which doesn't serve it. The key is in Secret shakedown-secrets, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
  • Tracing: LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to the ledger instead.
  • Images and manifests: built with buildah, in parallel, and pushed to cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/<component>:<version>. Manifests are rendered like cp1-agents', and every apply needs your approval.
  • The ledger records each run as a folder: context.yaml (with generated:), workflow.json, agents.json (image, version, model, max tokens and temperature for each step's agent), steps.json, usage.json, scores.json and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first data load.

5. The deep agent: "Bosun"

The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that changes state.

Built with LangChain Deep Agents. create_deep_agent provides the planning (to-do) tool, a file workspace, sub-agents with their own prompts and tools, and interrupt_on, which pauses before a named tool runs. Shakedown uses them as follows:

Deep Agents feature Used for
Planning tool Navigator's experiment plan, shown live in the UI
File workspace (backend) A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied
Sub-agents The six roles below, each with only the tools its job needs, which keeps the main agent's context small
interrupt_on Approval in the UI before start_run, start_replay_batch, open_pull_request and render_variant
LangGraph checkpointer A long experiment survives a pod restart, and its whole trail can be audited afterwards
Sub-agent Job Tools Writes
Navigator (planner) Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule ledger read, repo read experiment.yaml draft
Lookout (run operator) Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you KAB, replay, cluster read Ledger entries
Surveyor (judge) Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring case runner, rubric, ledger read scores.json
Shipwright (fixer) Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes repo read, gitea PR Pull request to cp1-agents, never merged by Bosun
Chartmaker (use-case author) Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions ADO read, repo read, gitea PR Pull request to Shakedown
Purser (cost) Tokens, time and cost per step; flags steps over limits: and the cheapest variant that keeps the score ledger read Cost section of the scores

Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores.

How a loop runs: a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today.

Guardrails, enforced in code (tool permissions, interrupt_on, RBAC), not only in prompts:

  • A HITL review (continue/stop/rerun) is a person's decision. Bosun has no tool that answers one.
  • Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
  • No changes to ADO work items. Bosun's ADO tools are read-only.
  • No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
  • Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as data in every sub-agent's context, and a planted-injection case runs against Bosun itself.
  • Checks are frozen before the run. A case's hash is recorded when a run starts, and a check edited afterwards can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README).
  • The patterns stay apart: a finding on individual-agents/ never proposes a change to sub-agents/ or to outcome-pattern/. Solution architecture and Discovery are never linked.
  • A model change for an agent under test is proposed, never made.

6. Business value: rating quality, not volume

Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric (rubrics/<workflow>.yaml), starting from the six dimensions below. Each score carries a short quote from the output and a reason.

Dimension Question Fails when
Decision-ready Can the reader make the decision this step exists for? It lists data and leaves the conclusion to the reader
Traceable Does every claim and figure trace to a fact and an evidence_ref? A figure appears that no fact supports (checked in code first)
Owned and actionable Does each finding name an owner and a next action? "Investigate" with no owner (extends the F01 scorecard's with_owner_share and with_action_share)
Explained Are the subjects described in business terms (which application it is, who uses it, why it matters)? Ids without descriptions: the APP-001 report gap
Consumable downstream Does the next workflow get every field it needs, filled or declared missing? Fields filled with placeholders (extends f02_fields.filled_or_declared)
Signal over volume Is the output proportionate: no restated inputs, no duplicates, no padding? Raw records passed through; the same finding written three times

The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a case can fail on value as well as on checks, through its value: bars.

Calibrating the judge: before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015). Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its model version or the rubric changes.

7. Delivery phases

Each phase ends with a check that proves it. The phase's status is kept in cp1-shakedown/.agents/PLAN.md once the repository exists.

# Phase Delivers Verified by
0 Foundations Repository scaffold in cp1-shakedown; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; shakedown namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's azure-crucible-kab-dev-strategy brought in as the first use case (agreed with whoever is running F03); first deploy Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets
1 Ledger and rating Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI The azure-crucible-kab-dev-assessment checks score runs 014 and 015; a report figure changed in a copy of a run is caught
2 Bosun v1, read-only Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" The acceptance test in §1 passes on runs 010 to 015
3 Experiments Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; experiments/ records by pull request 5 replays of F02 s5 on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread
4 Capability, links, value and variations cp1-agentic-landing-zone capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with direction, bound, unchanged and rationale relations The ENG-AZURE-KAB-DEV-002 voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence
5 Use-case factory Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding F04's use case, expectations and variations are merged in Shakedown before F04's first live run; F04 is onboarded with no new Shakedown code
6 Scale out F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its capability.yaml and links.yaml

Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back.

8. Seed experiments (first experiments/ entries)

These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it.

  1. EXP-001 P2 counting unit (improvement 1): stability on s5 with the current and the proposed wording; the metric is the spread of APP-001's confidence.
  2. EXP-002 P1 to P3 in code (improvement 3): a variant hook; the metric is the spread (target 0) and the change in the other checks.
  3. EXP-003 "Compute rate" definition (improvement 2): replay with the clarified 7R rule; the metric is a consistent Rehost precondition across runs.
  4. EXP-004 Security posture cost (improvement 5): LITELLM_MAX_TOKENS tiers, and gpt-4.1 against gpt-5.6-sol; the metrics are tokens and the value score. A model change remains your call.
  5. EXP-005 Report figures (improvement 4): turns the hand check into a permanent invariant.
  6. EXP-006 F03 business sensitivity: the six seed variations of azure-crucible-kab-dev-strategy (§3a); the metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment.
  7. Temperature, as context for all of them: every agent runs at LITELLM_TEMPERATURE: "1", probably because the gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming it can be tuned away.

9. Decisions

Decided (2026-11-09):

Decision Choice
Deep agent framework LangChain Deep Agents
Bosun's model and the judge Claude Sonnet 5.5 for both
Model route The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API
Repository agents/cp1-shakedown on gitea, for use cases and experiments, apart from the agents and workflows
Use cases Scenario folders like F03's azure-crucible-kab-dev-strategy. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path
Many workflows and linking Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07)
Variations A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale
Hosting The crucible cluster
Shadow agents The shakedown namespace
UI access Ingress with Keycloak sign-in from Phase 0 (no port-forward stage)

Still open:

  1. Keycloak client and hostname (platform ask, blocks Phase 0): the KAB platform team registers a cp1-shakedown OIDC client in realm kab, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS certificate under kyndemo.live.
  2. Gitea bot account: a gitea token that Bosun uses to open pull requests on cp1-shakedown and cp1-agents, held in a Secret, with rights limited to opening pull requests.
  3. Variant agent keys (to verify with KAB): whether a variant in shakedown may reuse the agent's MCP_AGENT_API_KEY Secret (copied by a person) or needs its own KAB registration.
  4. Rubric ownership: who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the rubric represents.

10. Out of scope

  • Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
  • Writing to ADO work items.
  • Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
  • Moving the existing cases out of cp1-agents.