# Shakedown: implementation plan **Shakedown** is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs, runs experiments on them, and builds use cases for them. On a ship, the *shakedown cruise* proves the vessel before it enters service. This app does the same for every workflow before its outputs reach an Engagement Lead. | | | |---|---| | Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) | | User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | | Runs on | The `crucible-kab-dev-aks` cluster | | Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource | | Judge | **Claude Sonnet 5.5**, a different model family from the gpt-5.x agents under test | | First workflow under test | ALZ-F02 Assessment & Disposition (`wf_7b6cae08-…`, individual pattern) | | Then | ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded *before* its first live run | --- ## 1. Why: the work Shakedown replaces The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004 to 015, commit `66c3274`). Almost every step of that work was done by hand: | Manual activity in the session | Evidence | What Shakedown does instead | |---|---|---| | Refreshing the 5-minute KAB token | about 40 "kab token refreshed" prompts | You sign in once through KAB's Keycloak; the server-side session renews the token itself | | Starting a run, polling it, pulling `steps.json` | `kab_run_workflow.py start/status/outputs` | A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage | | Keeping rollback copies of the workflow before each run | 11 `workflow-rollback-wf_*.json` files in `reports/cp1` | A snapshot of the workflow and the agents is taken automatically for every run | | Checking report figures against the facts | PLAN.md improvement 4: "checked by hand" in runs 014 and 015 | Every report figure is checked in code against the facts the run produced | | Spotting confidence drift between runs | APP-001 scored 0.50 or 0.60 on the same package (improvement 1) | Stability experiments: the same input N times, with the variance shown for each field | | Spotting ambiguous rules | "a compute rate" read two ways (improvement 2) | The drift report links each divergent field to the prompt rule the agent cited | | Finding the costliest agent | security posture agent at 520k to 540k tokens (improvement 5) | Cost and latency per step and per run, against the case's `limits:` | | Judging whether an output is *useful* | "it doesn't break down what the application is within the report" | A business-value rubric scored on every output, not only a schema check | | Writing checks for a case | Meridian checks still `todo` (F02 PLAN step 10) | The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run | **The acceptance test for the whole product:** given F02 runs 010 to 015, Shakedown finds the APP-001 confidence drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report, without anyone pointing it at them. ## 2. What Shakedown does Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5): 1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, severities), run by a generic case runner that works for any workflow, not only F01. - *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across steps, every `evidence_ref` names a real execution, and token and time `limits:` are met. - *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§6). 2. **Experiment.** Change one variable and measure the effect against a baseline: - *Stability:* the same input N times (default 5), with the spread for each output field. - *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on `http://.agents.svc.cluster.local:8080`). This needs no full 12-step workflow run, and no upstream agent tokens are spent. - *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with the same replay inputs sent to the baseline and to the variant. 3. **Build use cases and their variations.** A use case is a business scenario fed to a workflow (§3a): its run context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go in `alz-discovery-sample`, which is the one server that may be extended for eval data. 4. **Align.** Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03 intake: - *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's prompt reads. - *Live:* a voyage, where a real upstream package runs through the downstream intake. - *Across the chain:* the Epic's acceptance criteria checked over a whole voyage, not one run. - *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion. Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that make up one ADO capability are linked into a chain (§3c). ## 3. Repository: use cases and experiments, apart from the agents The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their **conclusions**. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each experiment's record links to its runs by execution id. ``` cp1-shakedown/ ├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/) ├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005) ├── skills//SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010) ├── bosun/ # the deep agent │ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend │ ├── subagents/ # one module per sub-agent: prompt + tool list │ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR ├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger ├── use-cases/// # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/ │ ├── run-context.yaml # the scenario, in the shape its workflow takes; no run_id, no credentials │ ├── uploads/ # CMDB exports, rate cards … │ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run │ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a) │ └── README.md # the business situation in a paragraph, and its upstream use case ├── workflows/.yaml # registry: one file per workflow under test (§3c) ├── capabilities// # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440) │ ├── capability.yaml # stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings │ ├── links.yaml # hand-off contracts between stages │ ├── expectations.yaml # chain-level checks from the Epic's acceptance criteria │ └── voyages/.yaml # one engagement's chain of use cases, and how ids flow between them ├── rubrics/.yaml # business-value rubric for each workflow (§6) ├── experiments/EXP-NNN-/ │ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule │ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person ├── variants/// # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag └── deploy/ # Shakedown's own Kubernetes manifests, with as in cp1-agents ``` - **New use cases start here, from F03.** F03's `azure-crucible-kab-dev-strategy` (written 2026-11-09, not yet committed to cp1-agents) is the first use case in this repository, and every later one is created here. The committed F01 cases (7) and F02 cases (2), and each agent's `eval/cases.yaml`, stay in `cp1-agents`. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown. **All** experiments go into this repository. - **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`, `load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents' `scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents. - **Reproducible results:** every ledger run records the commit of cp1-agents and of Shakedown that it ran against. - **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges. ## 3a. Use cases, voyages and variations **A use case is a business scenario, not a test script.** Today's folders show its parts, spread unevenly across Features: | Feature | Scenario | How it reaches the run | Expectations | |---|---|---|---| | F01 | `context.yaml` (customer, plan, scope, sources) and CMDB uploads for live cases; `brief.txt` for the sample cases | Uploaded through `kab_run_workflow.py --context`, or `--file` | `case.yaml` checks in 5 of 7; `azure-live` and `gcp-live` have none | | F02 | `rate-card.json`; everything else is typed as `--input` flags (execution id, approval reference, artifact id) | Run inputs, including `rate_card_document`; no context file, unlike F01 and F03 | Written as prose in the README only | | F03 | `run-context.yaml` (objectives, constraints, priorities, planning assumptions) | The run input `run_context_document`, as text (F03 ADR 0002) | None | Shakedown standardises the folder, not the workflows: - **`run-context.yaml` stays in its workflow's own format.** A small adapter for each workflow knows how to deliver it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their context three ways is reported as an alignment finding (§2, Align) for you to decide on. - **`expectations.yaml`** gathers what each Feature now keeps in different places (checks, value bars, `limits:`), and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned into checks by Chartmaker and reviewed by you. **A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's `azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`, but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). A voyage file (`capabilities//voyages/.yaml`) names the use case for each stage and which upstream output fills each downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run: latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as `kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live alignment test: the downstream intake has to accept what the upstream run really produced. **A variation changes one business input and states how the output must respond.** This is the most direct test of business value: an output that doesn't move when the business situation changes is not reasoning about the business. Each entry in `variations.yaml` has a `change` (a path in `run-context.yaml` and its new value) and one or more `expect` relations: | Relation | Meaning | Judged by | |---|---|---| | `direction` | A named output metric goes up or down against the base run | Code, from the metric's path | | `bound` | A limit from the input is respected (no plan above `budget_ceiling`) | Code | | `unchanged` | Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) | Code | | `rationale` | The output's reasoning cites the changed input | Claude Sonnet, with a quote | Seed variations for F03's `azure-crucible-kab-dev-strategy`, drafted from its run context: | Change | Expected | |---|---| | `priorities.cost` 0.3 → 0.7 (risk 0.4 → 0.1) | Total cost of the recommended strategy goes down; the rationale cites cost priority | | `constraints.budget_ceiling.amount` 40000 → 20000 | No recommended plan exceeds 20,000 USD, or the output says none can, and why | | `constraints.risk_appetite` medium → low | Refactor share goes down; risk mitigations go up | | `planning_assumptions.team_capacity` 2 → 1 | The timeline gets longer; the disposition set is unchanged | | `constraints.exit_date` 2027-06-30 → 2027-03-31 | Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't | | `constraints.data_residency` drop `eastus2` | No target in `eastus2` | **Variations stay cheap.** Only the stage whose input changed is run again. The upstream ids stay pinned to the base run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget for each batch, and `start_replay_batch` asks for your approval. ## 3b. The existing use cases Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place (§3), and F03's moves to this repository. Paths are relative to `cp1-agents/individual-agents/`. **ALZ-F01 Discovery:** `ALZ-F01: Discovery Workflow Agent/workflows/cases/` | Use case | Scenario | Engagement | Expectations today | What Shakedown adds | |---|---|---|---|---| | `azure-crucible-kab-dev-cmdb` | Live Azure, resource group `crucible-kab-dev`, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection | `ENG-AZURE-KAB-DEV-002` | 48 checks, `value:` bar, `limits:` | First stage of the `ENG-AZURE-KAB-DEV-002` voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner) | | `azure-crucible-kab-dev` | The same group, no CMDB or ITSM: every gap declared, a complete package | `ENG-AZURE-KAB-DEV-001` | 15 checks, `value:` bar, `limits:` | The base its `-cmdb` sibling is a variation of: `unchanged` relations for the estate facts, `direction` for drift findings | | `meridian-baseline` | Meridian Retail Group sample estate (`provider: sample`): six scope entries, one unscannable HSM, a planted prompt injection | `ENG-MERIDIAN-001` | 45 checks | First stage of the Meridian voyage; rubric scores (no `value:` bar yet) | | `meridian-without-itsm` | Meridian with ITSM turned off on the run form | `ENG-MERIDIAN-001` | 6 checks | Recast as a variation of `meridian-baseline` (`include_itsm` yes → no) with `unchanged` and `direction` relations | | `snapshot-missing-sources` | A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source | any | 8 checks | Adversarial base for "declare the gap, don't invent it" | | `azure-live` | Whole subscription `sub-kyn-RITM0250380-653-dev`, about 644 resources | `ENG-AZURE-LIVE-001` | None | Expectations drafted by Chartmaker (scale, cost and coverage `limits:` first) | | `gcp-live` | Whole live GCP project | `ENG-GCP-LIVE-001` | None | Expectations drafted by Chartmaker; the same expectations as `azure-live` where the providers should agree (hyperscaler servers stay separate, the workflow is the same) | **ALZ-F02 Assessment & Disposition:** `ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/` | Use case | Scenario | Engagement | Expectations today | What Shakedown adds | |---|---|---|---|---| | `azure-crucible-kab-dev-assessment` | The Discovery package of an `azure-crucible-kab-dev-cmdb` run, and an illustrative Azure rate card | `ENG-AZURE-KAB-DEV-002` | Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run | Prose turned into `expectations.yaml` (reviewed by you); a `run-context.yaml` gathering its `--input` flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent) | | `meridian-assessment` | A Meridian Discovery run (`RUN-008`, critic `pass`) and the sample rate card | `ENG-MERIDIAN-001` | Six expectations as prose in `cases/README.md` (no cloud, risk, partial score, dependents, sign-off, evidence); checks still `todo` in F02 PLAN step 10 | Expectations written and frozen before its first scored run; second stage of the Meridian voyage | **ALZ-F03 Strategy Synthesis:** `ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/` | Use case | Scenario | Engagement | Expectations today | What Shakedown adds | |---|---|---|---|---| | `azure-crucible-kab-dev-strategy` | Strategy for the KAB Platform on the signed-off Assessment `exec_374d5d577077`: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in `eastus` and `eastus2`, priorities and planning assumptions | `ENG-AZURE-KAB-DEV-002` | None | Moves to `cp1-shakedown/use-cases/alz-f03-strategy/`; expectations; third stage of the voyage; the six seed variations (§3a) | **Each agent's `eval/cases.yaml`** (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and reports agents with no eval case. **Voyages to define first** (in `capabilities/cp1-agentic-landing-zone/voyages/`): - `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03 `azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today. - `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker drafts one. **Gaps across the set** (each one a first job for Chartmaker, reviewed by you): - Expectations are missing for `azure-live`, `gcp-live` and F03, and are only prose for both F02 use cases. - F02 has no context file. Its scenario is spread across `--input` flags in a README, which goes against the convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on. Until then, Shakedown's F02 adapter turns `run-context.yaml` into those inputs. - No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02 variants vary sources and data. - Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file. ## 3c. Many workflows, linked into capabilities Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a **capability**: an ADO Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three stages of one capability, the Epic **3214440 "CP1 – Agentic Landing Zone"**. Its own description sets the chain ("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them. **The workflow registry: `workflows/.yaml`, one file for each workflow under test.** Onboarding a workflow means adding this file and its rubric, with no code. Each file holds: - its key (`alz-f02-assessment`), its ADO Feature (3214442), its workstream tag and its pattern (`individual-agents`, `sub-agents` or `outcome-pattern`); - its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so their history is kept and each ledger run records the one it used; - its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as `evaluate_workflow_run.py` already does; - its context adapter: an upload through `--context` (F01), `--input` flags (F02), or the run input `run_context_document` (F03); - its human gates (for example `disposition_signoff`), what it produces (its package and the package's schema), and what it consumes. A workflow needn't belong to a capability. The solution architecture agent (`outcome-pattern/`) can be registered and rated on its own, and is never linked to Discovery. **The capability: `capabilities/cp1-agentic-landing-zone/capability.yaml`.** Shakedown builds it from ADO, read-only: the Epic, its Features, their Stories and acceptance criteria, each with its `rev`. The stage order, gates and "mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves each stage. When an item's `rev` changes, Shakedown flags the expectations written against the older wording. | # | Stage | ADO Feature | Workflow binding | Human gate | State on 2026-11-09 | |---|---|---|---|---|---| | 1 | Discovery | 3214441 | `alz-f01-discovery` (individual). The sub-agent build is a second binding, never mixed with the first | none | Deployed; live runs | | 2 | Assessment | 3214442 | `alz-f02-assessment` | `disposition_signoff` | Deployed; runs 014 and 015 signed off | | 3 | Strategy | 3214443 | `alz-f03-strategy` | `strategy_decision` | Built (6 agents, export of 9 steps); not deployed or imported | | 4 | Business Case | 3214444 | none yet | assumption validation | Not built | | 5 | Target Architecture | 3214445 | none yet | architect blueprint | Not built | | 6 | Wave Planning | 3214446 | none yet | wave plan approval | Not built; tagged `Review - Overlap` | | 7 | Landing Zone | 3214447 | none yet | `lz_foundation_approval`, `tf_high_blast_approval` | Not built; the only stage that mutates | **The links: `capabilities/cp1-agentic-landing-zone/links.yaml`.** Each link is a hand-off contract between two stages. It names: - the producer's step and artifact, and the schema of what passes; - the consumer's intake step; - the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids; - the upstream gate that must be cleared first; - the ADRs that define the link. | Link | What passes | Consumer's check | Inputs it fills | Defined by | |---|---|---|---|---| | F01 → F02 | The Discovery package: a JSON package with a manifest and the value scorecard | F02 intake `s0`: one package, a sha256 for each part, complete, critic `pass`, the named run and engagement | `discovery_execution_id`, `discovery_approval_ref`, `artifact_ids` | F01 ADR 0020; F02 ADR 0008 | | F02 → F03 | The Assessment package | F03 intake | `assessment_execution_id`, `assessment_approval_ref`, the package's artifact id | F02 ADRs 0014, 0015; F03 ADR 0002 | | F03 → F04 onwards | Not defined yet | | | Drafted when F04 is designed | Shakedown checks each link four ways: - **Statically:** the producer's package schema against the consumer's intake schema and the fields its prompts read, at the pinned commit. - **Live:** a voyage. - **For drift:** when either side's schema changes in cp1-agents, the link is marked stale until the static check passes again on the new commit. - **For use:** a field that is produced but never consumed downstream is data passed on with no purpose, which is reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must be declared missing, never left out silently. **Chain-level expectations, from the Epic's acceptance criteria:** `capabilities/cp1-agentic-landing-zone/expectations.yaml` is checked over a voyage, not over one run. | Epic AC | Checked over a voyage | |---|---| | AC1 End-to-end chain | Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference | | AC2 Shared agentic rails | Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence | | AC3 Declared equals executed | Each run's executed steps and tool calls match its workflow export | | AC4 Critic verdict binds | No stage ends `completed` after a critic `fail` or an unparseable verdict | | AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't | | AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` | | AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record | | AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output | **Rules for linked workflows:** - A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is rated on its own and is never linked. - A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can draft its use cases, link and expectations before the workflow is built, starting with F04. - **Ripple variations** change an input at one stage and follow the effect down the chain. For example, adding the CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage downstream, so each one needs your approval. **Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its links declared. CP4 Epics are next after the landing zone. **In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the Epic and Feature acceptance criteria that have no check yet. ## 4. Architecture on the cluster ``` you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐ bearer token (you) ▼ ┌──────────────────── namespace: shakedown ──────────────────────────────────────┐ │ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │ │ ├─ acts as you: the portal's token, validated per request; never stored │ │ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │ │ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │ │ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │ │ └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│ │ shadow agents (variants/*, applied only with approval) │ └────────────────────────────────────────────────────────────────────────────────┘ │ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents` ``` - **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown does. - **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`) replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access. - **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views. - **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents in `agents`. They reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' rule that each cluster change is approved. - **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it. - **Tracing:** LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to the ledger instead. - **Images and manifests:** built with buildah, in parallel, and pushed to `cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/:`. Manifests are rendered like cp1-agents', and every `apply` needs your approval. - **The ledger** records each run as a folder: `context.yaml` (with `generated:`), `workflow.json`, `agents.json` (image, version, model, max tokens and temperature for each step's agent), `steps.json`, `usage.json`, `scores.json` and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first data load. ## 5. The deep agent: "Bosun" The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that changes state. **Built with LangChain Deep Agents.** `create_deep_agent` provides the planning (to-do) tool, a file workspace, sub-agents with their own prompts and tools, and `interrupt_on`, which pauses before a named tool runs. Shakedown uses them as follows: | Deep Agents feature | Used for | |---|---| | Planning tool | Navigator's experiment plan, shown live in the portal | | File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied | | Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small | | `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) | | LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards | | Sub-agent | Job | Tools | Writes | |---|---|---|---| | **Navigator** (planner) | Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule | ledger read, repo read | `experiment.yaml` draft | | **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries | | **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` | | **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun | | **Chartmaker** (use-case author) | Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions | ADO read, repo read, gitea PR | Pull request to Shakedown | | **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores | Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores. **Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice, quality controls and integration specifics are applied the same way every time: | Family | Skills (first set) | Used by | |---|---|---| | Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright | | Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker | | Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright | Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a skill change goes through a pull request like an agent change. **How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today. **Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:** - A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. - Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. - No changes to ADO work items. Bosun's ADO tools are read-only. - No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read. - Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as data in every sub-agent's context, and a planted-injection case runs against Bosun itself. - **Checks are frozen before the run.** A case's hash is recorded when a run starts, and a check edited afterwards can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README). - The patterns stay apart: a finding on `individual-agents/` never proposes a change to `sub-agents/` or to `outcome-pattern/`. Solution architecture and Discovery are never linked. - A model change for an agent under test is proposed, never made. ## 5a. The nightly cycle (amended 2026-11-12) The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull request against the agents and workflows automatically, for review the next morning. | Step | Who | Result | |---|---|---| | Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) | | Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it | | Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget | | Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence | | Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive | | Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs | | Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated | | Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more | Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`, `start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its pull requests. ## 6. Business value: rating quality, not volume Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric (`rubrics/.yaml`), starting from the six dimensions below. Each score carries a short quote from the output and a reason. | Dimension | Question | Fails when | |---|---|---| | **Decision-ready** | Can the reader make the decision this step exists for? | It lists data and leaves the conclusion to the reader | | **Traceable** | Does every claim and figure trace to a fact and an `evidence_ref`? | A figure appears that no fact supports (checked in code first) | | **Owned and actionable** | Does each finding name an owner and a next action? | "Investigate" with no owner (extends the F01 scorecard's `with_owner_share` and `with_action_share`) | | **Explained** | Are the subjects described in business terms (which application it is, who uses it, why it matters)? | Ids without descriptions: the APP-001 report gap | | **Consumable downstream** | Does the next workflow get every field it needs, filled or declared missing? | Fields filled with placeholders (extends `f02_fields.filled_or_declared`) | | **Signal over volume** | Is the output proportionate: no restated inputs, no duplicates, no padding? | Raw records passed through; the same finding written three times | The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a case can fail on value as well as on checks, through its `value:` bars. **Calibrating the judge:** before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015). Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its model version or the rubric changes. ## 7. Delivery phases Each phase ends with a check that proves it. The phase's status is kept in `cp1-shakedown/.agents/PLAN.md` once the repository exists. | # | Phase | Delivers | Verified by | |---|---|---|---| | 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | | 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | | 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | | 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | | 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence | | 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code | | 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` | | 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person | Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back. ## 8. Seed experiments (first `experiments/` entries) These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it. 1. **EXP-001 P2 counting unit** (improvement 1): stability on `s5` with the current and the proposed wording; the metric is the spread of APP-001's confidence. 2. **EXP-002 P1 to P3 in code** (improvement 3): a variant hook; the metric is the spread (target 0) and the change in the other checks. 3. **EXP-003 "Compute rate" definition** (improvement 2): replay with the clarified 7R rule; the metric is a consistent Rehost precondition across runs. 4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against `gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call. 5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant. 6. **EXP-006 F03 business sensitivity:** the six seed variations of `azure-crucible-kab-dev-strategy` (§3a); the metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment. 7. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming it can be tuned away. ## 9. Decisions **Decided (2026-11-09):** | Decision | Choice | |---|---| | Deep agent framework | LangChain Deep Agents | | Bosun's model and the judge | Claude Sonnet 5.5 for both | | Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API | | Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows | | Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path | | Many workflows and linking | Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with `cp1-agentic-landing-zone` (Epic 3214440, F01 to F07) | | Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale | | Hosting | The crucible cluster | | Shadow agents | The `shakedown` namespace | | UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API | | Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) | | Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) | | Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it | **Still open:** 1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates under `kyndemo.live` for the API (and the portal once it is shared). 5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run unattended, and where the skills library lives (here, or shared with cp1-agents). 2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`, held in a Secret, with rights limited to opening pull requests. 3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's `MCP_AGENT_API_KEY` Secret (copied by a person) or needs its own KAB registration. 4. **Rubric ownership:** who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the rubric represents. ## 10. Out of scope - Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests. - Writing to ADO work items. - Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else. - Moving the existing cases out of cp1-agents.