Workflow registry by config; capabilities built read-only from ADO Epics, starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07); links as hand-off contracts checked statically, live, for drift and for use; chain-level checks from the Epic's AC1 to AC8; ripple variations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
505 lines
43 KiB
Markdown
505 lines
43 KiB
Markdown
# Shakedown: implementation plan
|
||
|
||
**Shakedown** is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs,
|
||
runs experiments on them, and builds use cases for them. On a ship, the *shakedown cruise* proves the vessel before it
|
||
enters service. This app does the same for every workflow before its outputs reach an Engagement Lead.
|
||
|
||
| | |
|
||
|---|---|
|
||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
|
||
| Status | Draft. Nothing is built yet |
|
||
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
|
||
| Runs on | The `crucible-kab-dev-aks` cluster |
|
||
| Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource |
|
||
| Judge | **Claude Sonnet 5.5**, a different model family from the gpt-5.x agents under test |
|
||
| First workflow under test | ALZ-F02 Assessment & Disposition (`wf_7b6cae08-…`, individual pattern) |
|
||
| Then | ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded *before* its first live run |
|
||
|
||
---
|
||
|
||
## 1. Why: the work Shakedown replaces
|
||
|
||
The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004
|
||
to 015, commit `66c3274`). Almost every step of that work was done by hand:
|
||
|
||
| Manual activity in the session | Evidence | What Shakedown does instead |
|
||
|---|---|---|
|
||
| Refreshing the 5-minute KAB token | about 40 "kab token refreshed" prompts | You sign in once through KAB's Keycloak; the server-side session renews the token itself |
|
||
| Starting a run, polling it, pulling `steps.json` | `kab_run_workflow.py start/status/outputs` | A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage |
|
||
| Keeping rollback copies of the workflow before each run | 11 `workflow-rollback-wf_*.json` files in `reports/cp1` | A snapshot of the workflow and the agents is taken automatically for every run |
|
||
| Checking report figures against the facts | PLAN.md improvement 4: "checked by hand" in runs 014 and 015 | Every report figure is checked in code against the facts the run produced |
|
||
| Spotting confidence drift between runs | APP-001 scored 0.50 or 0.60 on the same package (improvement 1) | Stability experiments: the same input N times, with the variance shown for each field |
|
||
| Spotting ambiguous rules | "a compute rate" read two ways (improvement 2) | The drift report links each divergent field to the prompt rule the agent cited |
|
||
| Finding the costliest agent | security posture agent at 520k to 540k tokens (improvement 5) | Cost and latency per step and per run, against the case's `limits:` |
|
||
| Judging whether an output is *useful* | "it doesn't break down what the application is within the report" | A business-value rubric scored on every output, not only a schema check |
|
||
| Writing checks for a case | Meridian checks still `todo` (F02 PLAN step 10) | The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run |
|
||
|
||
**The acceptance test for the whole product:** given F02 runs 010 to 015, Shakedown finds the APP-001 confidence
|
||
drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report,
|
||
without anyone pointing it at them.
|
||
|
||
## 2. What Shakedown does
|
||
|
||
Shakedown does four things, each usable through the UI and by the deep agent (§5):
|
||
|
||
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
|
||
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
|
||
severities), run by a generic case runner that works for any workflow, not only F01.
|
||
- *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across
|
||
steps, every `evidence_ref` names a real execution, and token and time `limits:` are met.
|
||
- *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§6).
|
||
2. **Experiment.** Change one variable and measure the effect against a baseline:
|
||
- *Stability:* the same input N times (default 5), with the spread for each output field.
|
||
- *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on
|
||
`http://<agent>.agents.svc.cluster.local:8080`). This needs no full 12-step workflow run, and no upstream agent
|
||
tokens are spent.
|
||
- *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with
|
||
the same replay inputs sent to the baseline and to the variant.
|
||
3. **Build use cases and their variations.** A use case is a business scenario fed to a workflow (§3a): its run
|
||
context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new
|
||
ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial
|
||
patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it
|
||
derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go
|
||
in `alz-discovery-sample`, which is the one server that may be extended for eval data.
|
||
4. **Align.** Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03
|
||
intake:
|
||
- *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's
|
||
prompt reads.
|
||
- *Live:* a voyage, where a real upstream package runs through the downstream intake.
|
||
- *Across the chain:* the Epic's acceptance criteria checked over a whole voyage, not one run.
|
||
- *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion.
|
||
|
||
Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that
|
||
make up one ADO capability are linked into a chain (§3c).
|
||
|
||
## 3. Repository: use cases and experiments, apart from the agents
|
||
|
||
The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their
|
||
**conclusions**. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each
|
||
experiment's record links to its runs by execution id.
|
||
|
||
```
|
||
cp1-shakedown/
|
||
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
|
||
├── app/ # FastAPI API + UI
|
||
├── bosun/ # the deep agent
|
||
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
|
||
│ ├── subagents/ # one module per sub-agent: prompt + tool list
|
||
│ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
|
||
├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger
|
||
├── use-cases/<workflow>/<use-case-id>/ # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/
|
||
│ ├── run-context.yaml # the scenario, in the shape its workflow takes; no run_id, no credentials
|
||
│ ├── uploads/ # CMDB exports, rate cards …
|
||
│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run
|
||
│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a)
|
||
│ └── README.md # the business situation in a paragraph, and its upstream use case
|
||
├── workflows/<workflow>.yaml # registry: one file per workflow under test (§3c)
|
||
├── capabilities/<capability>/ # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440)
|
||
│ ├── capability.yaml # stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings
|
||
│ ├── links.yaml # hand-off contracts between stages
|
||
│ ├── expectations.yaml # chain-level checks from the Epic's acceptance criteria
|
||
│ └── voyages/<engagement-id>.yaml # one engagement's chain of use cases, and how ids flow between them
|
||
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6)
|
||
├── experiments/EXP-NNN-<slug>/
|
||
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
|
||
│ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
|
||
├── variants/<agent>/<variant>/ # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag
|
||
└── deploy/ # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
|
||
```
|
||
|
||
- **New use cases start here, from F03.** F03's `azure-crucible-kab-dev-strategy` (written 2026-11-09, not yet
|
||
committed to cp1-agents) is the first use case in this repository, and every later one is created here. The
|
||
committed F01 cases (7) and F02 cases (2), and each agent's `eval/cases.yaml`, stay in `cp1-agents`. Shakedown reads
|
||
them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown.
|
||
**All** experiments go into this repository.
|
||
- **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`,
|
||
`load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents'
|
||
`scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents.
|
||
- **Reproducible results:** every ledger run records the commit of cp1-agents and of Shakedown that it ran against.
|
||
- **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment
|
||
record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.
|
||
|
||
## 3a. Use cases, voyages and variations
|
||
|
||
**A use case is a business scenario, not a test script.** Today's folders show its parts, spread unevenly across
|
||
Features:
|
||
|
||
| Feature | Scenario | How it reaches the run | Expectations |
|
||
|---|---|---|---|
|
||
| F01 | `context.yaml` (customer, plan, scope, sources) and CMDB uploads for live cases; `brief.txt` for the sample cases | Uploaded through `kab_run_workflow.py --context`, or `--file` | `case.yaml` checks in 5 of 7; `azure-live` and `gcp-live` have none |
|
||
| F02 | `rate-card.json`; everything else is typed as `--input` flags (execution id, approval reference, artifact id) | Run inputs, including `rate_card_document`; no context file, unlike F01 and F03 | Written as prose in the README only |
|
||
| F03 | `run-context.yaml` (objectives, constraints, priorities, planning assumptions) | The run input `run_context_document`, as text (F03 ADR 0002) | None |
|
||
|
||
Shakedown standardises the folder, not the workflows:
|
||
- **`run-context.yaml` stays in its workflow's own format.** A small adapter for each workflow knows how to deliver
|
||
it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their
|
||
context three ways is reported as an alignment finding (§2, Align) for you to decide on.
|
||
- **`expectations.yaml`** gathers what each Feature now keeps in different places (checks, value bars, `limits:`),
|
||
and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned
|
||
into checks by Chartmaker and reviewed by you.
|
||
|
||
**A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's
|
||
`azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`,
|
||
but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package
|
||
artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). A voyage file
|
||
(`capabilities/<capability>/voyages/<engagement>.yaml`) names the use case for each stage and which upstream output fills each
|
||
downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run:
|
||
latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as
|
||
`kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live
|
||
alignment test: the downstream intake has to accept what the upstream run really produced.
|
||
|
||
**A variation changes one business input and states how the output must respond.** This is the most direct test of
|
||
business value: an output that doesn't move when the business situation changes is not reasoning about the business.
|
||
Each entry in `variations.yaml` has a `change` (a path in `run-context.yaml` and its new value) and one or more
|
||
`expect` relations:
|
||
|
||
| Relation | Meaning | Judged by |
|
||
|---|---|---|
|
||
| `direction` | A named output metric goes up or down against the base run | Code, from the metric's path |
|
||
| `bound` | A limit from the input is respected (no plan above `budget_ceiling`) | Code |
|
||
| `unchanged` | Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) | Code |
|
||
| `rationale` | The output's reasoning cites the changed input | Claude Sonnet, with a quote |
|
||
|
||
Seed variations for F03's `azure-crucible-kab-dev-strategy`, drafted from its run context:
|
||
|
||
| Change | Expected |
|
||
|---|---|
|
||
| `priorities.cost` 0.3 → 0.7 (risk 0.4 → 0.1) | Total cost of the recommended strategy goes down; the rationale cites cost priority |
|
||
| `constraints.budget_ceiling.amount` 40000 → 20000 | No recommended plan exceeds 20,000 USD, or the output says none can, and why |
|
||
| `constraints.risk_appetite` medium → low | Refactor share goes down; risk mitigations go up |
|
||
| `planning_assumptions.team_capacity` 2 → 1 | The timeline gets longer; the disposition set is unchanged |
|
||
| `constraints.exit_date` 2027-06-30 → 2027-03-31 | Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't |
|
||
| `constraints.data_residency` drop `eastus2` | No target in `eastus2` |
|
||
|
||
**Variations stay cheap.** Only the stage whose input changed is run again. The upstream ids stay pinned to the base
|
||
run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation
|
||
passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget
|
||
for each batch, and `start_replay_batch` asks for your approval.
|
||
|
||
## 3b. The existing use cases
|
||
|
||
Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place
|
||
(§3), and F03's moves to this repository. Paths are relative to `cp1-agents/individual-agents/`.
|
||
|
||
**ALZ-F01 Discovery:** `ALZ-F01: Discovery Workflow Agent/workflows/cases/`
|
||
|
||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||
|---|---|---|---|---|
|
||
| `azure-crucible-kab-dev-cmdb` | Live Azure, resource group `crucible-kab-dev`, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection | `ENG-AZURE-KAB-DEV-002` | 48 checks, `value:` bar, `limits:` | First stage of the `ENG-AZURE-KAB-DEV-002` voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner) |
|
||
| `azure-crucible-kab-dev` | The same group, no CMDB or ITSM: every gap declared, a complete package | `ENG-AZURE-KAB-DEV-001` | 15 checks, `value:` bar, `limits:` | The base its `-cmdb` sibling is a variation of: `unchanged` relations for the estate facts, `direction` for drift findings |
|
||
| `meridian-baseline` | Meridian Retail Group sample estate (`provider: sample`): six scope entries, one unscannable HSM, a planted prompt injection | `ENG-MERIDIAN-001` | 45 checks | First stage of the Meridian voyage; rubric scores (no `value:` bar yet) |
|
||
| `meridian-without-itsm` | Meridian with ITSM turned off on the run form | `ENG-MERIDIAN-001` | 6 checks | Recast as a variation of `meridian-baseline` (`include_itsm` yes → no) with `unchanged` and `direction` relations |
|
||
| `snapshot-missing-sources` | A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source | any | 8 checks | Adversarial base for "declare the gap, don't invent it" |
|
||
| `azure-live` | Whole subscription `sub-kyn-RITM0250380-653-dev`, about 644 resources | `ENG-AZURE-LIVE-001` | None | Expectations drafted by Chartmaker (scale, cost and coverage `limits:` first) |
|
||
| `gcp-live` | Whole live GCP project | `ENG-GCP-LIVE-001` | None | Expectations drafted by Chartmaker; the same expectations as `azure-live` where the providers should agree (hyperscaler servers stay separate, the workflow is the same) |
|
||
|
||
**ALZ-F02 Assessment & Disposition:** `ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/`
|
||
|
||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||
|---|---|---|---|---|
|
||
| `azure-crucible-kab-dev-assessment` | The Discovery package of an `azure-crucible-kab-dev-cmdb` run, and an illustrative Azure rate card | `ENG-AZURE-KAB-DEV-002` | Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run | Prose turned into `expectations.yaml` (reviewed by you); a `run-context.yaml` gathering its `--input` flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent) |
|
||
| `meridian-assessment` | A Meridian Discovery run (`RUN-008`, critic `pass`) and the sample rate card | `ENG-MERIDIAN-001` | Six expectations as prose in `cases/README.md` (no cloud, risk, partial score, dependents, sign-off, evidence); checks still `todo` in F02 PLAN step 10 | Expectations written and frozen before its first scored run; second stage of the Meridian voyage |
|
||
|
||
**ALZ-F03 Strategy Synthesis:** `ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/`
|
||
|
||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||
|---|---|---|---|---|
|
||
| `azure-crucible-kab-dev-strategy` | Strategy for the KAB Platform on the signed-off Assessment `exec_374d5d577077`: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in `eastus` and `eastus2`, priorities and planning assumptions | `ENG-AZURE-KAB-DEV-002` | None | Moves to `cp1-shakedown/use-cases/alz-f03-strategy/`; expectations; third stage of the voyage; the six seed variations (§3a) |
|
||
|
||
**Each agent's `eval/cases.yaml`** (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent
|
||
alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and
|
||
reports agents with no eval case.
|
||
|
||
**Voyages to define first** (in `capabilities/cp1-agentic-landing-zone/voyages/`):
|
||
- `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03
|
||
`azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today.
|
||
- `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker
|
||
drafts one.
|
||
|
||
**Gaps across the set** (each one a first job for Chartmaker, reviewed by you):
|
||
- Expectations are missing for `azure-live`, `gcp-live` and F03, and are only prose for both F02 use cases.
|
||
- F02 has no context file. Its scenario is spread across `--input` flags in a README, which goes against the
|
||
convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on.
|
||
Until then, Shakedown's F02 adapter turns `run-context.yaml` into those inputs.
|
||
- No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02
|
||
variants vary sources and data.
|
||
- Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file.
|
||
|
||
## 3c. Many workflows, linked into capabilities
|
||
|
||
Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a **capability**: an ADO
|
||
Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three
|
||
stages of one capability, the Epic **3214440 "CP1 – Agentic Landing Zone"**. Its own description sets the chain
|
||
("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them.
|
||
|
||
**The workflow registry: `workflows/<workflow>.yaml`, one file for each workflow under test.** Onboarding a workflow
|
||
means adding this file and its rubric, with no code. Each file holds:
|
||
- its key (`alz-f02-assessment`), its ADO Feature (3214442), its workstream tag and its pattern (`individual-agents`,
|
||
`sub-agents` or `outcome-pattern`);
|
||
- its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so
|
||
their history is kept and each ledger run records the one it used;
|
||
- its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as
|
||
`evaluate_workflow_run.py` already does;
|
||
- its context adapter: an upload through `--context` (F01), `--input` flags (F02), or the run input
|
||
`run_context_document` (F03);
|
||
- its human gates (for example `disposition_signoff`), what it produces (its package and the package's schema), and
|
||
what it consumes.
|
||
|
||
A workflow needn't belong to a capability. The solution architecture agent (`outcome-pattern/`) can be registered and
|
||
rated on its own, and is never linked to Discovery.
|
||
|
||
**The capability: `capabilities/cp1-agentic-landing-zone/capability.yaml`.** Shakedown builds it from ADO, read-only:
|
||
the Epic, its Features, their Stories and acceptance criteria, each with its `rev`. The stage order, gates and
|
||
"mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves
|
||
each stage. When an item's `rev` changes, Shakedown flags the expectations written against the older wording.
|
||
|
||
| # | Stage | ADO Feature | Workflow binding | Human gate | State on 2026-11-09 |
|
||
|---|---|---|---|---|---|
|
||
| 1 | Discovery | 3214441 | `alz-f01-discovery` (individual). The sub-agent build is a second binding, never mixed with the first | none | Deployed; live runs |
|
||
| 2 | Assessment | 3214442 | `alz-f02-assessment` | `disposition_signoff` | Deployed; runs 014 and 015 signed off |
|
||
| 3 | Strategy | 3214443 | `alz-f03-strategy` | `strategy_decision` | Built (6 agents, export of 9 steps); not deployed or imported |
|
||
| 4 | Business Case | 3214444 | none yet | assumption validation | Not built |
|
||
| 5 | Target Architecture | 3214445 | none yet | architect blueprint | Not built |
|
||
| 6 | Wave Planning | 3214446 | none yet | wave plan approval | Not built; tagged `Review - Overlap` |
|
||
| 7 | Landing Zone | 3214447 | none yet | `lz_foundation_approval`, `tf_high_blast_approval` | Not built; the only stage that mutates |
|
||
|
||
**The links: `capabilities/cp1-agentic-landing-zone/links.yaml`.** Each link is a hand-off contract between two
|
||
stages. It names:
|
||
- the producer's step and artifact, and the schema of what passes;
|
||
- the consumer's intake step;
|
||
- the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids;
|
||
- the upstream gate that must be cleared first;
|
||
- the ADRs that define the link.
|
||
|
||
| Link | What passes | Consumer's check | Inputs it fills | Defined by |
|
||
|---|---|---|---|---|
|
||
| F01 → F02 | The Discovery package: a JSON package with a manifest and the value scorecard | F02 intake `s0`: one package, a sha256 for each part, complete, critic `pass`, the named run and engagement | `discovery_execution_id`, `discovery_approval_ref`, `artifact_ids` | F01 ADR 0020; F02 ADR 0008 |
|
||
| F02 → F03 | The Assessment package | F03 intake | `assessment_execution_id`, `assessment_approval_ref`, the package's artifact id | F02 ADRs 0014, 0015; F03 ADR 0002 |
|
||
| F03 → F04 onwards | Not defined yet | | | Drafted when F04 is designed |
|
||
|
||
Shakedown checks each link four ways:
|
||
- **Statically:** the producer's package schema against the consumer's intake schema and the fields its prompts
|
||
read, at the pinned commit.
|
||
- **Live:** a voyage.
|
||
- **For drift:** when either side's schema changes in cp1-agents, the link is marked stale until the static check
|
||
passes again on the new commit.
|
||
- **For use:** a field that is produced but never consumed downstream is data passed on with no purpose, which is
|
||
reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must
|
||
be declared missing, never left out silently.
|
||
|
||
**Chain-level expectations, from the Epic's acceptance criteria:**
|
||
`capabilities/cp1-agentic-landing-zone/expectations.yaml` is checked over a voyage, not over one run.
|
||
|
||
| Epic AC | Checked over a voyage |
|
||
|---|---|
|
||
| AC1 End-to-end chain | Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference |
|
||
| AC2 Shared agentic rails | Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence |
|
||
| AC3 Declared equals executed | Each run's executed steps and tool calls match its workflow export |
|
||
| AC4 Critic verdict binds | No stage ends `completed` after a critic `fail` or an unparseable verdict |
|
||
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
|
||
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
|
||
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
|
||
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
|
||
|
||
**Rules for linked workflows:**
|
||
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
|
||
rated on its own and is never linked.
|
||
- A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can
|
||
draft its use cases, link and expectations before the workflow is built, starting with F04.
|
||
- **Ripple variations** change an input at one stage and follow the effect down the chain. For example, adding the
|
||
CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage
|
||
downstream, so each one needs your approval.
|
||
|
||
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
|
||
links declared. CP4 Epics are next after the landing zone.
|
||
|
||
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
|
||
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
|
||
Epic and Feature acceptance criteria that have no check yet.
|
||
|
||
## 4. Architecture on the cluster
|
||
|
||
```
|
||
you (browser) ── Keycloak sign-in (realm kab) ──┐
|
||
▼
|
||
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
|
||
│ shakedown-app (FastAPI + UI, one replica) │
|
||
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
|
||
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
|
||
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
|
||
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
|
||
│ └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│
|
||
│ shadow agents (variants/*, applied only with approval) │
|
||
└────────────────────────────────────────────────────────────────────────────────┘
|
||
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
|
||
```
|
||
|
||
- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`).
|
||
The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the
|
||
ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you.
|
||
This replaces `kab_token.py` for everything Shakedown does.
|
||
- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`)
|
||
replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access.
|
||
- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no
|
||
port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
|
||
- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents
|
||
in `agents`. They
|
||
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
|
||
manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents'
|
||
rule that each cluster change is approved.
|
||
- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
|
||
which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path
|
||
from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
|
||
- **Tracing:** LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to
|
||
the ledger instead.
|
||
- **Images and manifests:** built with buildah, in parallel, and pushed to
|
||
`cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/<component>:<version>`. Manifests are rendered like cp1-agents', and
|
||
every `apply` needs your approval.
|
||
- **The ledger** records each run as a folder: `context.yaml` (with `generated:`), `workflow.json`, `agents.json`
|
||
(image, version, model, max tokens and temperature for each step's agent), `steps.json`, `usage.json`,
|
||
`scores.json` and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first
|
||
data load.
|
||
|
||
## 5. The deep agent: "Bosun"
|
||
|
||
The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a
|
||
hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that
|
||
changes state.
|
||
|
||
**Built with LangChain Deep Agents.** `create_deep_agent` provides the planning (to-do) tool, a file workspace,
|
||
sub-agents with their own prompts and tools, and `interrupt_on`, which pauses before a named tool runs. Shakedown uses
|
||
them as follows:
|
||
|
||
| Deep Agents feature | Used for |
|
||
|---|---|
|
||
| Planning tool | Navigator's experiment plan, shown live in the UI |
|
||
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
|
||
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
|
||
| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` |
|
||
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
|
||
|
||
| Sub-agent | Job | Tools | Writes |
|
||
|---|---|---|---|
|
||
| **Navigator** (planner) | Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule | ledger read, repo read | `experiment.yaml` draft |
|
||
| **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries |
|
||
| **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` |
|
||
| **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun |
|
||
| **Chartmaker** (use-case author) | Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions | ADO read, repo read, gitea PR | Pull request to Shakedown |
|
||
| **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores |
|
||
|
||
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
|
||
the judge is visible in its scores.
|
||
|
||
**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and
|
||
Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You
|
||
review, merge, deploy and record the ADR, as today.
|
||
|
||
**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:**
|
||
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one.
|
||
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
|
||
- No changes to ADO work items. Bosun's ADO tools are read-only.
|
||
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
|
||
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as
|
||
data in every sub-agent's context, and a planted-injection case runs against Bosun itself.
|
||
- **Checks are frozen before the run.** A case's hash is recorded when a run starts, and a check edited afterwards
|
||
can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README).
|
||
- The patterns stay apart: a finding on `individual-agents/` never proposes a change to `sub-agents/` or to
|
||
`outcome-pattern/`. Solution architecture and Discovery are never linked.
|
||
- A model change for an agent under test is proposed, never made.
|
||
|
||
## 6. Business value: rating quality, not volume
|
||
|
||
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
|
||
Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric
|
||
(`rubrics/<workflow>.yaml`), starting from the six dimensions below. Each score carries a short quote from the output
|
||
and a reason.
|
||
|
||
| Dimension | Question | Fails when |
|
||
|---|---|---|
|
||
| **Decision-ready** | Can the reader make the decision this step exists for? | It lists data and leaves the conclusion to the reader |
|
||
| **Traceable** | Does every claim and figure trace to a fact and an `evidence_ref`? | A figure appears that no fact supports (checked in code first) |
|
||
| **Owned and actionable** | Does each finding name an owner and a next action? | "Investigate" with no owner (extends the F01 scorecard's `with_owner_share` and `with_action_share`) |
|
||
| **Explained** | Are the subjects described in business terms (which application it is, who uses it, why it matters)? | Ids without descriptions: the APP-001 report gap |
|
||
| **Consumable downstream** | Does the next workflow get every field it needs, filled or declared missing? | Fields filled with placeholders (extends `f02_fields.filled_or_declared`) |
|
||
| **Signal over volume** | Is the output proportionate: no restated inputs, no duplicates, no padding? | Raw records passed through; the same finding written three times |
|
||
|
||
The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a
|
||
case can fail on value as well as on checks, through its `value:` bars.
|
||
|
||
**Calibrating the judge:** before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015).
|
||
Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its
|
||
model version or the rubric changes.
|
||
|
||
## 7. Delivery phases
|
||
|
||
Each phase ends with a check that proves it. The phase's status is kept in `cp1-shakedown/.agents/PLAN.md` once the
|
||
repository exists.
|
||
|
||
| # | Phase | Delivers | Verified by |
|
||
|---|---|---|---|
|
||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
|
||
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
|
||
| 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
|
||
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
|
||
| 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
|
||
|
||
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
|
||
the experimentation loop pays back.
|
||
|
||
## 8. Seed experiments (first `experiments/` entries)
|
||
|
||
These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it.
|
||
|
||
1. **EXP-001 P2 counting unit** (improvement 1): stability on `s5` with the current and the proposed wording; the
|
||
metric is the spread of APP-001's confidence.
|
||
2. **EXP-002 P1 to P3 in code** (improvement 3): a variant hook; the metric is the spread (target 0) and the change
|
||
in the other checks.
|
||
3. **EXP-003 "Compute rate" definition** (improvement 2): replay with the clarified 7R rule; the metric is a
|
||
consistent Rehost precondition across runs.
|
||
4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against
|
||
`gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call.
|
||
5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant.
|
||
6. **EXP-006 F03 business sensitivity:** the six seed variations of `azure-crucible-kab-dev-strategy` (§3a); the
|
||
metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment.
|
||
7. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the
|
||
gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming
|
||
it can be tuned away.
|
||
|
||
## 9. Decisions
|
||
|
||
**Decided (2026-11-09):**
|
||
|
||
| Decision | Choice |
|
||
|---|---|
|
||
| Deep agent framework | LangChain Deep Agents |
|
||
| Bosun's model and the judge | Claude Sonnet 5.5 for both |
|
||
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
|
||
| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows |
|
||
| Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path |
|
||
| Many workflows and linking | Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with `cp1-agentic-landing-zone` (Epic 3214440, F01 to F07) |
|
||
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
|
||
| Hosting | The crucible cluster |
|
||
| Shadow agents | The `shakedown` namespace |
|
||
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
|
||
|
||
**Still open:**
|
||
|
||
1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown`
|
||
OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS
|
||
certificate under `kyndemo.live`.
|
||
2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`,
|
||
held in a Secret, with rights limited to opening pull requests.
|
||
3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's
|
||
`MCP_AGENT_API_KEY` Secret (copied by a person) or needs its own KAB registration.
|
||
4. **Rubric ownership:** who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the
|
||
rubric represents.
|
||
|
||
## 10. Out of scope
|
||
|
||
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
|
||
- Writing to ADO work items.
|
||
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
|
||
- Moving the existing cases out of cp1-agents.
|