Plan: many workflows, linked into capabilities

Workflow registry by config; capabilities built read-only from ADO Epics,
starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07); links
as hand-off contracts checked statically, live, for drift and for use;
chain-level checks from the Epic's AC1 to AC8; ripple variations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-09 15:34:41 +00:00
parent 91e118d900
commit 98e3ec69b4

View File

@@ -40,7 +40,7 @@ without anyone pointing it at them.
## 2. What Shakedown does ## 2. What Shakedown does
Four capabilities, each usable through the UI and by the deep agent (§5): Shakedown does four things, each usable through the UI and by the deep agent (§5):
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. 1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
@@ -61,12 +61,17 @@ Four capabilities, each usable through the UI and by the deep agent (§5):
patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it
derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go
in `alz-discovery-sample`, which is the one server that may be extended for eval data. in `alz-discovery-sample`, which is the one server that may be extended for eval data.
4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake: 4. **Align.** Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03
intake:
- *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's - *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's
prompt reads. prompt reads.
- *Live:* a real F01 package run through F02's intake. - *Live:* a voyage, where a real upstream package runs through the downstream intake.
- *Across the chain:* the Epic's acceptance criteria checked over a whole voyage, not one run.
- *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion. - *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion.
Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that
make up one ADO capability are linked into a chain (§3c).
## 3. Repository: use cases and experiments, apart from the agents ## 3. Repository: use cases and experiments, apart from the agents
The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their
@@ -88,7 +93,12 @@ cp1-shakedown/
│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run │ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run
│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a) │ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a)
│ └── README.md # the business situation in a paragraph, and its upstream use case │ └── README.md # the business situation in a paragraph, and its upstream use case
├── voyages/<engagement-id>.yaml # one engagement's chain: F01 → F02 → F03 use cases and how ids flow between them ├── workflows/<workflow>.yaml # registry: one file per workflow under test (§3c)
├── capabilities/<capability>/ # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440)
│ ├── capability.yaml # stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings
│ ├── links.yaml # hand-off contracts between stages
│ ├── expectations.yaml # chain-level checks from the Epic's acceptance criteria
│ └── voyages/<engagement-id>.yaml # one engagement's chain of use cases, and how ids flow between them
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6) ├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6)
├── experiments/EXP-NNN-<slug>/ ├── experiments/EXP-NNN-<slug>/
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule │ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
@@ -131,7 +141,8 @@ Shakedown standardises the folder, not the workflows:
**A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's **A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's
`azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`, `azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`,
but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package
artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). `voyages/<engagement>.yaml` names the use case for each stage and which upstream output fills each artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). A voyage file
(`capabilities/<capability>/voyages/<engagement>.yaml`) names the use case for each stage and which upstream output fills each
downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run: downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run:
latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as
`kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live `kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live
@@ -199,7 +210,7 @@ Shakedown starts from the use cases already written. The committed ones stay in
alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and
reports agents with no eval case. reports agents with no eval case.
**Voyages to define first:** **Voyages to define first** (in `capabilities/cp1-agentic-landing-zone/voyages/`):
- `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03 - `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03
`azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today. `azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today.
- `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker - `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker
@@ -214,6 +225,98 @@ reports agents with no eval case.
variants vary sources and data. variants vary sources and data.
- Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file. - Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file.
## 3c. Many workflows, linked into capabilities
Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a **capability**: an ADO
Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three
stages of one capability, the Epic **3214440 "CP1 – Agentic Landing Zone"**. Its own description sets the chain
("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them.
**The workflow registry: `workflows/<workflow>.yaml`, one file for each workflow under test.** Onboarding a workflow
means adding this file and its rubric, with no code. Each file holds:
- its key (`alz-f02-assessment`), its ADO Feature (3214442), its workstream tag and its pattern (`individual-agents`,
`sub-agents` or `outcome-pattern`);
- its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so
their history is kept and each ledger run records the one it used;
- its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as
`evaluate_workflow_run.py` already does;
- its context adapter: an upload through `--context` (F01), `--input` flags (F02), or the run input
`run_context_document` (F03);
- its human gates (for example `disposition_signoff`), what it produces (its package and the package's schema), and
what it consumes.
A workflow needn't belong to a capability. The solution architecture agent (`outcome-pattern/`) can be registered and
rated on its own, and is never linked to Discovery.
**The capability: `capabilities/cp1-agentic-landing-zone/capability.yaml`.** Shakedown builds it from ADO, read-only:
the Epic, its Features, their Stories and acceptance criteria, each with its `rev`. The stage order, gates and
"mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves
each stage. When an item's `rev` changes, Shakedown flags the expectations written against the older wording.
| # | Stage | ADO Feature | Workflow binding | Human gate | State on 2026-11-09 |
|---|---|---|---|---|---|
| 1 | Discovery | 3214441 | `alz-f01-discovery` (individual). The sub-agent build is a second binding, never mixed with the first | none | Deployed; live runs |
| 2 | Assessment | 3214442 | `alz-f02-assessment` | `disposition_signoff` | Deployed; runs 014 and 015 signed off |
| 3 | Strategy | 3214443 | `alz-f03-strategy` | `strategy_decision` | Built (6 agents, export of 9 steps); not deployed or imported |
| 4 | Business Case | 3214444 | none yet | assumption validation | Not built |
| 5 | Target Architecture | 3214445 | none yet | architect blueprint | Not built |
| 6 | Wave Planning | 3214446 | none yet | wave plan approval | Not built; tagged `Review - Overlap` |
| 7 | Landing Zone | 3214447 | none yet | `lz_foundation_approval`, `tf_high_blast_approval` | Not built; the only stage that mutates |
**The links: `capabilities/cp1-agentic-landing-zone/links.yaml`.** Each link is a hand-off contract between two
stages. It names:
- the producer's step and artifact, and the schema of what passes;
- the consumer's intake step;
- the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids;
- the upstream gate that must be cleared first;
- the ADRs that define the link.
| Link | What passes | Consumer's check | Inputs it fills | Defined by |
|---|---|---|---|---|
| F01 → F02 | The Discovery package: a JSON package with a manifest and the value scorecard | F02 intake `s0`: one package, a sha256 for each part, complete, critic `pass`, the named run and engagement | `discovery_execution_id`, `discovery_approval_ref`, `artifact_ids` | F01 ADR 0020; F02 ADR 0008 |
| F02 → F03 | The Assessment package | F03 intake | `assessment_execution_id`, `assessment_approval_ref`, the package's artifact id | F02 ADRs 0014, 0015; F03 ADR 0002 |
| F03 → F04 onwards | Not defined yet | | | Drafted when F04 is designed |
Shakedown checks each link four ways:
- **Statically:** the producer's package schema against the consumer's intake schema and the fields its prompts
read, at the pinned commit.
- **Live:** a voyage.
- **For drift:** when either side's schema changes in cp1-agents, the link is marked stale until the static check
passes again on the new commit.
- **For use:** a field that is produced but never consumed downstream is data passed on with no purpose, which is
reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must
be declared missing, never left out silently.
**Chain-level expectations, from the Epic's acceptance criteria:**
`capabilities/cp1-agentic-landing-zone/expectations.yaml` is checked over a voyage, not over one run.
| Epic AC | Checked over a voyage |
|---|---|
| AC1 End-to-end chain | Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference |
| AC2 Shared agentic rails | Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence |
| AC3 Declared equals executed | Each run's executed steps and tool calls match its workflow export |
| AC4 Critic verdict binds | No stage ends `completed` after a critic `fail` or an unparseable verdict |
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
**Rules for linked workflows:**
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
rated on its own and is never linked.
- A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can
draft its use cases, link and expectations before the workflow is built, starting with F04.
- **Ripple variations** change an input at one stage and follow the effect down the chain. For example, adding the
CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage
downstream, so each one needs your approval.
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
links declared. CP4 Epics are next after the landing zone.
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
Epic and Feature acceptance criteria that have no check yet.
## 4. Architecture on the cluster ## 4. Architecture on the cluster
``` ```
@@ -335,12 +438,12 @@ repository exists.
| # | Phase | Delivers | Verified by | | # | Phase | Delivers | Verified by |
|---|---|---|---| |---|---|---|---|
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | | 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | | 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | | 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | | 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
| 4 | **Alignment, value and variations** | Static contract checks F01 → F02 → F03; voyages with ids resolved from upstream runs; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F02 → F03 with no id typed by hand; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence | | 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code | | 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
| 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric | | 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
the experimentation loop pays back. the experimentation loop pays back.
@@ -375,6 +478,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API | | Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows | | Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows |
| Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path | | Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path |
| Many workflows and linking | Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with `cp1-agentic-landing-zone` (Epic 3214440, F01 to F07) |
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale | | Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
| Hosting | The crucible cluster | | Hosting | The crucible cluster |
| Shadow agents | The `shakedown` namespace | | Shadow agents | The `shakedown` namespace |