Plan: many workflows, linked into capabilities
Workflow registry by config; capabilities built read-only from ADO Epics, starting with cp1-agentic-landing-zone (Epic 3214440, F01 to F07); links as hand-off contracts checked statically, live, for drift and for use; chain-level checks from the Epic's AC1 to AC8; ripple variations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -40,7 +40,7 @@ without anyone pointing it at them.
|
||||
|
||||
## 2. What Shakedown does
|
||||
|
||||
Four capabilities, each usable through the UI and by the deep agent (§5):
|
||||
Shakedown does four things, each usable through the UI and by the deep agent (§5):
|
||||
|
||||
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
|
||||
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
|
||||
@@ -61,12 +61,17 @@ Four capabilities, each usable through the UI and by the deep agent (§5):
|
||||
patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it
|
||||
derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go
|
||||
in `alz-discovery-sample`, which is the one server that may be extended for eval data.
|
||||
4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake:
|
||||
4. **Align.** Check every link between workflows in a capability (§3c), such as F01 package → F02 intake → F03
|
||||
intake:
|
||||
- *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's
|
||||
prompt reads.
|
||||
- *Live:* a real F01 package run through F02's intake.
|
||||
- *Live:* a voyage, where a real upstream package runs through the downstream intake.
|
||||
- *Across the chain:* the Epic's acceptance criteria checked over a whole voyage, not one run.
|
||||
- *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion.
|
||||
|
||||
Shakedown is not built around one workflow. Every workflow is registered by configuration, and the workflows that
|
||||
make up one ADO capability are linked into a chain (§3c).
|
||||
|
||||
## 3. Repository: use cases and experiments, apart from the agents
|
||||
|
||||
The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their
|
||||
@@ -88,7 +93,12 @@ cp1-shakedown/
|
||||
│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run
|
||||
│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a)
|
||||
│ └── README.md # the business situation in a paragraph, and its upstream use case
|
||||
├── voyages/<engagement-id>.yaml # one engagement's chain: F01 → F02 → F03 use cases and how ids flow between them
|
||||
├── workflows/<workflow>.yaml # registry: one file per workflow under test (§3c)
|
||||
├── capabilities/<capability>/ # e.g. capabilities/cp1-agentic-landing-zone/ (ADO Epic 3214440)
|
||||
│ ├── capability.yaml # stages from the Epic's Features, synced read-only from ADO, plus their workflow bindings
|
||||
│ ├── links.yaml # hand-off contracts between stages
|
||||
│ ├── expectations.yaml # chain-level checks from the Epic's acceptance criteria
|
||||
│ └── voyages/<engagement-id>.yaml # one engagement's chain of use cases, and how ids flow between them
|
||||
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6)
|
||||
├── experiments/EXP-NNN-<slug>/
|
||||
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
|
||||
@@ -131,7 +141,8 @@ Shakedown standardises the folder, not the workflows:
|
||||
**A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's
|
||||
`azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`,
|
||||
but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package
|
||||
artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). `voyages/<engagement>.yaml` names the use case for each stage and which upstream output fills each
|
||||
artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). A voyage file
|
||||
(`capabilities/<capability>/voyages/<engagement>.yaml`) names the use case for each stage and which upstream output fills each
|
||||
downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run:
|
||||
latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as
|
||||
`kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live
|
||||
@@ -199,7 +210,7 @@ Shakedown starts from the use cases already written. The committed ones stay in
|
||||
alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and
|
||||
reports agents with no eval case.
|
||||
|
||||
**Voyages to define first:**
|
||||
**Voyages to define first** (in `capabilities/cp1-agentic-landing-zone/voyages/`):
|
||||
- `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03
|
||||
`azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today.
|
||||
- `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker
|
||||
@@ -214,6 +225,98 @@ reports agents with no eval case.
|
||||
variants vary sources and data.
|
||||
- Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file.
|
||||
|
||||
## 3c. Many workflows, linked into capabilities
|
||||
|
||||
Shakedown tests any number of workflows. A workflow can stand alone, or it can be a stage of a **capability**: an ADO
|
||||
Epic whose Features are workflows that hand their outputs on to each other. ALZ-F01, F02 and F03 are the first three
|
||||
stages of one capability, the Epic **3214440 "CP1 – Agentic Landing Zone"**. Its own description sets the chain
|
||||
("seven linked phases, Discovery → … → Landing Zone") and its acceptance criteria apply across all of them.
|
||||
|
||||
**The workflow registry: `workflows/<workflow>.yaml`, one file for each workflow under test.** Onboarding a workflow
|
||||
means adding this file and its rubric, with no code. Each file holds:
|
||||
- its key (`alz-f02-assessment`), its ADO Feature (3214442), its workstream tag and its pattern (`individual-agents`,
|
||||
`sub-agents` or `outcome-pattern`);
|
||||
- its source path in cp1-agents, and its KAB workflow id for each environment. Ids change on every re-import, so
|
||||
their history is kept and each ledger run records the one it used;
|
||||
- its step-to-agent map, read from the export. Import renames step ids, so steps are matched by agent name, as
|
||||
`evaluate_workflow_run.py` already does;
|
||||
- its context adapter: an upload through `--context` (F01), `--input` flags (F02), or the run input
|
||||
`run_context_document` (F03);
|
||||
- its human gates (for example `disposition_signoff`), what it produces (its package and the package's schema), and
|
||||
what it consumes.
|
||||
|
||||
A workflow needn't belong to a capability. The solution architecture agent (`outcome-pattern/`) can be registered and
|
||||
rated on its own, and is never linked to Discovery.
|
||||
|
||||
**The capability: `capabilities/cp1-agentic-landing-zone/capability.yaml`.** Shakedown builds it from ADO, read-only:
|
||||
the Epic, its Features, their Stories and acceptance criteria, each with its `rev`. The stage order, gates and
|
||||
"mutates" flag come from the Epic's phase-chain table. You add what ADO doesn't know: which registered workflow serves
|
||||
each stage. When an item's `rev` changes, Shakedown flags the expectations written against the older wording.
|
||||
|
||||
| # | Stage | ADO Feature | Workflow binding | Human gate | State on 2026-11-09 |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | Discovery | 3214441 | `alz-f01-discovery` (individual). The sub-agent build is a second binding, never mixed with the first | none | Deployed; live runs |
|
||||
| 2 | Assessment | 3214442 | `alz-f02-assessment` | `disposition_signoff` | Deployed; runs 014 and 015 signed off |
|
||||
| 3 | Strategy | 3214443 | `alz-f03-strategy` | `strategy_decision` | Built (6 agents, export of 9 steps); not deployed or imported |
|
||||
| 4 | Business Case | 3214444 | none yet | assumption validation | Not built |
|
||||
| 5 | Target Architecture | 3214445 | none yet | architect blueprint | Not built |
|
||||
| 6 | Wave Planning | 3214446 | none yet | wave plan approval | Not built; tagged `Review - Overlap` |
|
||||
| 7 | Landing Zone | 3214447 | none yet | `lz_foundation_approval`, `tf_high_blast_approval` | Not built; the only stage that mutates |
|
||||
|
||||
**The links: `capabilities/cp1-agentic-landing-zone/links.yaml`.** Each link is a hand-off contract between two
|
||||
stages. It names:
|
||||
- the producer's step and artifact, and the schema of what passes;
|
||||
- the consumer's intake step;
|
||||
- the input mapping (which downstream input is filled from which upstream field). Voyages use this to fill ids;
|
||||
- the upstream gate that must be cleared first;
|
||||
- the ADRs that define the link.
|
||||
|
||||
| Link | What passes | Consumer's check | Inputs it fills | Defined by |
|
||||
|---|---|---|---|---|
|
||||
| F01 → F02 | The Discovery package: a JSON package with a manifest and the value scorecard | F02 intake `s0`: one package, a sha256 for each part, complete, critic `pass`, the named run and engagement | `discovery_execution_id`, `discovery_approval_ref`, `artifact_ids` | F01 ADR 0020; F02 ADR 0008 |
|
||||
| F02 → F03 | The Assessment package | F03 intake | `assessment_execution_id`, `assessment_approval_ref`, the package's artifact id | F02 ADRs 0014, 0015; F03 ADR 0002 |
|
||||
| F03 → F04 onwards | Not defined yet | | | Drafted when F04 is designed |
|
||||
|
||||
Shakedown checks each link four ways:
|
||||
- **Statically:** the producer's package schema against the consumer's intake schema and the fields its prompts
|
||||
read, at the pinned commit.
|
||||
- **Live:** a voyage.
|
||||
- **For drift:** when either side's schema changes in cp1-agents, the link is marked stale until the static check
|
||||
passes again on the new commit.
|
||||
- **For use:** a field that is produced but never consumed downstream is data passed on with no purpose, which is
|
||||
reported against the producer's "signal over volume" score (§6). A field that is consumed but never produced must
|
||||
be declared missing, never left out silently.
|
||||
|
||||
**Chain-level expectations, from the Epic's acceptance criteria:**
|
||||
`capabilities/cp1-agentic-landing-zone/expectations.yaml` is checked over a voyage, not over one run.
|
||||
|
||||
| Epic AC | Checked over a voyage |
|
||||
|---|---|
|
||||
| AC1 End-to-end chain | Every downstream figure, disposition and decision traces by evidence reference back to a Discovery record. Shakedown walks the chain and lists each broken reference |
|
||||
| AC2 Shared agentic rails | Where they can be observed: the planted-injection use cases are refused at every stage, and every stage reads back its evidence |
|
||||
| AC3 Declared equals executed | Each run's executed steps and tool calls match its workflow export |
|
||||
| AC4 Critic verdict binds | No stage ends `completed` after a critic `fail` or an unparseable verdict |
|
||||
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
|
||||
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
|
||||
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
|
||||
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
|
||||
|
||||
**Rules for linked workflows:**
|
||||
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
|
||||
rated on its own and is never linked.
|
||||
- A stage with no workflow yet shows in the capability view as a gap, with its ADO stories ready. Chartmaker can
|
||||
draft its use cases, link and expectations before the workflow is built, starting with F04.
|
||||
- **Ripple variations** change an input at one stage and follow the effect down the chain. For example, adding the
|
||||
CMDB upload at F01: does F03's strategy change, and is the change explained? These cost a run at every stage
|
||||
downstream, so each one needs your approval.
|
||||
|
||||
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
|
||||
links declared. CP4 Epics are next after the landing zone.
|
||||
|
||||
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
|
||||
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
|
||||
Epic and Feature acceptance criteria that have no check yet.
|
||||
|
||||
## 4. Architecture on the cluster
|
||||
|
||||
```
|
||||
@@ -335,12 +438,12 @@ repository exists.
|
||||
| # | Phase | Delivers | Verified by |
|
||||
|---|---|---|---|
|
||||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||||
| 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||||
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||||
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
|
||||
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
|
||||
| 4 | **Alignment, value and variations** | Static contract checks F01 → F02 → F03; voyages with ids resolved from upstream runs; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F02 → F03 with no id typed by hand; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
|
||||
| 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
|
||||
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
|
||||
| 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric |
|
||||
| 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
|
||||
|
||||
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
|
||||
the experimentation loop pays back.
|
||||
@@ -375,6 +478,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
|
||||
| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows |
|
||||
| Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path |
|
||||
| Many workflows and linking | Every workflow is registered by config; workflows of one ADO Epic are linked into a capability, starting with `cp1-agentic-landing-zone` (Epic 3214440, F01 to F07) |
|
||||
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
|
||||
| Hosting | The crucible cluster |
|
||||
| Shadow agents | The `shakedown` namespace |
|
||||
|
||||
Reference in New Issue
Block a user