diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 83aed85..e7bd261 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea | | | |---|---| -| Date | 2026-11-09 (revised the same day with the product owner's decisions, §8) | +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | | Runs on | The `crucible-kab-dev-aks` cluster | @@ -40,14 +40,14 @@ without anyone pointing it at them. ## 2. What Shakedown does -Four capabilities, each usable through the UI and by the deep agent (§4): +Four capabilities, each usable through the UI and by the deep agent (§5): 1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, severities), run by a generic case runner that works for any workflow, not only F01. - *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across steps, every `evidence_ref` names a real execution, and token and time `limits:` are met. - - *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§5). + - *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§6). 2. **Experiment.** Change one variable and measure the effect against a baseline: - *Stability:* the same input N times (default 5), with the spread for each output field. - *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on @@ -55,9 +55,12 @@ Four capabilities, each usable through the UI and by the deep agent (§4): tokens are spent. - *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with the same replay inputs sent to the baseline and to the variant. -3. **Build use cases.** Cases are drafted from ADO stories (acceptance criteria and worked examples), from earlier - failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). - Synthetic estates go in `alz-discovery-sample`, which is the one server that may be extended for eval data. +3. **Build use cases and their variations.** A use case is a business scenario fed to a workflow (§3a): its run + context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new + ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial + patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it + derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go + in `alz-discovery-sample`, which is the one server that may be extended for eval data. 4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake: - *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's prompt reads. @@ -79,11 +82,14 @@ cp1-shakedown/ │ ├── subagents/ # one module per sub-agent: prompt + tool list │ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR ├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger -├── use-cases//// -│ ├── case.yaml # checks, value bars, limits; frozen (hashed) before the first run -│ ├── context.yaml # run context (workflow_inputs, customer, plan, context): no run_id, no credentials -│ └── uploads/, README.md -├── rubrics/.yaml # business-value rubric for each workflow (§5) +├── use-cases/// # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/ +│ ├── run-context.yaml # the scenario, in the shape its workflow takes; no run_id, no credentials +│ ├── uploads/ # CMDB exports, rate cards … +│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run +│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a) +│ └── README.md # the business situation in a paragraph, and its upstream use case +├── voyages/.yaml # one engagement's chain: F01 → F02 → F03 use cases and how ids flow between them +├── rubrics/.yaml # business-value rubric for each workflow (§6) ├── experiments/EXP-NNN-/ │ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule │ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person @@ -91,9 +97,11 @@ cp1-shakedown/ └── deploy/ # Shakedown's own Kubernetes manifests, with as in cp1-agents ``` -- **Existing cases stay where they are.** F01's 7 workflow cases, F02's 2, and each agent's `eval/cases.yaml` remain - in `cp1-agents`. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its - own cases with their source shown. Only **new** use cases and **all** experiments go into the Shakedown repository. +- **New use cases start here, from F03.** F03's `azure-crucible-kab-dev-strategy` (written 2026-11-09, not yet + committed to cp1-agents) is the first use case in this repository, and every later one is created here. The + committed F01 cases (7) and F02 cases (2), and each agent's `eval/cases.yaml`, stay in `cp1-agents`. Shakedown reads + them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown. + **All** experiments go into this repository. - **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`, `load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents' `scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents. @@ -101,6 +109,111 @@ cp1-shakedown/ - **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges. +## 3a. Use cases, voyages and variations + +**A use case is a business scenario, not a test script.** Today's folders show its parts, spread unevenly across +Features: + +| Feature | Scenario | How it reaches the run | Expectations | +|---|---|---|---| +| F01 | `context.yaml` (customer, plan, scope, sources) and CMDB uploads for live cases; `brief.txt` for the sample cases | Uploaded through `kab_run_workflow.py --context`, or `--file` | `case.yaml` checks in 5 of 7; `azure-live` and `gcp-live` have none | +| F02 | `rate-card.json`; everything else is typed as `--input` flags (execution id, approval reference, artifact id) | Run inputs, including `rate_card_document`; no context file, unlike F01 and F03 | Written as prose in the README only | +| F03 | `run-context.yaml` (objectives, constraints, priorities, planning assumptions) | The run input `run_context_document`, as text (F03 ADR 0002) | None | + +Shakedown standardises the folder, not the workflows: +- **`run-context.yaml` stays in its workflow's own format.** A small adapter for each workflow knows how to deliver + it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their + context three ways is reported as an alignment finding (§2, Align) for you to decide on. +- **`expectations.yaml`** gathers what each Feature now keeps in different places (checks, value bars, `limits:`), + and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned + into checks by Chartmaker and reviewed by you. + +**A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's +`azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`, +but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package +artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). `voyages/.yaml` names the use case for each stage and which upstream output fills each +downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run: +latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as +`kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live +alignment test: the downstream intake has to accept what the upstream run really produced. + +**A variation changes one business input and states how the output must respond.** This is the most direct test of +business value: an output that doesn't move when the business situation changes is not reasoning about the business. +Each entry in `variations.yaml` has a `change` (a path in `run-context.yaml` and its new value) and one or more +`expect` relations: + +| Relation | Meaning | Judged by | +|---|---|---| +| `direction` | A named output metric goes up or down against the base run | Code, from the metric's path | +| `bound` | A limit from the input is respected (no plan above `budget_ceiling`) | Code | +| `unchanged` | Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) | Code | +| `rationale` | The output's reasoning cites the changed input | Claude Sonnet, with a quote | + +Seed variations for F03's `azure-crucible-kab-dev-strategy`, drafted from its run context: + +| Change | Expected | +|---|---| +| `priorities.cost` 0.3 → 0.7 (risk 0.4 → 0.1) | Total cost of the recommended strategy goes down; the rationale cites cost priority | +| `constraints.budget_ceiling.amount` 40000 → 20000 | No recommended plan exceeds 20,000 USD, or the output says none can, and why | +| `constraints.risk_appetite` medium → low | Refactor share goes down; risk mitigations go up | +| `planning_assumptions.team_capacity` 2 → 1 | The timeline gets longer; the disposition set is unchanged | +| `constraints.exit_date` 2027-06-30 → 2027-03-31 | Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't | +| `constraints.data_residency` drop `eastus2` | No target in `eastus2` | + +**Variations stay cheap.** Only the stage whose input changed is run again. The upstream ids stay pinned to the base +run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation +passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget +for each batch, and `start_replay_batch` asks for your approval. + +## 3b. The existing use cases + +Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place +(§3), and F03's moves to this repository. Paths are relative to `cp1-agents/individual-agents/`. + +**ALZ-F01 Discovery:** `ALZ-F01: Discovery Workflow Agent/workflows/cases/` + +| Use case | Scenario | Engagement | Expectations today | What Shakedown adds | +|---|---|---|---|---| +| `azure-crucible-kab-dev-cmdb` | Live Azure, resource group `crucible-kab-dev`, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection | `ENG-AZURE-KAB-DEV-002` | 48 checks, `value:` bar, `limits:` | First stage of the `ENG-AZURE-KAB-DEV-002` voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner) | +| `azure-crucible-kab-dev` | The same group, no CMDB or ITSM: every gap declared, a complete package | `ENG-AZURE-KAB-DEV-001` | 15 checks, `value:` bar, `limits:` | The base its `-cmdb` sibling is a variation of: `unchanged` relations for the estate facts, `direction` for drift findings | +| `meridian-baseline` | Meridian Retail Group sample estate (`provider: sample`): six scope entries, one unscannable HSM, a planted prompt injection | `ENG-MERIDIAN-001` | 45 checks | First stage of the Meridian voyage; rubric scores (no `value:` bar yet) | +| `meridian-without-itsm` | Meridian with ITSM turned off on the run form | `ENG-MERIDIAN-001` | 6 checks | Recast as a variation of `meridian-baseline` (`include_itsm` yes → no) with `unchanged` and `direction` relations | +| `snapshot-missing-sources` | A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source | any | 8 checks | Adversarial base for "declare the gap, don't invent it" | +| `azure-live` | Whole subscription `sub-kyn-RITM0250380-653-dev`, about 644 resources | `ENG-AZURE-LIVE-001` | None | Expectations drafted by Chartmaker (scale, cost and coverage `limits:` first) | +| `gcp-live` | Whole live GCP project | `ENG-GCP-LIVE-001` | None | Expectations drafted by Chartmaker; the same expectations as `azure-live` where the providers should agree (hyperscaler servers stay separate, the workflow is the same) | + +**ALZ-F02 Assessment & Disposition:** `ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/` + +| Use case | Scenario | Engagement | Expectations today | What Shakedown adds | +|---|---|---|---|---| +| `azure-crucible-kab-dev-assessment` | The Discovery package of an `azure-crucible-kab-dev-cmdb` run, and an illustrative Azure rate card | `ENG-AZURE-KAB-DEV-002` | Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run | Prose turned into `expectations.yaml` (reviewed by you); a `run-context.yaml` gathering its `--input` flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent) | +| `meridian-assessment` | A Meridian Discovery run (`RUN-008`, critic `pass`) and the sample rate card | `ENG-MERIDIAN-001` | Six expectations as prose in `cases/README.md` (no cloud, risk, partial score, dependents, sign-off, evidence); checks still `todo` in F02 PLAN step 10 | Expectations written and frozen before its first scored run; second stage of the Meridian voyage | + +**ALZ-F03 Strategy Synthesis:** `ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/` + +| Use case | Scenario | Engagement | Expectations today | What Shakedown adds | +|---|---|---|---|---| +| `azure-crucible-kab-dev-strategy` | Strategy for the KAB Platform on the signed-off Assessment `exec_374d5d577077`: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in `eastus` and `eastus2`, priorities and planning assumptions | `ENG-AZURE-KAB-DEV-002` | None | Moves to `cp1-shakedown/use-cases/alz-f03-strategy/`; expectations; third stage of the voyage; the six seed variations (§3a) | + +**Each agent's `eval/cases.yaml`** (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent +alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and +reports agents with no eval case. + +**Voyages to define first:** +- `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03 + `azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today. +- `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker + drafts one. + +**Gaps across the set** (each one a first job for Chartmaker, reviewed by you): +- Expectations are missing for `azure-live`, `gcp-live` and F03, and are only prose for both F02 use cases. +- F02 has no context file. Its scenario is spread across `--input` flags in a README, which goes against the + convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on. + Until then, Shakedown's F02 adapter turns `run-context.yaml` into those inputs. +- No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02 + variants vary sources and data. +- Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file. + ## 4. Architecture on the cluster ``` @@ -168,7 +281,7 @@ them as follows: | **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries | | **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` | | **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun | -| **Chartmaker** (case author) | Drafts cases and checks from ADO acceptance criteria and worked examples, and adversarial variants of them | ADO read, repo read, gitea PR | Pull request to Shakedown | +| **Chartmaker** (use-case author) | Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions | ADO read, repo read, gitea PR | Pull request to Shakedown | | **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores | Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to @@ -221,12 +334,12 @@ repository exists. | # | Phase | Delivers | Verified by | |---|---|---|---| -| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | +| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | | 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | | 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | | 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | -| 4 | **Alignment and value** | Static contract checks F01 → F02 → F03; live hand-off test; acceptance-criteria traceability from ADO; F02 rubric and judge calibration | Every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs | -| 5 | **Use-case factory** | Chartmaker and Shipwright; drafting of adversarial cases; a patch pull request and ADR draft for each confirmed finding | F03's cases and checks are merged in Shakedown **before** F03's first live run; F03 is onboarded with no new Shakedown code | +| 4 | **Alignment, value and variations** | Static contract checks F01 → F02 → F03; voyages with ids resolved from upstream runs; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F02 → F03 with no id typed by hand; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence | +| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code | | 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric | Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where @@ -245,7 +358,9 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e 4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against `gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call. 5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant. -6. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the +6. **EXP-006 F03 business sensitivity:** the six seed variations of `azure-crucible-kab-dev-strategy` (§3a); the + metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment. +7. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming it can be tuned away. @@ -259,7 +374,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e | Bosun's model and the judge | Claude Sonnet 5.5 for both | | Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API | | Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows | -| Existing cases | Stay in cp1-agents and are read by path; only new use cases and all experiments go to Shakedown | +| Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path | +| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale | | Hosting | The crucible cluster | | Shadow agents | The `shakedown` namespace | | UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |