commit c0e4710eb79488439dc80fdb6e9bfeb018bc0dd3 Author: Jonathan Boniface Date: Mon Nov 9 14:55:07 2026 +0000 Shakedown implementation plan Rate, experiment on and build use cases for KAB workflows, starting with ALZ-F02. LangChain Deep Agents with Claude Sonnet 5.5 (Azure) as the agent and judge; deployed to crucible-kab-dev-aks. Co-Authored-By: Claude Opus 5.5 diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..83aed85 --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,284 @@ +# Shakedown: implementation plan + +**Shakedown** is the internal product for testing and evaluating workflows deployed to KAB. It rates their runs, +runs experiments on them, and builds use cases for them. On a ship, the *shakedown cruise* proves the vessel before it +enters service. This app does the same for every workflow before its outputs reach an Engagement Lead. + +| | | +|---|---| +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §8) | +| Status | Draft. Nothing is built yet | +| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | +| Runs on | The `crucible-kab-dev-aks` cluster | +| Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource | +| Judge | **Claude Sonnet 5.5**, a different model family from the gpt-5.x agents under test | +| First workflow under test | ALZ-F02 Assessment & Disposition (`wf_7b6cae08-…`, individual pattern) | +| Then | ALZ-F01 Discovery, then ALZ-F03 Strategy Synthesis as the first workflow onboarded *before* its first live run | + +--- + +## 1. Why: the work Shakedown replaces + +The "ALZ-F02 workflow and agents issues" session (2026-11-02 to 2026-11-09) got F02 to a signed-off package (runs 004 +to 015, commit `66c3274`). Almost every step of that work was done by hand: + +| Manual activity in the session | Evidence | What Shakedown does instead | +|---|---|---| +| Refreshing the 5-minute KAB token | about 40 "kab token refreshed" prompts | You sign in once through KAB's Keycloak; the server-side session renews the token itself | +| Starting a run, polling it, pulling `steps.json` | `kab_run_workflow.py start/status/outputs` | A run ledger: every run is recorded with its context, the workflow version, the agent images and models, its outputs and its usage | +| Keeping rollback copies of the workflow before each run | 11 `workflow-rollback-wf_*.json` files in `reports/cp1` | A snapshot of the workflow and the agents is taken automatically for every run | +| Checking report figures against the facts | PLAN.md improvement 4: "checked by hand" in runs 014 and 015 | Every report figure is checked in code against the facts the run produced | +| Spotting confidence drift between runs | APP-001 scored 0.50 or 0.60 on the same package (improvement 1) | Stability experiments: the same input N times, with the variance shown for each field | +| Spotting ambiguous rules | "a compute rate" read two ways (improvement 2) | The drift report links each divergent field to the prompt rule the agent cited | +| Finding the costliest agent | security posture agent at 520k to 540k tokens (improvement 5) | Cost and latency per step and per run, against the case's `limits:` | +| Judging whether an output is *useful* | "it doesn't break down what the application is within the report" | A business-value rubric scored on every output, not only a schema check | +| Writing checks for a case | Meridian checks still `todo` (F02 PLAN step 10) | The deep agent drafts the checks from the ADO acceptance criteria, and they are frozen before the first run | + +**The acceptance test for the whole product:** given F02 runs 010 to 015, Shakedown finds the APP-001 confidence +drift, the "compute rate" ambiguity, the security posture agent's cost and the unexplained applications in the report, +without anyone pointing it at them. + +## 2. What Shakedown does + +Four capabilities, each usable through the UI and by the deep agent (§4): + +1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. + - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, + severities), run by a generic case runner that works for any workflow, not only F01. + - *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across + steps, every `evidence_ref` names a real execution, and token and time `limits:` are met. + - *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§5). +2. **Experiment.** Change one variable and measure the effect against a baseline: + - *Stability:* the same input N times (default 5), with the spread for each output field. + - *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on + `http://.agents.svc.cluster.local:8080`). This needs no full 12-step workflow run, and no upstream agent + tokens are spent. + - *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with + the same replay inputs sent to the baseline and to the variant. +3. **Build use cases.** Cases are drafted from ADO stories (acceptance criteria and worked examples), from earlier + failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). + Synthetic estates go in `alz-discovery-sample`, which is the one server that may be extended for eval data. +4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake: + - *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's + prompt reads. + - *Live:* a real F01 package run through F02's intake. + - *Traceability:* each ADO acceptance criterion has at least one check, and each check names its criterion. + +## 3. Repository: use cases and experiments, apart from the agents + +The new gitea repository holds Shakedown's code, the **definitions** of new use cases and experiments, and their +**conclusions**. Bulky run data (step outputs, usage, scores) lives in the ledger on the cluster (§4), and each +experiment's record links to its runs by execution id. + +``` +cp1-shakedown/ +├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/) +├── app/ # FastAPI API + UI +├── bosun/ # the deep agent +│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend +│ ├── subagents/ # one module per sub-agent: prompt + tool list +│ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR +├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger +├── use-cases//// +│ ├── case.yaml # checks, value bars, limits; frozen (hashed) before the first run +│ ├── context.yaml # run context (workflow_inputs, customer, plan, context): no run_id, no credentials +│ └── uploads/, README.md +├── rubrics/.yaml # business-value rubric for each workflow (§5) +├── experiments/EXP-NNN-/ +│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule +│ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person +├── variants/// # kustomize overlays for shadow agents: prompt, ConfigMap patch, image tag +└── deploy/ # Shakedown's own Kubernetes manifests, with as in cp1-agents +``` + +- **Existing cases stay where they are.** F01's 7 workflow cases, F02's 2, and each agent's `eval/cases.yaml` remain + in `cp1-agents`. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its + own cases with their source shown. Only **new** use cases and **all** experiments go into the Shakedown repository. +- **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`, + `load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents' + `scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents. +- **Reproducible results:** every ledger run records the commit of cp1-agents and of Shakedown that it ran against. +- **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment + record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges. + +## 4. Architecture on the cluster + +``` + you (browser) ── Keycloak sign-in (realm kab) ──┐ + ▼ +┌──────────────────── namespace: shakedown ──────────────────────────────────────┐ +│ shakedown-app (FastAPI + UI, one replica) │ +│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │ +│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │ +│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │ +│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │ +│ └─ PVC: ledger (SQLite index + one folder per run) and LangGraph checkpoints│ +│ shadow agents (variants/*, applied only with approval) │ +└────────────────────────────────────────────────────────────────────────────────┘ + │ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents` +``` + +- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`). + The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the + ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. + This replaces `kab_token.py` for everything Shakedown does. +- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`) + replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access. +- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no + port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team. +- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents + in `agents`. They + reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply + manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' + rule that each cluster change is approved. +- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, + which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path + from LangChain to the Azure-hosted Claude endpoint before anything else is built on it. +- **Tracing:** LangSmith is off (data would leave the tenant). LangGraph checkpoints and Bosun's tool-call log go to + the ledger instead. +- **Images and manifests:** built with buildah, in parallel, and pushed to + `cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/:`. Manifests are rendered like cp1-agents', and + every `apply` needs your approval. +- **The ledger** records each run as a folder: `context.yaml` (with `generated:`), `workflow.json`, `agents.json` + (image, version, model, max tokens and temperature for each step's agent), `steps.json`, `usage.json`, + `scores.json` and the two commits. A SQLite index sits over the folders. Backfilling runs 004 to 015 is the first + data load. + +## 5. The deep agent: "Bosun" + +The deep agent makes Shakedown scale. It does the investigative work the F02 session did: read outputs, form a +hypothesis, design the experiment, run it, judge the result and propose the change. A person approves anything that +changes state. + +**Built with LangChain Deep Agents.** `create_deep_agent` provides the planning (to-do) tool, a file workspace, +sub-agents with their own prompts and tools, and `interrupt_on`, which pauses before a named tool runs. Shakedown uses +them as follows: + +| Deep Agents feature | Used for | +|---|---| +| Planning tool | Navigator's experiment plan, shown live in the UI | +| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied | +| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small | +| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` | +| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards | + +| Sub-agent | Job | Tools | Writes | +|---|---|---|---| +| **Navigator** (planner) | Turns a goal ("why does APP-001 drift?") into a plan: hypothesis, single variable, baseline, N, metric, stop rule | ledger read, repo read | `experiment.yaml` draft | +| **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries | +| **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` | +| **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun | +| **Chartmaker** (case author) | Drafts cases and checks from ADO acceptance criteria and worked examples, and adversarial variants of them | ADO read, repo read, gitea PR | Pull request to Shakedown | +| **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores | + +Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to +the judge is visible in its scores. + +**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and +Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You +review, merge, deploy and record the ADR, as today. + +**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:** +- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. +- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. +- No changes to ADO work items. Bosun's ADO tools are read-only. +- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read. +- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as + data in every sub-agent's context, and a planted-injection case runs against Bosun itself. +- **Checks are frozen before the run.** A case's hash is recorded when a run starts, and a check edited afterwards + can't score that run. This keeps "written before the run, not tuned to it afterwards" (F02 cases README). +- The patterns stay apart: a finding on `individual-agents/` never proposes a change to `sub-agents/` or to + `outcome-pattern/`. Solution architecture and Discovery are never linked. +- A model change for an agent under test is proposed, never made. + +## 6. Business value: rating quality, not volume + +Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. +Surveyor scores every output that a person or the next workflow consumes against the workflow's rubric +(`rubrics/.yaml`), starting from the six dimensions below. Each score carries a short quote from the output +and a reason. + +| Dimension | Question | Fails when | +|---|---|---| +| **Decision-ready** | Can the reader make the decision this step exists for? | It lists data and leaves the conclusion to the reader | +| **Traceable** | Does every claim and figure trace to a fact and an `evidence_ref`? | A figure appears that no fact supports (checked in code first) | +| **Owned and actionable** | Does each finding name an owner and a next action? | "Investigate" with no owner (extends the F01 scorecard's `with_owner_share` and `with_action_share`) | +| **Explained** | Are the subjects described in business terms (which application it is, who uses it, why it matters)? | Ids without descriptions: the APP-001 report gap | +| **Consumable downstream** | Does the next workflow get every field it needs, filled or declared missing? | Fields filled with placeholders (extends `f02_fields.filled_or_declared`) | +| **Signal over volume** | Is the output proportionate: no restated inputs, no duplicates, no padding? | Raw records passed through; the same finding written three times | + +The F01 package's value scorecard (ADR 0020) is the first instance of this rubric. Shakedown generalises it, so a +case can fail on value as well as on checks, through its `value:` bars. + +**Calibrating the judge:** before Surveyor's scores count, you score 10 outputs by hand (from runs 012 to 015). +Surveyor must agree on at least 8, and every disagreement is reviewed. It is re-calibrated whenever its prompt, its +model version or the rubric changes. + +## 7. Delivery phases + +Each phase ends with a check that proves it. The phase's status is kept in `cp1-shakedown/.agents/PLAN.md` once the +repository exists. + +| # | Phase | Delivers | Verified by | +|---|---|---|---| +| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | +| 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | +| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | +| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | +| 4 | **Alignment and value** | Static contract checks F01 → F02 → F03; live hand-off test; acceptance-criteria traceability from ADO; F02 rubric and judge calibration | Every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs | +| 5 | **Use-case factory** | Chartmaker and Shipwright; drafting of adversarial cases; a patch pull request and ADR draft for each confirmed finding | F03's cases and checks are merged in Shakedown **before** F03's first live run; F03 is onboarded with no new Shakedown code | +| 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric | + +Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where +the experimentation loop pays back. + +## 8. Seed experiments (first `experiments/` entries) + +These come straight from F02's "Improvements to make" list. Shakedown measures each one; you still decide it. + +1. **EXP-001 P2 counting unit** (improvement 1): stability on `s5` with the current and the proposed wording; the + metric is the spread of APP-001's confidence. +2. **EXP-002 P1 to P3 in code** (improvement 3): a variant hook; the metric is the spread (target 0) and the change + in the other checks. +3. **EXP-003 "Compute rate" definition** (improvement 2): replay with the clarified 7R rule; the metric is a + consistent Rehost precondition across runs. +4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against + `gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call. +5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant. +6. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the + gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming + it can be tuned away. + +## 9. Decisions + +**Decided (2026-11-09):** + +| Decision | Choice | +|---|---| +| Deep agent framework | LangChain Deep Agents | +| Bosun's model and the judge | Claude Sonnet 5.5 for both | +| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API | +| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows | +| Existing cases | Stay in cp1-agents and are read by path; only new use cases and all experiments go to Shakedown | +| Hosting | The crucible cluster | +| Shadow agents | The `shakedown` namespace | +| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) | + +**Still open:** + +1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown` + OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS + certificate under `kyndemo.live`. +2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`, + held in a Secret, with rights limited to opening pull requests. +3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's + `MCP_AGENT_API_KEY` Secret (copied by a person) or needs its own KAB registration. +4. **Rubric ownership:** who signs off each workflow's business-value rubric. The Engagement Lead is the consumer the + rubric represents. + +## 10. Out of scope + +- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests. +- Writing to ADO work items. +- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else. +- Moving the existing cases out of cp1-agents.