Plan §5b: end-to-end use cases that fix what fails (owner, 2026-11-15)

Name a customer: the system generates its use case and runs Discovery,
Assessment and Strategy remotely. It decides gates only in its own projects
and only on a critic pass, fixes failures on an agentic/<customer>-<date>
branch, builds and deploys over the shared agents (one branch at a time,
rolled back if the PR is closed), and raises one PR to main per repository.
AC8, §4, §5, §5a, §9 and §10 amended to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-15 21:03:50 +00:00
parent 0890bc5c31
commit 07060dbb37

View File

@@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea
| | |
|---|---|
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) |
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) |
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
| Status | Draft. Nothing is built yet |
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
@@ -301,7 +301,7 @@ Shakedown checks each link four ways:
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output. *Amended 2026-11-15:* in Shakedown's own use-case projects, the system may decide a gate under the rule in §5b, and the ledger records that decision as the system's, never as a person's |
**Rules for linked workflows:**
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
@@ -350,7 +350,8 @@ Epic and Feature acceptance criteria that have no check yet.
in `agents`. They
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents'
rule that each cluster change is approved.
rule that each cluster change is approved. *Amended 2026-11-15:* end-to-end use cases (§5b) are the exception.
There the system builds agents from its branch and deploys them over the shared agents in `agents`.
- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path
from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
@@ -412,8 +413,12 @@ Surveyor scores them. If the change helps without regressing other checks, Shipw
review, merge, deploy and record the ADR, as today.
**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:**
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one.
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. *Amended
2026-11-15:* in an end-to-end use case (§5b), Lookout's `decide_gate` tool continues a gate only under §5b's rule,
and only in Shakedown's own use-case projects. Any other project's gates stay a person's.
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. *Amended
2026-11-15:* the one exception is §5b's branch deploy: images built from the use case's branch, rolled out to the
cp1-agents components in `agents`, under a lock and with the previous tags recorded.
- No changes to ADO work items. Bosun's ADO tools are read-only.
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as
@@ -442,9 +447,61 @@ request against the agents and workflows automatically, for review the next morn
Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`,
`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a
gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
gate, except within an end-to-end use case (§5b), whose own rules then apply. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
pull requests.
## 5b. End-to-end use cases that fix what fails (amended 2026-11-15)
The owner's direction (2026-11-15): name a customer, and the system generates a use case for it and runs Discovery,
Assessment and Strategy end to end, remotely, on demand or on a schedule. When a workflow fails, the system fixes the
cause on a feature branch it owns, builds the agents from that branch on the platform, and carries on. When the use
case finishes, it raises a pull request to `main` for review. This is what the manual Cymbal Bank case of 2026-11-15
took a person a day to do (§1).
**Owner decisions (2026-11-15):**
| Decision | Choice |
|---|---|
| Gates | The system decides them, **only** in its own use-case projects. It continues a gate when the run completed, the stage's critic verdict is `pass` and the code checks hold. Otherwise it stops the gate and opens a fix. It never continues a gate on a critic `fail` |
| Agents built from the branch | Built on the platform and deployed **over the shared agents** in `agents`, not as shadow agents |
| Branches | **One per use case**, `agentic/<customer-slug>-<yyyymmdd>`, owned by the system's gitea identity; one pull request to `main` per repository when the use case ends |
| Home | Shakedown (Bosun): Chartmaker, Lookout, Surveyor and Shipwright gain the steps below |
**How a use case runs:**
| Step | Who | What |
|---|---|---|
| 1. Generate | Chartmaker | From the customer's name and brief: the F01 context, F02 rate card and F03 run context, one `engagement_id` (`ENG-<SEGMENTS>`, F01 ADR 0022), and the Discovery source. The source is either one that exists (a live cloud connection, or an estate the sample server already serves) or a new estate added to `alz-discovery-sample` as data on the branch (F01 ADR 0025). Expectations are written before the first run. The case goes to `cp1-shakedown` on the branch |
| 2. Set up | Lookout | A KAB project for the customer in CP1, the three workflows associated with it, and the case registered in the portal's `bindings.yaml` (on a branch of the portal) |
| 3. Run | Lookout | F01, then F02 from the approved F01 package, then F03 from the approved F02 package; each started from its context file (`--context`) |
| 4. Decide each gate | Lookout | By the gate rule above. Each decision carries a note naming the system, the stage's critic verdict and the run that was checked |
| 5. Fix on failure | Surveyor, then Shipwright | Surveyor sorts the failure into one kind: a step refused its input (schema); the critic misread a rule; an agent broke a rule (wrong or missing citations); a data or server bug; the platform (auth, quota, an agent not reachable). Shipwright changes the prompt, schema, hook or server data on the branch, with a release note and version bump, and runs `check_agents.py` and the agent's tests. It commits with the failing run's id and the reason |
| 6. Build and deploy | Lookout | Builds the changed agents' images from the branch, in parallel, and rolls them out over the shared agents. Then it reruns the stage that failed, not the whole chain: an approved upstream stage is kept |
| 7. Prove | Surveyor | A fix holds only when the rerun passes. Because models vary from run to run, a fix to a critic rule needs two passing runs |
| 8. Finish | Shipwright | One pull request to `main` per repository (`cp1-agents`, `cp1-shakedown`, the portal if bindings changed). It lists every failure, its kind, the commit that fixed it, and the run that proved it; the versions now deployed; and anything left unresolved |
**Limits and safety:**
- **Three fix attempts per stage,** and the daily budget (§5a). When either runs out, the use case stops and the pull
request is raised with the failure marked unresolved.
- **One branch deploys at a time.** Shared agents run one branch's code, so a lock in the ledger allows one use
case to deploy at a time. Others wait or run without deploying.
- **The shared agents run branch code until the pull request is decided.** The portal shows which agents are
deployed from which branch. If the pull request is closed unmerged, Lookout rolls those agents back to the image
tags it recorded before the first deploy. Merging puts `main` back in line with what's deployed.
- **What stays a person's:** merging a pull request; anything in another project, including gates; kamp-demo
(read-only); platform-admin actions (claims, registrations owned by another tenant); a model change for an agent;
accepting an ADR.
- **What the system never changes:** secrets (it reads none); `main`; ADO work items.
**What it needs from the platform (blocks the build):**
1. A **service identity** in realm `kab` (client credentials) with CP1 rights to create projects, start runs and
decide gates in its own projects. Personal tokens can't run unattended.
2. A **gitea bot** that can push `agentic/*` branches and open pull requests on `cp1-agents`, `cp1-shakedown` and
the portal, and nothing more.
3. **Cluster rights** for Shakedown's ServiceAccount: push to `cruciblecoredevacr.azurecr.io/kab/cp1/*`, and
`patch` on the Deployments of cp1-agents components in `agents` (image only). No `secrets` access, nothing else.
4. **In-cluster image builds** (buildah or kaniko Jobs), because there is no laptop in the loop.
## 6. Business value: rating quality, not volume
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
@@ -524,6 +581,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API |
| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) |
| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) |
| End-to-end use cases (2026-11-15) | Generate a customer's use case and run F01 → F02 → F03 remotely; the system decides gates in its own projects on a critic pass, fixes failures on its own `agentic/<customer>-<date>` branch, builds and deploys over the shared agents, and raises one pull request to `main` at the end (§5b) |
| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it |
**Still open:**
@@ -542,7 +600,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
## 10. Out of scope
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests, except as
§5b allows for end-to-end use cases in Shakedown's own projects. Merging stays a person's in every case.
- Writing to ADO work items.
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
- Moving the existing cases out of cp1-agents.