From 07060dbb372c895280af165f1ebab9d992a950c3 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Sun, 15 Nov 2026 21:03:50 +0000 Subject: [PATCH] =?UTF-8?q?Plan=20=C2=A75b:=20end-to-end=20use=20cases=20t?= =?UTF-8?q?hat=20fix=20what=20fails=20(owner,=202026-11-15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Name a customer: the system generates its use case and runs Discovery, Assessment and Strategy remotely. It decides gates only in its own projects and only on a critic pass, fixes failures on an agentic/- branch, builds and deploys over the shared agents (one branch at a time, rolled back if the PR is closed), and raises one PR to main per repository. AC8, §4, §5, §5a, §9 and §10 amended to match. Co-Authored-By: Claude Opus 5.5 --- docs/implementation-plan.md | 73 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 7 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 0a073e1..3052ecc 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea | | | |---|---| -| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) | +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) | | User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | @@ -301,7 +301,7 @@ Shakedown checks each link four ways: | AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't | | AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` | | AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record | -| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output | +| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output. *Amended 2026-11-15:* in Shakedown's own use-case projects, the system may decide a gate under the rule in §5b, and the ledger records that decision as the system's, never as a person's | **Rules for linked workflows:** - A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is @@ -350,7 +350,8 @@ Epic and Feature acceptance criteria that have no check yet. in `agents`. They reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' - rule that each cluster change is approved. + rule that each cluster change is approved. *Amended 2026-11-15:* end-to-end use cases (§5b) are the exception. + There the system builds agents from its branch and deploys them over the shared agents in `agents`. - **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it. @@ -412,8 +413,12 @@ Surveyor scores them. If the change helps without regressing other checks, Shipw review, merge, deploy and record the ADR, as today. **Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:** -- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. -- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. +- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. *Amended + 2026-11-15:* in an end-to-end use case (§5b), Lookout's `decide_gate` tool continues a gate only under §5b's rule, + and only in Shakedown's own use-case projects. Any other project's gates stay a person's. +- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. *Amended + 2026-11-15:* the one exception is §5b's branch deploy: images built from the use case's branch, rolled out to the + cp1-agents components in `agents`, under a lock and with the previous tags recorded. - No changes to ADO work items. Bosun's ADO tools are read-only. - No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read. - Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as @@ -442,9 +447,61 @@ request against the agents and workflows automatically, for review the next morn Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`, `start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a -gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its +gate, except within an end-to-end use case (§5b), whose own rules then apply. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its pull requests. +## 5b. End-to-end use cases that fix what fails (amended 2026-11-15) + +The owner's direction (2026-11-15): name a customer, and the system generates a use case for it and runs Discovery, +Assessment and Strategy end to end, remotely, on demand or on a schedule. When a workflow fails, the system fixes the +cause on a feature branch it owns, builds the agents from that branch on the platform, and carries on. When the use +case finishes, it raises a pull request to `main` for review. This is what the manual Cymbal Bank case of 2026-11-15 +took a person a day to do (§1). + +**Owner decisions (2026-11-15):** + +| Decision | Choice | +|---|---| +| Gates | The system decides them, **only** in its own use-case projects. It continues a gate when the run completed, the stage's critic verdict is `pass` and the code checks hold. Otherwise it stops the gate and opens a fix. It never continues a gate on a critic `fail` | +| Agents built from the branch | Built on the platform and deployed **over the shared agents** in `agents`, not as shadow agents | +| Branches | **One per use case**, `agentic/-`, owned by the system's gitea identity; one pull request to `main` per repository when the use case ends | +| Home | Shakedown (Bosun): Chartmaker, Lookout, Surveyor and Shipwright gain the steps below | + +**How a use case runs:** + +| Step | Who | What | +|---|---|---| +| 1. Generate | Chartmaker | From the customer's name and brief: the F01 context, F02 rate card and F03 run context, one `engagement_id` (`ENG-`, F01 ADR 0022), and the Discovery source. The source is either one that exists (a live cloud connection, or an estate the sample server already serves) or a new estate added to `alz-discovery-sample` as data on the branch (F01 ADR 0025). Expectations are written before the first run. The case goes to `cp1-shakedown` on the branch | +| 2. Set up | Lookout | A KAB project for the customer in CP1, the three workflows associated with it, and the case registered in the portal's `bindings.yaml` (on a branch of the portal) | +| 3. Run | Lookout | F01, then F02 from the approved F01 package, then F03 from the approved F02 package; each started from its context file (`--context`) | +| 4. Decide each gate | Lookout | By the gate rule above. Each decision carries a note naming the system, the stage's critic verdict and the run that was checked | +| 5. Fix on failure | Surveyor, then Shipwright | Surveyor sorts the failure into one kind: a step refused its input (schema); the critic misread a rule; an agent broke a rule (wrong or missing citations); a data or server bug; the platform (auth, quota, an agent not reachable). Shipwright changes the prompt, schema, hook or server data on the branch, with a release note and version bump, and runs `check_agents.py` and the agent's tests. It commits with the failing run's id and the reason | +| 6. Build and deploy | Lookout | Builds the changed agents' images from the branch, in parallel, and rolls them out over the shared agents. Then it reruns the stage that failed, not the whole chain: an approved upstream stage is kept | +| 7. Prove | Surveyor | A fix holds only when the rerun passes. Because models vary from run to run, a fix to a critic rule needs two passing runs | +| 8. Finish | Shipwright | One pull request to `main` per repository (`cp1-agents`, `cp1-shakedown`, the portal if bindings changed). It lists every failure, its kind, the commit that fixed it, and the run that proved it; the versions now deployed; and anything left unresolved | + +**Limits and safety:** +- **Three fix attempts per stage,** and the daily budget (§5a). When either runs out, the use case stops and the pull + request is raised with the failure marked unresolved. +- **One branch deploys at a time.** Shared agents run one branch's code, so a lock in the ledger allows one use + case to deploy at a time. Others wait or run without deploying. +- **The shared agents run branch code until the pull request is decided.** The portal shows which agents are + deployed from which branch. If the pull request is closed unmerged, Lookout rolls those agents back to the image + tags it recorded before the first deploy. Merging puts `main` back in line with what's deployed. +- **What stays a person's:** merging a pull request; anything in another project, including gates; kamp-demo + (read-only); platform-admin actions (claims, registrations owned by another tenant); a model change for an agent; + accepting an ADR. +- **What the system never changes:** secrets (it reads none); `main`; ADO work items. + +**What it needs from the platform (blocks the build):** +1. A **service identity** in realm `kab` (client credentials) with CP1 rights to create projects, start runs and + decide gates in its own projects. Personal tokens can't run unattended. +2. A **gitea bot** that can push `agentic/*` branches and open pull requests on `cp1-agents`, `cp1-shakedown` and + the portal, and nothing more. +3. **Cluster rights** for Shakedown's ServiceAccount: push to `cruciblecoredevacr.azurecr.io/kab/cp1/*`, and + `patch` on the Deployments of cp1-agents components in `agents` (image only). No `secrets` access, nothing else. +4. **In-cluster image builds** (buildah or kaniko Jobs), because there is no laptop in the loop. + ## 6. Business value: rating quality, not volume Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. @@ -524,6 +581,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e | UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API | | Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) | | Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) | +| End-to-end use cases (2026-11-15) | Generate a customer's use case and run F01 → F02 → F03 remotely; the system decides gates in its own projects on a critic pass, fixes failures on its own `agentic/-` branch, builds and deploys over the shared agents, and raises one pull request to `main` at the end (§5b) | | Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it | **Still open:** @@ -542,7 +600,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e ## 10. Out of scope -- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests. +- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests, except as + §5b allows for end-to-end use cases in Shakedown's own projects. Merging stays a person's in every case. - Writing to ADO work items. - Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else. - Moving the existing cases out of cp1-agents.