Runner: run a voyage remotely as the kab-automation identity #1
@@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) |
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) |
|
||||
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
|
||||
| Status | Draft. Nothing is built yet |
|
||||
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
|
||||
@@ -301,7 +301,7 @@ Shakedown checks each link four ways:
|
||||
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
|
||||
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
|
||||
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
|
||||
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
|
||||
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output. *Amended 2026-11-15:* in Shakedown's own use-case projects, the system may decide a gate under the rule in §5b, and the ledger records that decision as the system's, never as a person's |
|
||||
|
||||
**Rules for linked workflows:**
|
||||
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
|
||||
@@ -350,7 +350,8 @@ Epic and Feature acceptance criteria that have no check yet.
|
||||
in `agents`. They
|
||||
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
|
||||
manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents'
|
||||
rule that each cluster change is approved.
|
||||
rule that each cluster change is approved. *Amended 2026-11-15:* end-to-end use cases (§5b) are the exception.
|
||||
There the system builds agents from its branch and deploys them over the shared agents in `agents`.
|
||||
- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
|
||||
which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path
|
||||
from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
|
||||
@@ -412,8 +413,12 @@ Surveyor scores them. If the change helps without regressing other checks, Shipw
|
||||
review, merge, deploy and record the ADR, as today.
|
||||
|
||||
**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:**
|
||||
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one.
|
||||
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
|
||||
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. *Amended
|
||||
2026-11-15:* in an end-to-end use case (§5b), Lookout's `decide_gate` tool continues a gate only under §5b's rule,
|
||||
and only in Shakedown's own use-case projects. Any other project's gates stay a person's.
|
||||
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. *Amended
|
||||
2026-11-15:* the one exception is §5b's branch deploy: images built from the use case's branch, rolled out to the
|
||||
cp1-agents components in `agents`, under a lock and with the previous tags recorded.
|
||||
- No changes to ADO work items. Bosun's ADO tools are read-only.
|
||||
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
|
||||
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as
|
||||
@@ -442,9 +447,61 @@ request against the agents and workflows automatically, for review the next morn
|
||||
|
||||
Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`,
|
||||
`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a
|
||||
gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
|
||||
gate, except within an end-to-end use case (§5b), whose own rules then apply. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
|
||||
pull requests.
|
||||
|
||||
## 5b. End-to-end use cases that fix what fails (amended 2026-11-15)
|
||||
|
||||
The owner's direction (2026-11-15): name a customer, and the system generates a use case for it and runs Discovery,
|
||||
Assessment and Strategy end to end, remotely, on demand or on a schedule. When a workflow fails, the system fixes the
|
||||
cause on a feature branch it owns, builds the agents from that branch on the platform, and carries on. When the use
|
||||
case finishes, it raises a pull request to `main` for review. This is what the manual Cymbal Bank case of 2026-11-15
|
||||
took a person a day to do (§1).
|
||||
|
||||
**Owner decisions (2026-11-15):**
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Gates | The system decides them, **only** in its own use-case projects. It continues a gate when the run completed, the stage's critic verdict is `pass` and the code checks hold. Otherwise it stops the gate and opens a fix. It never continues a gate on a critic `fail` |
|
||||
| Agents built from the branch | Built on the platform and deployed **over the shared agents** in `agents`, not as shadow agents |
|
||||
| Branches | **One per use case**, `agentic/<customer-slug>-<yyyymmdd>`, owned by the system's gitea identity; one pull request to `main` per repository when the use case ends |
|
||||
| Home | Shakedown (Bosun): Chartmaker, Lookout, Surveyor and Shipwright gain the steps below |
|
||||
|
||||
**How a use case runs:**
|
||||
|
||||
| Step | Who | What |
|
||||
|---|---|---|
|
||||
| 1. Generate | Chartmaker | From the customer's name and brief: the F01 context, F02 rate card and F03 run context, one `engagement_id` (`ENG-<SEGMENTS>`, F01 ADR 0022), and the Discovery source. The source is either one that exists (a live cloud connection, or an estate the sample server already serves) or a new estate added to `alz-discovery-sample` as data on the branch (F01 ADR 0025). Expectations are written before the first run. The case goes to `cp1-shakedown` on the branch |
|
||||
| 2. Set up | Lookout | A KAB project for the customer in CP1, the three workflows associated with it, and the case registered in the portal's `bindings.yaml` (on a branch of the portal) |
|
||||
| 3. Run | Lookout | F01, then F02 from the approved F01 package, then F03 from the approved F02 package; each started from its context file (`--context`) |
|
||||
| 4. Decide each gate | Lookout | By the gate rule above. Each decision carries a note naming the system, the stage's critic verdict and the run that was checked |
|
||||
| 5. Fix on failure | Surveyor, then Shipwright | Surveyor sorts the failure into one kind: a step refused its input (schema); the critic misread a rule; an agent broke a rule (wrong or missing citations); a data or server bug; the platform (auth, quota, an agent not reachable). Shipwright changes the prompt, schema, hook or server data on the branch, with a release note and version bump, and runs `check_agents.py` and the agent's tests. It commits with the failing run's id and the reason |
|
||||
| 6. Build and deploy | Lookout | Builds the changed agents' images from the branch, in parallel, and rolls them out over the shared agents. Then it reruns the stage that failed, not the whole chain: an approved upstream stage is kept |
|
||||
| 7. Prove | Surveyor | A fix holds only when the rerun passes. Because models vary from run to run, a fix to a critic rule needs two passing runs |
|
||||
| 8. Finish | Shipwright | One pull request to `main` per repository (`cp1-agents`, `cp1-shakedown`, the portal if bindings changed). It lists every failure, its kind, the commit that fixed it, and the run that proved it; the versions now deployed; and anything left unresolved |
|
||||
|
||||
**Limits and safety:**
|
||||
- **Three fix attempts per stage,** and the daily budget (§5a). When either runs out, the use case stops and the pull
|
||||
request is raised with the failure marked unresolved.
|
||||
- **One branch deploys at a time.** Shared agents run one branch's code, so a lock in the ledger allows one use
|
||||
case to deploy at a time. Others wait or run without deploying.
|
||||
- **The shared agents run branch code until the pull request is decided.** The portal shows which agents are
|
||||
deployed from which branch. If the pull request is closed unmerged, Lookout rolls those agents back to the image
|
||||
tags it recorded before the first deploy. Merging puts `main` back in line with what's deployed.
|
||||
- **What stays a person's:** merging a pull request; anything in another project, including gates; kamp-demo
|
||||
(read-only); platform-admin actions (claims, registrations owned by another tenant); a model change for an agent;
|
||||
accepting an ADR.
|
||||
- **What the system never changes:** secrets (it reads none); `main`; ADO work items.
|
||||
|
||||
**What it needs from the platform (blocks the build):**
|
||||
1. A **service identity** in realm `kab` (client credentials) with CP1 rights to create projects, start runs and
|
||||
decide gates in its own projects. Personal tokens can't run unattended.
|
||||
2. A **gitea bot** that can push `agentic/*` branches and open pull requests on `cp1-agents`, `cp1-shakedown` and
|
||||
the portal, and nothing more.
|
||||
3. **Cluster rights** for Shakedown's ServiceAccount: push to `cruciblecoredevacr.azurecr.io/kab/cp1/*`, and
|
||||
`patch` on the Deployments of cp1-agents components in `agents` (image only). No `secrets` access, nothing else.
|
||||
4. **In-cluster image builds** (buildah or kaniko Jobs), because there is no laptop in the loop.
|
||||
|
||||
## 6. Business value: rating quality, not volume
|
||||
|
||||
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
|
||||
@@ -524,6 +581,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API |
|
||||
| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) |
|
||||
| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) |
|
||||
| End-to-end use cases (2026-11-15) | Generate a customer's use case and run F01 → F02 → F03 remotely; the system decides gates in its own projects on a critic pass, fixes failures on its own `agentic/<customer>-<date>` branch, builds and deploys over the shared agents, and raises one pull request to `main` at the end (§5b) |
|
||||
| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it |
|
||||
|
||||
**Still open:**
|
||||
@@ -542,7 +600,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
|
||||
## 10. Out of scope
|
||||
|
||||
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
|
||||
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests, except as
|
||||
§5b allows for end-to-end use cases in Shakedown's own projects. Merging stays a person's in every case.
|
||||
- Writing to ADO work items.
|
||||
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
|
||||
- Moving the existing cases out of cp1-agents.
|
||||
|
||||
Reference in New Issue
Block a user