Runner: run a voyage remotely as the kab-automation identity #1
@@ -6,7 +6,8 @@ enters service. This app does the same for every workflow before its outputs rea
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) |
|
||||
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
|
||||
| Status | Draft. Nothing is built yet |
|
||||
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
|
||||
| Runs on | The `crucible-kab-dev-aks` cluster |
|
||||
@@ -40,7 +41,7 @@ without anyone pointing it at them.
|
||||
|
||||
## 2. What Shakedown does
|
||||
|
||||
Shakedown does four things, each usable through the UI and by the deep agent (§5):
|
||||
Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5):
|
||||
|
||||
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
|
||||
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
|
||||
@@ -81,7 +82,8 @@ experiment's record links to its runs by execution id.
|
||||
```
|
||||
cp1-shakedown/
|
||||
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
|
||||
├── app/ # FastAPI API + UI
|
||||
├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005)
|
||||
├── skills/<skill>/SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010)
|
||||
├── bosun/ # the deep agent
|
||||
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
|
||||
│ ├── subagents/ # one module per sub-agent: prompt + tool list
|
||||
@@ -313,18 +315,18 @@ Shakedown checks each link four ways:
|
||||
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
|
||||
links declared. CP4 Epics are next after the landing zone.
|
||||
|
||||
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
|
||||
**In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
|
||||
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
|
||||
Epic and Feature acceptance criteria that have no check yet.
|
||||
|
||||
## 4. Architecture on the cluster
|
||||
|
||||
```
|
||||
you (browser) ── Keycloak sign-in (realm kab) ──┐
|
||||
▼
|
||||
you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐
|
||||
bearer token (you) ▼
|
||||
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
|
||||
│ shakedown-app (FastAPI + UI, one replica) │
|
||||
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
|
||||
│ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │
|
||||
│ ├─ acts as you: the portal's token, validated per request; never stored │
|
||||
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
|
||||
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
|
||||
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
|
||||
@@ -334,14 +336,16 @@ Epic and Feature acceptance criteria that have no check yet.
|
||||
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
|
||||
```
|
||||
|
||||
- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`).
|
||||
The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the
|
||||
ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you.
|
||||
This replaces `kab_token.py` for everything Shakedown does.
|
||||
- **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the
|
||||
session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates
|
||||
that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores
|
||||
it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity
|
||||
of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown
|
||||
does.
|
||||
- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`)
|
||||
replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access.
|
||||
- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no
|
||||
port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
|
||||
- **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the
|
||||
portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views.
|
||||
- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents
|
||||
in `agents`. They
|
||||
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
|
||||
@@ -372,10 +376,10 @@ them as follows:
|
||||
|
||||
| Deep Agents feature | Used for |
|
||||
|---|---|
|
||||
| Planning tool | Navigator's experiment plan, shown live in the UI |
|
||||
| Planning tool | Navigator's experiment plan, shown live in the portal |
|
||||
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
|
||||
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
|
||||
| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` |
|
||||
| `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) |
|
||||
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
|
||||
|
||||
| Sub-agent | Job | Tools | Writes |
|
||||
@@ -390,6 +394,19 @@ them as follows:
|
||||
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
|
||||
the judge is visible in its scores.
|
||||
|
||||
**Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned
|
||||
skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice,
|
||||
quality controls and integration specifics are applied the same way every time:
|
||||
|
||||
| Family | Skills (first set) | Used by |
|
||||
|---|---|---|
|
||||
| Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright |
|
||||
| Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker |
|
||||
| Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright |
|
||||
|
||||
Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a
|
||||
skill change goes through a pull request like an agent change.
|
||||
|
||||
**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and
|
||||
Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You
|
||||
review, merge, deploy and record the ADR, as today.
|
||||
@@ -407,6 +424,27 @@ review, merge, deploy and record the ADR, as today.
|
||||
`outcome-pattern/`. Solution architecture and Discovery are never linked.
|
||||
- A model change for an agent under test is proposed, never made.
|
||||
|
||||
## 5a. The nightly cycle (amended 2026-11-12)
|
||||
|
||||
The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull
|
||||
request against the agents and workflows automatically, for review the next morning.
|
||||
|
||||
| Step | Who | Result |
|
||||
|---|---|---|
|
||||
| Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) |
|
||||
| Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it |
|
||||
| Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget |
|
||||
| Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence |
|
||||
| Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive |
|
||||
| Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs |
|
||||
| Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated |
|
||||
| Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more |
|
||||
|
||||
Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`,
|
||||
`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a
|
||||
gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
|
||||
pull requests.
|
||||
|
||||
## 6. Business value: rating quality, not volume
|
||||
|
||||
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
|
||||
@@ -437,13 +475,14 @@ repository exists.
|
||||
|
||||
| # | Phase | Delivers | Verified by |
|
||||
|---|---|---|---|
|
||||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||||
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||||
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||||
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
|
||||
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
|
||||
| 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
|
||||
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
|
||||
| 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
|
||||
| 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person |
|
||||
|
||||
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
|
||||
the experimentation loop pays back.
|
||||
@@ -482,13 +521,18 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
|
||||
| Hosting | The crucible cluster |
|
||||
| Shadow agents | The `shakedown` namespace |
|
||||
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
|
||||
| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API |
|
||||
| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) |
|
||||
| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) |
|
||||
| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it |
|
||||
|
||||
**Still open:**
|
||||
|
||||
1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown`
|
||||
OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS
|
||||
certificate under `kyndemo.live`.
|
||||
1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens
|
||||
Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates
|
||||
under `kyndemo.live` for the API (and the portal once it is shared).
|
||||
5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run
|
||||
unattended, and where the skills library lives (here, or shared with cp1-agents).
|
||||
2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`,
|
||||
held in a Secret, with rights limited to opening pull requests.
|
||||
3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's
|
||||
|
||||
Reference in New Issue
Block a user