Plan: one UI in the KAB Docs Portal, the nightly cycle, skills

Amended 2026-11-12 with the owner: Shakedown has no UI of its own; the KAB
Docs Portal is the one UI and Shakedown its visualisation layer for workflow
case outputs, so Shakedown serves an API that accepts the portal's realm-kab
tokens. Adds the nightly cycle (5a: harvest, generate, execute, qualify,
prove, raise pull requests automatically for next-morning review, within a
standing daily budget; never merge or deploy), skills Bosun's sub-agents work
through (5), phase 7, and the engagement-id rule from cp1-agents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-13 10:28:36 +00:00
parent 98e3ec69b4
commit 0890bc5c31

View File

@@ -6,7 +6,8 @@ enters service. This app does the same for every workflow before its outputs rea
| | |
|---|---|
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) |
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
| Status | Draft. Nothing is built yet |
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
| Runs on | The `crucible-kab-dev-aks` cluster |
@@ -40,7 +41,7 @@ without anyone pointing it at them.
## 2. What Shakedown does
Shakedown does four things, each usable through the UI and by the deep agent (§5):
Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5):
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
@@ -81,7 +82,8 @@ experiment's record links to its runs by execution id.
```
cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/ # FastAPI API + UI
├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005)
├── skills/<skill>/SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010)
├── bosun/ # the deep agent
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│ ├── subagents/ # one module per sub-agent: prompt + tool list
@@ -313,18 +315,18 @@ Shakedown checks each link four ways:
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
links declared. CP4 Epics are next after the landing zone.
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
**In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
Epic and Feature acceptance criteria that have no check yet.
## 4. Architecture on the cluster
```
you (browser) ── Keycloak sign-in (realm kab) ──┐
▼
you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐
bearer token (you) ▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│ shakedown-app (FastAPI + UI, one replica) │
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │
│ ├─ acts as you: the portal's token, validated per request; never stored │
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
@@ -334,14 +336,16 @@ Epic and Feature acceptance criteria that have no check yet.
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
```
- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`).
The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the
ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you.
This replaces `kab_token.py` for everything Shakedown does.
- **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the
session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates
that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores
it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity
of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown
does.
- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`)
replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access.
- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no
port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
- **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the
portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views.
- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents
in `agents`. They
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
@@ -372,10 +376,10 @@ them as follows:
| Deep Agents feature | Used for |
|---|---|
| Planning tool | Navigator's experiment plan, shown live in the UI |
| Planning tool | Navigator's experiment plan, shown live in the portal |
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` |
| `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) |
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
| Sub-agent | Job | Tools | Writes |
@@ -390,6 +394,19 @@ them as follows:
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
the judge is visible in its scores.
**Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned
skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice,
quality controls and integration specifics are applied the same way every time:
| Family | Skills (first set) | Used by |
|---|---|---|
| Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright |
| Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker |
| Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright |
Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a
skill change goes through a pull request like an agent change.
**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and
Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You
review, merge, deploy and record the ADR, as today.
@@ -407,6 +424,27 @@ review, merge, deploy and record the ADR, as today.
`outcome-pattern/`. Solution architecture and Discovery are never linked.
- A model change for an agent under test is proposed, never made.
## 5a. The nightly cycle (amended 2026-11-12)
The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull
request against the agents and workflows automatically, for review the next morning.
| Step | Who | Result |
|---|---|---|
| Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) |
| Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it |
| Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget |
| Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence |
| Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive |
| Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs |
| Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated |
| Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more |
Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`,
`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a
gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
pull requests.
## 6. Business value: rating quality, not volume
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
@@ -437,13 +475,14 @@ repository exists.
| # | Phase | Delivers | Verified by |
|---|---|---|---|
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
| 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
| 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
| 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person |
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
the experimentation loop pays back.
@@ -482,13 +521,18 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
| Hosting | The crucible cluster |
| Shadow agents | The `shakedown` namespace |
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API |
| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) |
| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) |
| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it |
**Still open:**
1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown`
OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS
certificate under `kyndemo.live`.
1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens
Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates
under `kyndemo.live` for the API (and the portal once it is shared).
5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run
unattended, and where the skills library lives (here, or shared with cp1-agents).
2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`,
held in a Secret, with rights limited to opening pull requests.
3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's