diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index cf1befa..0a073e1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -6,7 +6,8 @@ enters service. This app does the same for every workflow before its outputs rea | | | |---|---| -| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) | +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) | +| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | | Runs on | The `crucible-kab-dev-aks` cluster | @@ -40,7 +41,7 @@ without anyone pointing it at them. ## 2. What Shakedown does -Shakedown does four things, each usable through the UI and by the deep agent (§5): +Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5): 1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, @@ -81,7 +82,8 @@ experiment's record links to its runs by execution id. ``` cp1-shakedown/ ├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/) -├── app/ # FastAPI API + UI +├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005) +├── skills//SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010) ├── bosun/ # the deep agent │ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend │ ├── subagents/ # one module per sub-agent: prompt + tool list @@ -313,18 +315,18 @@ Shakedown checks each link four ways: **Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its links declared. CP4 Epics are next after the landing zone. -**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of +**In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the Epic and Feature acceptance criteria that have no check yet. ## 4. Architecture on the cluster ``` - you (browser) ── Keycloak sign-in (realm kab) ──┐ - ▼ + you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐ + bearer token (you) ▼ ┌──────────────────── namespace: shakedown ──────────────────────────────────────┐ -│ shakedown-app (FastAPI + UI, one replica) │ -│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │ +│ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │ +│ ├─ acts as you: the portal's token, validated per request; never stored │ │ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │ │ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │ │ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │ @@ -334,14 +336,16 @@ Epic and Feature acceptance criteria that have no check yet. │ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents` ``` -- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`). - The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the - ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. - This replaces `kab_token.py` for everything Shakedown does. +- **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the + session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates + that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores + it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity + of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown + does. - **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`) replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access. -- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no - port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team. +- **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the + portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views. - **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents in `agents`. They reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply @@ -372,10 +376,10 @@ them as follows: | Deep Agents feature | Used for | |---|---| -| Planning tool | Navigator's experiment plan, shown live in the UI | +| Planning tool | Navigator's experiment plan, shown live in the portal | | File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied | | Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small | -| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` | +| `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) | | LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards | | Sub-agent | Job | Tools | Writes | @@ -390,6 +394,19 @@ them as follows: Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores. +**Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned +skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice, +quality controls and integration specifics are applied the same way every time: + +| Family | Skills (first set) | Used by | +|---|---|---| +| Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright | +| Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker | +| Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright | + +Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a +skill change goes through a pull request like an agent change. + **How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today. @@ -407,6 +424,27 @@ review, merge, deploy and record the ADR, as today. `outcome-pattern/`. Solution architecture and Discovery are never linked. - A model change for an agent under test is proposed, never made. +## 5a. The nightly cycle (amended 2026-11-12) + +The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull +request against the agents and workflows automatically, for review the next morning. + +| Step | Who | Result | +|---|---|---| +| Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) | +| Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it | +| Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget | +| Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence | +| Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive | +| Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs | +| Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated | +| Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more | + +Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`, +`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a +gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its +pull requests. + ## 6. Business value: rating quality, not volume Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. @@ -437,13 +475,14 @@ repository exists. | # | Phase | Delivers | Verified by | |---|---|---|---| -| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | -| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | +| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | +| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | | 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | | 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | | 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence | | 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code | | 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` | +| 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person | Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back. @@ -482,13 +521,18 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e | Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale | | Hosting | The crucible cluster | | Shadow agents | The `shakedown` namespace | -| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) | +| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API | +| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) | +| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) | +| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it | **Still open:** -1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown` - OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS - certificate under `kyndemo.live`. +1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens + Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates + under `kyndemo.live` for the API (and the portal once it is shared). +5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run + unattended, and where the skills library lives (here, or shared with cp1-agents). 2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`, held in a Secret, with rights limited to opening pull requests. 3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's