From 0890bc5c3170fdd0e45a4c8091a4de005bc9db36 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Fri, 13 Nov 2026 10:28:36 +0000 Subject: [PATCH 1/4] Plan: one UI in the KAB Docs Portal, the nightly cycle, skills Amended 2026-11-12 with the owner: Shakedown has no UI of its own; the KAB Docs Portal is the one UI and Shakedown its visualisation layer for workflow case outputs, so Shakedown serves an API that accepts the portal's realm-kab tokens. Adds the nightly cycle (5a: harvest, generate, execute, qualify, prove, raise pull requests automatically for next-morning review, within a standing daily budget; never merge or deploy), skills Bosun's sub-agents work through (5), phase 7, and the engagement-id rule from cp1-agents. Co-Authored-By: Claude Opus 5.5 --- docs/implementation-plan.md | 88 +++++++++++++++++++++++++++---------- 1 file changed, 66 insertions(+), 22 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index cf1befa..0a073e1 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -6,7 +6,8 @@ enters service. This app does the same for every workflow before its outputs rea | | | |---|---| -| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) | +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) | +| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | | Runs on | The `crucible-kab-dev-aks` cluster | @@ -40,7 +41,7 @@ without anyone pointing it at them. ## 2. What Shakedown does -Shakedown does four things, each usable through the UI and by the deep agent (§5): +Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5): 1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own. - *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`, @@ -81,7 +82,8 @@ experiment's record links to its runs by execution id. ``` cp1-shakedown/ ├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/) -├── app/ # FastAPI API + UI +├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005) +├── skills//SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010) ├── bosun/ # the deep agent │ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend │ ├── subagents/ # one module per sub-agent: prompt + tool list @@ -313,18 +315,18 @@ Shakedown checks each link four ways: **Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its links declared. CP4 Epics are next after the landing zone. -**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of +**In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the Epic and Feature acceptance criteria that have no check yet. ## 4. Architecture on the cluster ``` - you (browser) ── Keycloak sign-in (realm kab) ──┐ - ▼ + you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐ + bearer token (you) ▼ ┌──────────────────── namespace: shakedown ──────────────────────────────────────┐ -│ shakedown-app (FastAPI + UI, one replica) │ -│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │ +│ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │ +│ ├─ acts as you: the portal's token, validated per request; never stored │ │ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │ │ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │ │ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │ @@ -334,14 +336,16 @@ Epic and Feature acceptance criteria that have no check yet. │ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents` ``` -- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`). - The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the - ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you. - This replaces `kab_token.py` for everything Shakedown does. +- **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the + session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates + that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores + it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity + of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown + does. - **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`) replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access. -- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no - port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team. +- **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the + portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views. - **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents in `agents`. They reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply @@ -372,10 +376,10 @@ them as follows: | Deep Agents feature | Used for | |---|---| -| Planning tool | Navigator's experiment plan, shown live in the UI | +| Planning tool | Navigator's experiment plan, shown live in the portal | | File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied | | Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small | -| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` | +| `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) | | LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards | | Sub-agent | Job | Tools | Writes | @@ -390,6 +394,19 @@ them as follows: Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to the judge is visible in its scores. +**Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned +skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice, +quality controls and integration specifics are applied the same way every time: + +| Family | Skills (first set) | Used by | +|---|---|---| +| Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright | +| Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker | +| Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright | + +Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a +skill change goes through a pull request like an agent change. + **How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You review, merge, deploy and record the ADR, as today. @@ -407,6 +424,27 @@ review, merge, deploy and record the ADR, as today. `outcome-pattern/`. Solution architecture and Discovery are never linked. - A model change for an agent under test is proposed, never made. +## 5a. The nightly cycle (amended 2026-11-12) + +The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull +request against the agents and workflows automatically, for review the next morning. + +| Step | Who | Result | +|---|---|---| +| Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) | +| Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it | +| Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget | +| Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence | +| Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive | +| Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs | +| Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated | +| Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more | + +Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`, +`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a +gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its +pull requests. + ## 6. Business value: rating quality, not volume Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. @@ -437,13 +475,14 @@ repository exists. | # | Phase | Delivers | Verified by | |---|---|---|---| -| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | -| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | +| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets | +| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught | | 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 | | 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread | | 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence | | 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code | | 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` | +| 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person | Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where the experimentation loop pays back. @@ -482,13 +521,18 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e | Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale | | Hosting | The crucible cluster | | Shadow agents | The `shakedown` namespace | -| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) | +| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API | +| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) | +| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) | +| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it | **Still open:** -1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown` - OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS - certificate under `kyndemo.live`. +1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens + Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates + under `kyndemo.live` for the API (and the portal once it is shared). +5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run + unattended, and where the skills library lives (here, or shared with cp1-agents). 2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`, held in a Secret, with rights limited to opening pull requests. 3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's -- 2.49.1 From 07060dbb372c895280af165f1ebab9d992a950c3 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Sun, 15 Nov 2026 21:03:50 +0000 Subject: [PATCH 2/4] =?UTF-8?q?Plan=20=C2=A75b:=20end-to-end=20use=20cases?= =?UTF-8?q?=20that=20fix=20what=20fails=20(owner,=202026-11-15)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Name a customer: the system generates its use case and runs Discovery, Assessment and Strategy remotely. It decides gates only in its own projects and only on a critic pass, fixes failures on an agentic/- branch, builds and deploys over the shared agents (one branch at a time, rolled back if the PR is closed), and raises one PR to main per repository. AC8, §4, §5, §5a, §9 and §10 amended to match. Co-Authored-By: Claude Opus 5.5 --- docs/implementation-plan.md | 73 +++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 7 deletions(-) diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 0a073e1..3052ecc 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea | | | |---|---| -| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9) | +| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) | | User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | | Status | Draft. Nothing is built yet | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | @@ -301,7 +301,7 @@ Shakedown checks each link four ways: | AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't | | AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` | | AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record | -| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output | +| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output. *Amended 2026-11-15:* in Shakedown's own use-case projects, the system may decide a gate under the rule in §5b, and the ledger records that decision as the system's, never as a person's | **Rules for linked workflows:** - A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is @@ -350,7 +350,8 @@ Epic and Feature acceptance criteria that have no check yet. in `agents`. They reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents' - rule that each cluster change is approved. + rule that each cluster change is approved. *Amended 2026-11-15:* end-to-end use cases (§5b) are the exception. + There the system builds agents from its branch and deploys them over the shared agents in `agents`. - **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM, which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path from LangChain to the Azure-hosted Claude endpoint before anything else is built on it. @@ -412,8 +413,12 @@ Surveyor scores them. If the change helps without regressing other checks, Shipw review, merge, deploy and record the ADR, as today. **Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:** -- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. -- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. +- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. *Amended + 2026-11-15:* in an end-to-end use case (§5b), Lookout's `decide_gate` tool continues a gate only under §5b's rule, + and only in Shakedown's own use-case projects. Any other project's gates stay a person's. +- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. *Amended + 2026-11-15:* the one exception is §5b's branch deploy: images built from the use case's branch, rolled out to the + cp1-agents components in `agents`, under a lock and with the previous tags recorded. - No changes to ADO work items. Bosun's ADO tools are read-only. - No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read. - Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as @@ -442,9 +447,61 @@ request against the agents and workflows automatically, for review the next morn Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`, `start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a -gate. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its +gate, except within an end-to-end use case (§5b), whose own rules then apply. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its pull requests. +## 5b. End-to-end use cases that fix what fails (amended 2026-11-15) + +The owner's direction (2026-11-15): name a customer, and the system generates a use case for it and runs Discovery, +Assessment and Strategy end to end, remotely, on demand or on a schedule. When a workflow fails, the system fixes the +cause on a feature branch it owns, builds the agents from that branch on the platform, and carries on. When the use +case finishes, it raises a pull request to `main` for review. This is what the manual Cymbal Bank case of 2026-11-15 +took a person a day to do (§1). + +**Owner decisions (2026-11-15):** + +| Decision | Choice | +|---|---| +| Gates | The system decides them, **only** in its own use-case projects. It continues a gate when the run completed, the stage's critic verdict is `pass` and the code checks hold. Otherwise it stops the gate and opens a fix. It never continues a gate on a critic `fail` | +| Agents built from the branch | Built on the platform and deployed **over the shared agents** in `agents`, not as shadow agents | +| Branches | **One per use case**, `agentic/-`, owned by the system's gitea identity; one pull request to `main` per repository when the use case ends | +| Home | Shakedown (Bosun): Chartmaker, Lookout, Surveyor and Shipwright gain the steps below | + +**How a use case runs:** + +| Step | Who | What | +|---|---|---| +| 1. Generate | Chartmaker | From the customer's name and brief: the F01 context, F02 rate card and F03 run context, one `engagement_id` (`ENG-`, F01 ADR 0022), and the Discovery source. The source is either one that exists (a live cloud connection, or an estate the sample server already serves) or a new estate added to `alz-discovery-sample` as data on the branch (F01 ADR 0025). Expectations are written before the first run. The case goes to `cp1-shakedown` on the branch | +| 2. Set up | Lookout | A KAB project for the customer in CP1, the three workflows associated with it, and the case registered in the portal's `bindings.yaml` (on a branch of the portal) | +| 3. Run | Lookout | F01, then F02 from the approved F01 package, then F03 from the approved F02 package; each started from its context file (`--context`) | +| 4. Decide each gate | Lookout | By the gate rule above. Each decision carries a note naming the system, the stage's critic verdict and the run that was checked | +| 5. Fix on failure | Surveyor, then Shipwright | Surveyor sorts the failure into one kind: a step refused its input (schema); the critic misread a rule; an agent broke a rule (wrong or missing citations); a data or server bug; the platform (auth, quota, an agent not reachable). Shipwright changes the prompt, schema, hook or server data on the branch, with a release note and version bump, and runs `check_agents.py` and the agent's tests. It commits with the failing run's id and the reason | +| 6. Build and deploy | Lookout | Builds the changed agents' images from the branch, in parallel, and rolls them out over the shared agents. Then it reruns the stage that failed, not the whole chain: an approved upstream stage is kept | +| 7. Prove | Surveyor | A fix holds only when the rerun passes. Because models vary from run to run, a fix to a critic rule needs two passing runs | +| 8. Finish | Shipwright | One pull request to `main` per repository (`cp1-agents`, `cp1-shakedown`, the portal if bindings changed). It lists every failure, its kind, the commit that fixed it, and the run that proved it; the versions now deployed; and anything left unresolved | + +**Limits and safety:** +- **Three fix attempts per stage,** and the daily budget (§5a). When either runs out, the use case stops and the pull + request is raised with the failure marked unresolved. +- **One branch deploys at a time.** Shared agents run one branch's code, so a lock in the ledger allows one use + case to deploy at a time. Others wait or run without deploying. +- **The shared agents run branch code until the pull request is decided.** The portal shows which agents are + deployed from which branch. If the pull request is closed unmerged, Lookout rolls those agents back to the image + tags it recorded before the first deploy. Merging puts `main` back in line with what's deployed. +- **What stays a person's:** merging a pull request; anything in another project, including gates; kamp-demo + (read-only); platform-admin actions (claims, registrations owned by another tenant); a model change for an agent; + accepting an ADR. +- **What the system never changes:** secrets (it reads none); `main`; ADO work items. + +**What it needs from the platform (blocks the build):** +1. A **service identity** in realm `kab` (client credentials) with CP1 rights to create projects, start runs and + decide gates in its own projects. Personal tokens can't run unattended. +2. A **gitea bot** that can push `agentic/*` branches and open pull requests on `cp1-agents`, `cp1-shakedown` and + the portal, and nothing more. +3. **Cluster rights** for Shakedown's ServiceAccount: push to `cruciblecoredevacr.azurecr.io/kab/cp1/*`, and + `patch` on the Deployments of cp1-agents components in `agents` (image only). No `secrets` access, nothing else. +4. **In-cluster image builds** (buildah or kaniko Jobs), because there is no laptop in the loop. + ## 6. Business value: rating quality, not volume Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it. @@ -524,6 +581,7 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e | UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API | | Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) | | Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) | +| End-to-end use cases (2026-11-15) | Generate a customer's use case and run F01 → F02 → F03 remotely; the system decides gates in its own projects on a critic pass, fixes failures on its own `agentic/-` branch, builds and deploys over the shared agents, and raises one pull request to `main` at the end (§5b) | | Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it | **Still open:** @@ -542,7 +600,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e ## 10. Out of scope -- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests. +- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests, except as + §5b allows for end-to-end use cases in Shakedown's own projects. Merging stays a person's in every case. - Writing to ADO work items. - Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else. - Moving the existing cases out of cp1-agents. -- 2.49.1 From 099f9a87e80fdf965c7b1bab63a9e2ea249d73c9 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Mon, 16 Nov 2026 05:08:31 +0000 Subject: [PATCH 3/4] Runner: run a voyage remotely as the kab-automation identity - shakedown_lib: runs F01 -> F02 -> F03 for one engagement on KAB with no personal token. Signs in as the kab-automation client (client credentials, provisioned by Stellwerk from crucible-platform-deploy) and renews the token itself. - Each stage checks its workflow id against the registered sub-agent name, builds inputs from the case and upstream stages, starts the run, follows it and judges it with cp1-agents evaluate_case.py. - Gates (plan 5b): in config/runner.yaml gate_projects only, continue on a critic pass with no failed step and a note saying the system decided. Never on a fail. Elsewhere, wait for a person and record who decided. gate_projects is empty for now. - Ledger per voyage run, with --resume; nothing secret is written. - Cymbal voyage ENG-CYMBAL-001; workflow registry; Dockerfile and an unapplied Job template; 15 tests against a fake KAB. Co-Authored-By: Claude Opus 5.5 --- .gitignore | 5 + README.md | 72 ++++ .../voyages/ENG-CYMBAL-001.yaml | 40 ++ config/runner.yaml | 13 + deploy/Dockerfile | 16 + deploy/runner-job.yaml | 52 +++ docs/implementation-plan.md | 2 +- requirements.txt | 2 + shakedown_lib/__init__.py | 1 + shakedown_lib/__main__.py | 76 ++++ shakedown_lib/auth.py | 80 ++++ shakedown_lib/cp1_agents.py | 59 +++ shakedown_lib/gate.py | 97 +++++ shakedown_lib/kab.py | 104 +++++ shakedown_lib/ledger.py | 78 ++++ shakedown_lib/runner.py | 389 ++++++++++++++++++ shakedown_lib/voyage.py | 116 ++++++ tests/__init__.py | 0 tests/test_runner.py | 320 ++++++++++++++ workflows/alz-f01-discovery.yaml | 8 + workflows/alz-f02-assessment.yaml | 8 + workflows/alz-f03-strategy.yaml | 9 + 22 files changed, 1546 insertions(+), 1 deletion(-) create mode 100644 .gitignore create mode 100644 README.md create mode 100644 capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml create mode 100644 config/runner.yaml create mode 100644 deploy/Dockerfile create mode 100644 deploy/runner-job.yaml create mode 100644 requirements.txt create mode 100644 shakedown_lib/__init__.py create mode 100644 shakedown_lib/__main__.py create mode 100644 shakedown_lib/auth.py create mode 100644 shakedown_lib/cp1_agents.py create mode 100644 shakedown_lib/gate.py create mode 100644 shakedown_lib/kab.py create mode 100644 shakedown_lib/ledger.py create mode 100644 shakedown_lib/runner.py create mode 100644 shakedown_lib/voyage.py create mode 100644 tests/__init__.py create mode 100644 tests/test_runner.py create mode 100644 workflows/alz-f01-discovery.yaml create mode 100644 workflows/alz-f02-assessment.yaml create mode 100644 workflows/alz-f03-strategy.yaml diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a583c7a --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +ledger/ +.env +__pycache__/ +*.pyc +build/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..f5e6d81 --- /dev/null +++ b/README.md @@ -0,0 +1,72 @@ +# Shakedown + +Runs, rates and experiments on KAB workflows. The plan is [docs/implementation-plan.md](docs/implementation-plan.md). +What is built so far is the **runner**, which runs a voyage remotely with no person's token: F01 Discovery → F02 +Assessment → F03 Strategy for one engagement. + +## The runner + +```sh +export KAB_CLIENT_SECRET=... # the kab-automation client's secret; never in a file +python3 -m shakedown_lib token-check +python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run +python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml +python3 -m shakedown_lib run --resume ledger/ENG-CYMBAL-001/ # pick up where it stopped +python3 -m unittest discover -s tests -t . # tests, against a fake KAB +``` + +**Identity.** The runner signs in as the confidential client `kab-automation` of realm `kab`, using client +credentials. Its tokens carry the automation user's email, which KAB makes an Editor. The Stellwerk orchestrator +provisions the client and its secret: crucible-platform-deploy, `kab-cluster/dev` component 85. The secret is held in +Key Vault `crucible-kab-dev-kv` as `kab-automation-client-secret`. A token is renewed a minute before it expires, so a +run that lasts hours needs nobody. + +**What one stage does:** +1. **Check the workflow.** The stage's workflow id must still name its registered workflow (`workflows/*.yaml`, sub-agent pattern). Ids change on every re-import. +2. **Build the inputs.** They come from the case's context file and the voyage. Upstream execution ids, approval + references and package artifact ids (`_step_`) come from earlier stages. Each context + adapter delivers the file differently: + - `upload` (F01): uploaded as the run's file; + - `document` (F03): sent as text in one run input; + - `none` (F02): inputs come from the voyage. +3. **Check and start.** Before starting, it checks the required inputs, the engagement id (one per chain) and the + customer block, using cp1-agents' own helpers. +4. **Follow the run.** At a review point: + - in a project listed in `config/runner.yaml` `gate_projects`, the runner continues **only** when the critic's + verdict is `pass` and no step has failed. Its note says the system decided, not a person. Otherwise the pause is + left as it is and the voyage stops for a fix; + - in any other project, it waits for a person to decide in KAB and records who did. +5. **Finish.** It pulls the outputs, finds the package and judges the run against its case with cp1-agents + `scripts/evaluate_case.py`. A failed run, a missing package or a failed critical check stops the voyage. + +**Ledger.** Each voyage run gets its own folder, `ledger///` (or `$SHAKEDOWN_LEDGER`). It holds: +- `state.json`; +- for each stage: its inputs (uploads by name only), the context as sent, the status, the steps and the evaluation. + +No token or secret is written there. + +**Exit status:** + +| Code | Meaning | +|---|---| +| 0 | Done | +| 2 | Configuration error, or KAB refused | +| 3 | Stopped for a fix | +| 4 | Nobody decided a review point in time | +| 5 | A stage ran too long | + +**cp1-agents** is read from a clone beside this repository, or from `$CP1_AGENTS_DIR`, and its commit is recorded +with every run. Nothing in cp1-agents is changed. + +**On the cluster:** `deploy/Dockerfile` and `deploy/runner-job.yaml`, a Job template with placeholders. Nothing is +applied without approval. + +## Not built yet + +These come from the plan: +- Bosun (the deep agent); +- fixes on `agentic/*` branches; +- deploying from a branch; +- experiments and variations; +- the portal views; +- the nightly cycle. diff --git a/capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml b/capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml new file mode 100644 index 0000000..a5fc1db --- /dev/null +++ b/capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml @@ -0,0 +1,40 @@ +# Cymbal Bank (fictional; estate served by alz-discovery-sample): Discovery -> Assessment -> Strategy. +# The cases stay in cp1-agents and are read in place. Upstream ids are filled from the earlier stages' runs. +# The workflow ids are unverified: each is checked against its registry name (sub-agent pattern) before a stage +# starts, so a stale or individual-pattern id is refused. `--dry-run` checks them all without starting anything. +voyage: ENG-CYMBAL-001 +capability: cp1-agentic-landing-zone +engagement_id: ENG-CYMBAL-001 +project: 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 # KAB project "Cymbal Bank" (kab-dev) + +stages: + - key: f01 + workflow: alz-f01-discovery + workflow_id: 6aea4d516bc9323ca8d4bd12 + # Run exec_e60e23f0a4ac passed its case and was approved on 2026-11-15, so it is adopted, not run again. + # Remove these two lines to run Discovery afresh from the context below. + use_execution: exec_e60e23f0a4ac + approval_ref: "Continued by a person in KAB on 2026-11-15 (Cymbal Discovery run approved)" + context: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/context.yaml" + case: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/case.yaml" + + - key: f02 + workflow: alz-f02-assessment + workflow_id: 6aecac756bc9323ca8d4be1c # from the case README; checked against the registry names + inputs: + discovery_execution_id: {from: f01, value: execution_id} + discovery_approval_ref: {from: f01, value: approval_ref} + artifact_ids: {from: f01, value: package_artifact_ids} + documents: + rate_card_document: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/rate-card.json" + case: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/case.yaml" + + - key: f03 + workflow: alz-f03-strategy + workflow_id: wf_0f80f01c-e423-4553-9ec0-afa21b3bbb52 # from the docs portal registry; checked by name + context: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/run-context.yaml" + inputs: + assessment_execution_id: {from: f02, value: execution_id} + assessment_approval_ref: {from: f02, value: approval_ref} + artifact_ids: {from: f02, value: package_artifact_ids} + case: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/case.yaml" diff --git a/config/runner.yaml b/config/runner.yaml new file mode 100644 index 0000000..d2bae79 --- /dev/null +++ b/config/runner.yaml @@ -0,0 +1,13 @@ +# Shakedown runner settings. Committed, so every change is reviewed; the environment can't widen them. + +# The KAB projects whose review points the runner may decide itself (plan §5b): only Shakedown's own use-case +# projects. In any other project it waits for a person. It continues a gate only on a critic `pass` with no failed +# step (shakedown_lib/gate.py), never on a `fail`, and its note says the system decided. +# +# Empty until the owner names the projects. For the Cymbal voyage that would be the "Cymbal Bank" project: +# - 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 +gate_projects: [] + +poll_seconds: 30 +stage_timeout_minutes: 360 # one stage, start to package +person_wait_hours: 24 # the workflows' own review timeout (hitl.timeout_hours) diff --git a/deploy/Dockerfile b/deploy/Dockerfile new file mode 100644 index 0000000..d70b992 --- /dev/null +++ b/deploy/Dockerfile @@ -0,0 +1,16 @@ +# The Shakedown runner. Build from the repository root: +# buildah bud -f deploy/Dockerfile -t cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/runner: . +ARG BASE=python:3.12-slim +FROM ${BASE} +RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates && rm -rf /var/lib/apt/lists/* +WORKDIR /app +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt +COPY shakedown_lib/ shakedown_lib/ +COPY config/ config/ +COPY workflows/ workflows/ +COPY capabilities/ capabilities/ +RUN useradd -u 10001 -m shakedown && mkdir -p /ledger && chown shakedown /ledger +USER 10001 +ENV CP1_AGENTS_DIR=/cp1-agents SHAKEDOWN_LEDGER=/ledger PYTHONUNBUFFERED=1 +ENTRYPOINT ["python3", "-m", "shakedown_lib"] diff --git a/deploy/runner-job.yaml b/deploy/runner-job.yaml new file mode 100644 index 0000000..0fa67e7 --- /dev/null +++ b/deploy/runner-job.yaml @@ -0,0 +1,52 @@ +# One voyage run as a Kubernetes Job in namespace `shakedown`. A template: placeholders are filled at render time, +# and nothing here is applied without the owner's approval. +# +# Provisioned outside this repository by the Stellwerk orchestrator (crucible-platform-deploy, kab-cluster/dev): +# - the `shakedown` namespace and the PVC `shakedown-ledger`; +# - Secret `shakedown-kab-automation`, key `client-secret`: the kab-automation client's secret, from Key Vault +# crucible-kab-dev-kv `kab-automation-client-secret`. The runner reads it from the environment only; +# - Secret `shakedown-gitea-read`, key `token`: read access to agents/cp1-agents, for the pinned clone. +apiVersion: batch/v1 +kind: Job +metadata: + generateName: shakedown-voyage- + namespace: shakedown + labels: {app.kubernetes.io/name: shakedown-runner, app.kubernetes.io/part-of: cp1-shakedown} +spec: + backoffLimit: 0 # a failed run is resumed on purpose (--resume), never retried blind + activeDeadlineSeconds: 172800 # two days: three stages plus waiting for people at review points + template: + metadata: + labels: {app.kubernetes.io/name: shakedown-runner} + spec: + restartPolicy: Never + securityContext: {runAsNonRoot: true, runAsUser: 10001, fsGroup: 10001} + initContainers: + - name: clone-cp1-agents + image: + command: ["sh", "-c"] + args: + - >- + git -c http.extraHeader="Authorization: token $GITEA_TOKEN" clone --quiet + https://gitea.kyndemo.live/agents/cp1-agents.git /cp1-agents && + git -C /cp1-agents checkout --quiet + env: + - name: GITEA_TOKEN + valueFrom: {secretKeyRef: {name: shakedown-gitea-read, key: token}} + volumeMounts: [{name: cp1-agents, mountPath: /cp1-agents}] + containers: + - name: runner + image: + args: ["run", ""] + env: + - name: KAB_CLIENT_SECRET + valueFrom: {secretKeyRef: {name: shakedown-kab-automation, key: client-secret}} + resources: + requests: {cpu: 50m, memory: 128Mi} + limits: {cpu: 500m, memory: 512Mi} + volumeMounts: + - {name: cp1-agents, mountPath: /cp1-agents, readOnly: true} + - {name: ledger, mountPath: /ledger} + volumes: + - {name: cp1-agents, emptyDir: {}} + - {name: ledger, persistentVolumeClaim: {claimName: shakedown-ledger}} diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index 3052ecc..354d7c3 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -8,7 +8,7 @@ enters service. This app does the same for every workflow before its outputs rea |---|---| | Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) | | User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API | -| Status | Draft. Nothing is built yet | +| Status | Draft. Built so far (2026-11-16): the voyage **runner** (README), which runs F01 → F02 → F03 remotely as the `kab-automation` identity provisioned by Stellwerk, with the §5b gate rule in code | | Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` | | Runs on | The `crucible-kab-dev-aks` cluster | | Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource | diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..3548cbf --- /dev/null +++ b/requirements.txt @@ -0,0 +1,2 @@ +pyyaml>=6 +jsonschema>=4 # cp1-agents scripts/check_data_contract.py, used by evaluate_case.py diff --git a/shakedown_lib/__init__.py b/shakedown_lib/__init__.py new file mode 100644 index 0000000..dfb3449 --- /dev/null +++ b/shakedown_lib/__init__.py @@ -0,0 +1 @@ +"""Shakedown: runs, rates and experiments on KAB workflows (docs/implementation-plan.md). This package holds the runner.""" diff --git a/shakedown_lib/__main__.py b/shakedown_lib/__main__.py new file mode 100644 index 0000000..a3e2e8f --- /dev/null +++ b/shakedown_lib/__main__.py @@ -0,0 +1,76 @@ +"""Shakedown runner: run a voyage's workflows on KAB remotely, as the kab-automation identity. + + python3 -m shakedown_lib token-check + python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run + python3 -m shakedown_lib run [--only f02 f03] + python3 -m shakedown_lib run --resume ledger/ENG-CYMBAL-001/ + +Environment: KAB_CLIENT_SECRET (required), KAB_CLIENT_ID (kab-automation), KAB_TOKEN_URL, KAB_API, KAB_TENANT, +CP1_AGENTS_DIR (the cp1-agents clone, default ../cp1-agents), SHAKEDOWN_LEDGER (default ./ledger). +Which projects the runner may decide gates in is set only in config/runner.yaml. + +Exit status: 0 done, 2 configuration or KAB refused, 3 stopped for a fix, 4 nobody decided a review point in time, +5 a stage ran too long. +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import os +import sys +from pathlib import Path + +from .auth import AuthError, ClientCredentials +from .kab import CP1_TENANT, DEFAULT_API, Kab +from .ledger import Ledger, default_root +from .runner import EXIT_CONFIG, Runner, Settings +from .voyage import VoyageError, load_config, load_voyage + + +def _claims(token: str) -> dict: + try: + payload = token.split(".")[1] + return json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4))) + except (IndexError, ValueError): + return {} + + +def token_check() -> int: + tokens = ClientCredentials.from_env() + claims = _claims(tokens.token()) + shown = {k: claims.get(k) for k in ("iss", "azp", "email", "preferred_username", "exp")} + print(f"Token issued to {tokens.client_id}, valid for {tokens.expires_in()} s. Claims: {json.dumps(shown)}") + return 0 + + +def run(args) -> int: + voyage = load_voyage(Path(args.voyage)) + settings = Settings.from_config(load_config()) + kab = Kab(ClientCredentials.from_env(), api=os.getenv("KAB_API", DEFAULT_API), + tenant=os.getenv("KAB_TENANT", CP1_TENANT), project=voyage.project) + ledger = Ledger(Path(args.resume)) if args.resume else Ledger.new(default_root(), voyage.name) + return Runner(kab=kab, ledger=ledger, voyage=voyage, settings=settings).run(dry_run=args.dry_run, only=args.only) + + +def main(argv=None) -> int: + parser = argparse.ArgumentParser(prog="shakedown_lib", description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + commands = parser.add_subparsers(dest="command", required=True) + commands.add_parser("token-check", help="get a token as kab-automation and show who it names") + start = commands.add_parser("run", help="run a voyage") + start.add_argument("voyage") + start.add_argument("--dry-run", action="store_true", help="build and check every stage's inputs; start nothing") + start.add_argument("--resume", help="a ledger folder of an earlier run of this voyage") + start.add_argument("--only", nargs="+", help="run only these stages (earlier ones must be completed)") + args = parser.parse_args(argv) + try: + return token_check() if args.command == "token-check" else run(args) + except (AuthError, VoyageError, FileNotFoundError, RuntimeError) as error: + print(f"Refused: {error}", file=sys.stderr) + return EXIT_CONFIG + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/shakedown_lib/auth.py b/shakedown_lib/auth.py new file mode 100644 index 0000000..a7dab13 --- /dev/null +++ b/shakedown_lib/auth.py @@ -0,0 +1,80 @@ +"""KAB access tokens for unattended runs: the `kab-automation` client of realm `kab`, by client credentials. + +The client and its secret are provisioned by the Stellwerk orchestrator (crucible-platform-deploy, `kab-cluster/dev` +component 85; module `kab-identity`). The secret is in Key Vault `crucible-kab-dev-kv` as +`kab-automation-client-secret` and reaches the runner as the environment variable KAB_CLIENT_SECRET (a mounted +Secret on the cluster). Tokens carry the fixed email claim KAB knows the automation user by. + +The secret and every token stay in memory: never written to a file, the ledger or a log, and never printed. +""" + +from __future__ import annotations + +import json +import os +import time +import urllib.error +import urllib.parse +import urllib.request +from typing import Callable + +DEFAULT_TOKEN_URL = "https://id.kab-dev.kyndemo.live/realms/kab/protocol/openid-connect/token" +DEFAULT_CLIENT_ID = "kab-automation" +REFRESH_MARGIN_S = 60 + + +class AuthError(RuntimeError): + pass + + +def _post_form(url: str, form: dict, timeout: int = 30) -> dict: + body = urllib.parse.urlencode(form).encode() + request = urllib.request.Request(url, data=body, headers={"Content-Type": "application/x-www-form-urlencoded"}) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + return json.load(response) + except urllib.error.HTTPError as error: + try: + reason = json.loads(error.read() or b"{}").get("error_description") or "no reason given" + except ValueError: + reason = "no reason given" + raise AuthError(f"token request refused: HTTP {error.code}: {reason}") from None + except urllib.error.URLError as error: + raise AuthError(f"token endpoint unreachable: {error.reason}") from None + + +class ClientCredentials: + """A token source that renews itself a minute before the token expires.""" + + def __init__(self, client_id: str, secret: str, token_url: str = DEFAULT_TOKEN_URL, + post: Callable[[str, dict], dict] = _post_form, clock: Callable[[], float] = time.time) -> None: + if not secret: + raise AuthError("no client secret: set KAB_CLIENT_SECRET (the kab-automation client's secret)") + self.client_id = client_id + self._secret = secret + self.token_url = token_url + self._post = post + self._clock = clock + self._token: str | None = None + self._expires_at = 0.0 + + @classmethod + def from_env(cls) -> "ClientCredentials": + return cls(os.getenv("KAB_CLIENT_ID", DEFAULT_CLIENT_ID), os.getenv("KAB_CLIENT_SECRET", ""), + os.getenv("KAB_TOKEN_URL", DEFAULT_TOKEN_URL)) + + def __repr__(self) -> str: # never show the secret or the token + return f"ClientCredentials(client_id={self.client_id!r}, token_url={self.token_url!r})" + + def token(self, force: bool = False) -> str: + if force or self._token is None or self._clock() >= self._expires_at - REFRESH_MARGIN_S: + reply = self._post(self.token_url, {"grant_type": "client_credentials", "client_id": self.client_id, + "client_secret": self._secret}) + if not reply.get("access_token"): + raise AuthError("token reply has no access_token") + self._token = reply["access_token"] + self._expires_at = self._clock() + float(reply.get("expires_in") or 300) + return self._token + + def expires_in(self) -> int: + return max(0, int(self._expires_at - self._clock())) diff --git a/shakedown_lib/cp1_agents.py b/shakedown_lib/cp1_agents.py new file mode 100644 index 0000000..454baf4 --- /dev/null +++ b/shakedown_lib/cp1_agents.py @@ -0,0 +1,59 @@ +"""The cp1-agents scripts Shakedown reuses, imported by path from a clone (plan §3: no change to cp1-agents). + +`kab_run_workflow.py` gives the run-context helpers (load_context, new_ids, context_uploads, upload_type, +customer_problem, with_run_metadata, check_engagement) and `evaluate_case.py` judges a run against a case. The clone's +commit is recorded with every run, so a result can be reproduced. +""" + +from __future__ import annotations + +import importlib.util +import os +import subprocess +import sys +from pathlib import Path +from types import ModuleType + +DEFAULT_DIR = Path(__file__).resolve().parents[2] / "cp1-agents" + + +def clone_dir() -> Path: + path = Path(os.getenv("CP1_AGENTS_DIR") or DEFAULT_DIR) + if not (path / "scripts" / "kab_run_workflow.py").is_file(): + raise RuntimeError(f"no cp1-agents clone at {path}: set CP1_AGENTS_DIR") + return path + + +def _load(name: str) -> ModuleType: + scripts = clone_dir() / "scripts" + if str(scripts) not in sys.path: + sys.path.insert(0, str(scripts)) # evaluate_case imports check_data_contract beside it + spec = importlib.util.spec_from_file_location(f"cp1_agents_{name}", scripts / f"{name}.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +_cache: dict[str, ModuleType] = {} + + +def module(name: str) -> ModuleType: + if name not in _cache: + _cache[name] = _load(name) + return _cache[name] + + +def run_workflow() -> ModuleType: + return module("kab_run_workflow") + + +def evaluate_case() -> ModuleType: + return module("evaluate_case") + + +def commit() -> str | None: + try: + return subprocess.run(["git", "-C", str(clone_dir()), "rev-parse", "HEAD"], capture_output=True, text=True, + check=True).stdout.strip() + except (subprocess.CalledProcessError, FileNotFoundError, RuntimeError): + return None diff --git a/shakedown_lib/gate.py b/shakedown_lib/gate.py new file mode 100644 index 0000000..a241a90 --- /dev/null +++ b/shakedown_lib/gate.py @@ -0,0 +1,97 @@ +"""The gate rule for end-to-end use cases (plan §5b), decided in code from what the run recorded. + +At a review pause the system may continue only when, for the steps up to the pause: +- no step reported `failed` or `error`, and +- the stage's latest critic verdict is exactly `pass`. + +Anything else (a `fail`, an unparseable or missing verdict, a failed step) is never continued: the pause is left for a +person and the use case stops for a fix. Whether the system may decide at all depends on the project (the runner's +`gate_projects`), not on this module. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass +from typing import Any, Dict, List, Optional + +FAILED = {"failed", "error"} + + +def parsed(value: Any) -> Any: + if isinstance(value, str) and value[:1] in "{[": + try: + return json.loads(value) + except ValueError: + return value + return value + + +def unwrap(value: Any) -> Any: + """A step's own output, out of KAB's wrappers (`response`, `output`, `result`, `data`), JSON strings decoded.""" + value = parsed(value) + for _ in range(4): + if isinstance(value, dict) and "phase" not in value and "manifest" not in value: + inner = next((value[k] for k in ("response", "output", "result", "data") if k in value), None) + if inner is None: + break + value = parsed(inner) + else: + break + return value + + +def step_outputs(steps: Any) -> Dict[str, Any]: + """step id -> its output, from `/executions/{id}/steps` (a mapping by step id, or a list of step records).""" + body = steps.get("steps", steps) if isinstance(steps, dict) else steps + if isinstance(body, dict): + return {str(k): unwrap(v) for k, v in body.items() if k != "execution_id"} + out: Dict[str, Any] = {} + for item in body or []: + if isinstance(item, dict): + key = item.get("step_id") or item.get("id") + if key is not None: + out[str(key)] = unwrap(item.get("output", item.get("result", item))) + return out + + +def template_order(status: dict) -> List[str]: + return [str(s.get("id")) for s in ((status.get("template") or {}).get("steps") or [])] + + +def _verdict(value: Any) -> Optional[str]: + if isinstance(value, dict): + value = value.get("verdict") + return value if isinstance(value, str) else None + + +@dataclass +class Decision: + proceed: bool + reason: str + verdict: Optional[str] + verdict_step: Optional[str] + + +def decide(status: dict, steps: Any, paused_step: Optional[str]) -> Decision: + if str(status.get("status", "")).lower() in FAILED: + return Decision(False, f"run status is {status.get('status')}", None, None) + outputs = step_outputs(steps) + order = template_order(status) or list(outputs) + if paused_step in order: + order = order[: order.index(paused_step) + 1] + verdict, verdict_step = None, None + for step_id in order: + out = outputs.get(step_id) + if not isinstance(out, dict): + continue + if str(out.get("status", "")).lower() in FAILED: + return Decision(False, f"step {step_id} reported {out.get('status')}", verdict, verdict_step) + found = _verdict(out.get("verdict")) + if found is not None: + verdict, verdict_step = found, step_id + if verdict is None: + return Decision(False, "no critic verdict before the review point", None, None) + if verdict != "pass": + return Decision(False, f"critic verdict is {verdict!r} (step {verdict_step})", verdict, verdict_step) + return Decision(True, f"critic verdict pass (step {verdict_step}), no failed step", verdict, verdict_step) diff --git a/shakedown_lib/kab.py b/shakedown_lib/kab.py new file mode 100644 index 0000000..df553a3 --- /dev/null +++ b/shakedown_lib/kab.py @@ -0,0 +1,104 @@ +"""The KAB API, as the automation identity: the same endpoints `cp1-agents/scripts/kab_run_workflow.py` uses. + +Every call takes a fresh token from the token source, so a run that lasts hours never needs a person. A 401 renews +the token once; 429 and 5xx are retried with back-off. Errors raise KabError; nothing calls sys.exit. +""" + +from __future__ import annotations + +import json +import time +import urllib.error +import urllib.request +from typing import Any, Callable, Protocol + +DEFAULT_API = "https://api.kab-dev.kyndemo.live" +CP1_TENANT = "6ae37d13999438461a529409" +RETRIES = 4 + + +class TokenSource(Protocol): + def token(self, force: bool = False) -> str: ... + + +class KabError(RuntimeError): + def __init__(self, message: str, status: int | None = None) -> None: + super().__init__(message) + self.status = status + + +Transport = Callable[[str, str, dict, bytes | None], tuple[int, bytes]] + + +def _urllib_transport(method: str, url: str, headers: dict, data: bytes | None) -> tuple[int, bytes]: + request = urllib.request.Request(url, data=data, headers=headers, method=method) + try: + with urllib.request.urlopen(request, timeout=120) as response: + return response.status, response.read() + except urllib.error.HTTPError as error: + return error.code, error.read() or b"" + except urllib.error.URLError as error: + return 599, str(error.reason).encode() + + +class Kab: + def __init__(self, tokens: TokenSource, api: str = DEFAULT_API, tenant: str = CP1_TENANT, + project: str | None = None, transport: Transport = _urllib_transport, + sleep: Callable[[float], None] = time.sleep) -> None: + self.tokens = tokens + self.api = api.rstrip("/") + self.tenant = tenant + self.project = project + self._transport = transport + self._sleep = sleep + + def call(self, method: str, path: str, body: dict | None = None) -> Any: + data = json.dumps(body).encode() if body is not None else None + force = False + for attempt in range(RETRIES + 1): + headers = {"Authorization": f"Bearer {self.tokens.token(force=force)}", "X-Tenant-ID": self.tenant, + "Content-Type": "application/json"} + if self.project: + headers["X-Project-ID"] = self.project + status, raw = self._transport(method, f"{self.api}{path}", headers, data) + if 200 <= status < 300: + return json.loads(raw) if raw else {} + if status == 401 and not force: + force = True + continue + if (status == 429 or status >= 500) and attempt < RETRIES: + self._sleep(min(60, 5 * 2 ** attempt)) + continue + raise KabError(f"{method} {path}: HTTP {status}: {raw[:400].decode(errors='replace')}", status) + raise KabError(f"{method} {path}: gave up after {RETRIES} retries") + + # The calls the runner makes. + def workflow(self, workflow_id: str) -> dict: + found = self.call("GET", f"/api/v1/workflow-catalog/workflows/{workflow_id}") + return found.get("workflow", found) + + def api_sample(self, workflow_id: str) -> dict: + return self.call("GET", f"/api/workflows/{workflow_id}/api-sample") + + def project_name(self, project_id: str) -> str | None: + try: + found = self.call("GET", f"/api/v1/projects/{project_id}") + except KabError: + return None + return (found.get("project") or found).get("name") + + def execute(self, template: dict, inputs: dict) -> dict: + return self.call("POST", "/api/workflows/execute", {"template": template, "inputs": inputs}) + + def status(self, execution_id: str) -> dict: + return self.call("GET", f"/api/workflows/status/{execution_id}") + + def steps(self, execution_id: str) -> Any: + return self.call("GET", f"/api/workflows/executions/{execution_id}/steps") + + def reviews(self, execution_id: str) -> dict: + return self.call("GET", f"/api/workflows/executions/{execution_id}/reviews") + + def decide(self, execution_id: str, checkpoint_id: str, decision: str, note: str) -> dict: + return self.call("POST", f"/api/workflows/executions/{execution_id}/reviews/{checkpoint_id}", + {"decision": decision, "note": note}) diff --git a/shakedown_lib/ledger.py b/shakedown_lib/ledger.py new file mode 100644 index 0000000..f0610cf --- /dev/null +++ b/shakedown_lib/ledger.py @@ -0,0 +1,78 @@ +"""The run ledger: one folder per voyage run, and one per stage inside it (plan §4). + + ///state.json every stage's state, ids, gate decisions; rewritten on each change + ////inputs.json the run inputs as sent, uploads shown by name only + ////context.yaml the context file as uploaded or sent (with `generated:`) + ////status.json, steps.json, evaluation.json + +No token or secret is ever written here. state.json is what a resumed run reads. +""" + +from __future__ import annotations + +import json +import os +from datetime import datetime, timezone +from pathlib import Path +from typing import Any + + +def utc_stamp(now: float | None = None) -> str: + moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc) + return moment.strftime("%Y%m%dT%H%M%SZ") + + +def utc_iso(now: float | None = None) -> str: + moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc) + return moment.strftime("%Y-%m-%dT%H:%M:%SZ") + + +def default_root() -> Path: + return Path(os.getenv("SHAKEDOWN_LEDGER") or Path(__file__).resolve().parents[1] / "ledger") + + +class Ledger: + def __init__(self, folder: Path) -> None: + self.folder = Path(folder) + self.folder.mkdir(parents=True, exist_ok=True) + + @classmethod + def new(cls, root: Path, voyage: str, now: float | None = None) -> "Ledger": + return cls(Path(root) / voyage / utc_stamp(now)) + + def path(self, *parts: str) -> Path: + target = self.folder.joinpath(*parts) + target.parent.mkdir(parents=True, exist_ok=True) + return target + + def write(self, name: str, data: Any, stage: str | None = None) -> Path: + target = self.path(*([stage] if stage else []), name) + tmp = target.with_suffix(target.suffix + ".tmp") + if isinstance(data, (bytes, bytearray)): + tmp.write_bytes(data) + elif isinstance(data, str): + tmp.write_text(data) + else: + tmp.write_text(json.dumps(data, indent=1, default=str)) + tmp.replace(target) + return target + + def load_state(self) -> dict: + path = self.folder / "state.json" + return json.loads(path.read_text()) if path.is_file() else {} + + def save_state(self, state: dict) -> None: + self.write("state.json", state) + + +def shown_inputs(inputs: dict) -> dict: + """The inputs as recorded: uploads by name, long documents by length (their text is kept beside, in the stage).""" + out = {} + for key, value in inputs.items(): + if key == "file" and isinstance(value, list): + out[key] = [item.get("name") for item in value if isinstance(item, dict)] + elif isinstance(value, str) and len(value) > 200: + out[key] = f"<{len(value)} characters>" + else: + out[key] = value + return out diff --git a/shakedown_lib/runner.py b/shakedown_lib/runner.py new file mode 100644 index 0000000..98c52dc --- /dev/null +++ b/shakedown_lib/runner.py @@ -0,0 +1,389 @@ +"""Run a voyage remotely, stage by stage, with no person in the loop except where the rules keep one. + +For each stage, in order: +1. **Start**: check the workflow id still names the registered workflow; build the inputs from the context file and + the voyage (upstream ids resolved); check required inputs and the engagement id; start the run. + A stage with `use_execution` adopts an existing, finished run instead. +2. **Follow**: poll the run. At a review point: + - in a project listed in `config/runner.yaml` `gate_projects`, the gate rule (gate.py) decides: continue only on a + critic `pass` with no failed step, with a note saying the system decided. Otherwise the pause is left for a + person and the voyage stops for a fix; + - in any other project, wait for a person to decide in KAB. +3. **Finish**: pull the outputs, find the package and its artifact id, judge the run against its case + (cp1-agents `evaluate_case.py`), and stop the voyage on a failed run, a missing package or a failed critical check. + +The ledger is written on every change, so `--resume` picks a voyage up where it stopped (a started run is followed, +never started twice). +""" + +from __future__ import annotations + +import base64 +import re +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional + +import yaml + +from . import cp1_agents, gate +from .kab import Kab, KabError +from .ledger import Ledger, shown_inputs, utc_iso +from .voyage import Stage, Voyage, VoyageError, Workflow, load_workflow, resolve_path + +DONE_OK = {"completed", "succeeded", "success"} +DONE_BAD = {"failed", "error", "cancelled", "canceled", "aborted", "stopped", "rejected", "timed_out", "timeout"} +PLACEHOLDER = re.compile(r"^<[^>]+>$") + +EXIT_OK, EXIT_CONFIG, EXIT_STOPPED, EXIT_PERSON_TIMEOUT, EXIT_TIMEOUT = 0, 2, 3, 4, 5 + + +class Stop(Exception): + def __init__(self, code: int, reason: str) -> None: + super().__init__(reason) + self.code = code + self.reason = reason + + +@dataclass +class Settings: + gate_projects: set + poll_seconds: int = 30 + stage_timeout_s: int = 6 * 3600 + person_wait_s: int = 24 * 3600 + + @classmethod + def from_config(cls, config: dict) -> "Settings": + return cls(gate_projects=set(config.get("gate_projects") or set()), + poll_seconds=int(config.get("poll_seconds", 30)), + stage_timeout_s=int(config.get("stage_timeout_minutes", 360)) * 60, + person_wait_s=int(config.get("person_wait_hours", 24)) * 3600) + + +Evaluator = Callable[[str, dict, Any], tuple] + + +def default_evaluator(feature: str, case: dict, steps: Any) -> tuple: + return cp1_agents.evaluate_case().evaluate(feature, case, steps) + + +@dataclass +class Runner: + kab: Kab + ledger: Ledger + voyage: Voyage + settings: Settings + evaluator: Evaluator = default_evaluator + clock: Callable[[], float] = time.time + sleep: Callable[[float], None] = time.sleep + log: Callable[[str], None] = print + state: dict = field(default_factory=dict) + + # ---- state ------------------------------------------------------------------------------------------------- + def _load(self) -> None: + self.state = self.ledger.load_state() or { + "voyage": self.voyage.name, "engagement_id": self.voyage.engagement_id, "project": self.voyage.project, + "voyage_file": str(self.voyage.source) if self.voyage.source else None, "started_at": utc_iso(self.clock()), + "cp1_agents_commit": cp1_agents.commit(), "decided_by": "kab-automation (Shakedown runner)", + "stages": {}} + + def _save(self) -> None: + self.state["updated_at"] = utc_iso(self.clock()) + self.ledger.save_state(self.state) + + def _stage_state(self, key: str) -> dict: + return self.state["stages"].setdefault(key, {"state": "pending", "decisions": []}) + + # ---- the voyage -------------------------------------------------------------------------------------------- + def run(self, dry_run: bool = False, only: Optional[List[str]] = None) -> int: + self._load() + if self.state.get("project") != self.voyage.project: + self.log(f"refused: the ledger is for project {self.state.get('project')}, the voyage for {self.voyage.project}") + return EXIT_CONFIG + gates = "decided by the system" if self.voyage.project in self.settings.gate_projects else "left to a person" + self.log(f"Voyage {self.voyage.name} ({self.voyage.engagement_id}) in project {self.voyage.project}; gates {gates}.") + self.log(f"Ledger: {self.ledger.folder}") + try: + for stage in self.voyage.stages: + if only and stage.key not in only and self._stage_state(stage.key).get("state") != "completed": + if any(s.key in only for s in self.voyage.stages[self.voyage.stages.index(stage) + 1:]): + raise Stop(EXIT_CONFIG, f"stage {stage.key} isn't completed, and a later stage was asked for") + continue + self._stage(stage, dry_run) + except Stop as stop: + self.state["stopped"] = {"code": stop.code, "reason": stop.reason, "at": utc_iso(self.clock())} + self._save() + self.log(f"Stopped: {stop.reason}") + return stop.code + except (KabError, VoyageError) as error: + self.state["stopped"] = {"code": EXIT_CONFIG, "reason": str(error), "at": utc_iso(self.clock())} + self._save() + self.log(f"Stopped: {error}") + return EXIT_CONFIG + self.state.pop("stopped", None) + self.state["finished_at"] = utc_iso(self.clock()) + self._save() + self.log("Dry run: nothing started." if dry_run else "Voyage complete.") + return EXIT_OK + + def _stage(self, stage: Stage, dry_run: bool) -> None: + st = self._stage_state(stage.key) + if st.get("state") == "completed": + self.log(f"[{stage.key}] already completed: {st.get('execution_id')}") + return + wf = load_workflow(stage.workflow) + workflow_id = stage.workflow_id or wf.workflow_id + if stage.use_execution: + st.update(execution_id=stage.use_execution, adopted=True, approval_ref=stage.approval_ref, + workflow=wf.key, state="started") + self._save() + if not st.get("execution_id"): + if not workflow_id: + raise VoyageError(f"stage {stage.key}: no workflow_id in the voyage or workflows/{wf.key}.yaml") + template = self._checked_template(stage, wf, workflow_id) + inputs, context_text = self._inputs(stage, wf, workflow_id) + st.update(workflow=wf.key, workflow_id=workflow_id, workflow_name=template.get("name"), + workflow_version=template.get("version"), run_id=inputs.get("run_id")) + self.ledger.write("inputs.json", shown_inputs(inputs), stage.key) + if context_text is not None: + self.ledger.write("context.yaml", context_text, stage.key) + if dry_run: + st["state"] = "planned" + self._save() + self.log(f"[{stage.key}] would start {template.get('name')} ({workflow_id}) with " + f"{shown_inputs(inputs)}") + return + name = self.kab.project_name(self.voyage.project) + body = cp1_agents.run_workflow().with_run_metadata(template, workflow_id, self.voyage.project, name) + result = self.kab.execute(body, inputs) + if not result.get("execution_id"): + raise Stop(EXIT_STOPPED, f"stage {stage.key}: KAB started no execution: {str(result)[:300]}") + st.update(execution_id=result["execution_id"], state="started", started_at=utc_iso(self.clock())) + self._save() + self.log(f"[{stage.key}] started {result['execution_id']} ({template.get('name')})") + self._follow(stage, st) + self._finish(stage, wf, st) + + # ---- start --------------------------------------------------------------------------------------------------- + def _checked_template(self, stage: Stage, wf: Workflow, workflow_id: str) -> dict: + workflow = self.kab.workflow(workflow_id) + template = workflow.get("template") or workflow + if template.get("name") not in wf.names: + raise VoyageError(f"stage {stage.key}: workflow {workflow_id} is {template.get('name')!r}, not one of " + f"{wf.names} (ids change on re-import: update the voyage)") + return template + + def _resolve(self, stage: Stage, value: Any) -> Any: + if not isinstance(value, dict): + return value + upstream = self.state["stages"].get(value["from"]) or {} + if upstream.get("state") == "planned": # a dry run: the upstream run doesn't exist yet + deferred = f"(at run time: {value['from']}.{value['value']})" + return [deferred] if value["value"] == "package_artifact_ids" else deferred + if upstream.get("state") != "completed": + raise VoyageError(f"stage {stage.key} needs stage {value['from']} completed first") + found = upstream.get(value["value"]) + if found in (None, "", []): + raise VoyageError(f"stage {stage.key}: stage {value['from']} has no {value['value']}") + return found + + def _inputs(self, stage: Stage, wf: Workflow, workflow_id: str) -> tuple[dict, Optional[str]]: + helpers = cp1_agents.run_workflow() + given = {name: self._resolve(stage, value) for name, value in stage.inputs.items()} + inputs: Dict[str, Any] = {} + context_doc, context_text = None, None + if stage.context: + path = resolve_path(stage.context) + context_doc = yaml.safe_load(path.read_text()) or {} + from_file = (context_doc.get("workflow_inputs") or {}) if isinstance(context_doc, dict) else {} + for key, value in from_file.items(): + value = "yes" if value is True else "no" if value is False else value + if key in given and not (isinstance(value, str) and PLACEHOLDER.match(value)) and str(value) != str(given[key]): + raise VoyageError(f"stage {stage.key}: {key} is {value!r} in {path.name} and {given[key]!r} in " + "the voyage: keep one") + inputs[key] = value if not isinstance(value, (int, float)) else str(value) + inputs.update(given) + if "run_id" in inputs: + raise VoyageError(f"stage {stage.key}: run_id is generated for every run; remove it") + if inputs.get("engagement_id") not in (None, self.voyage.engagement_id): + raise VoyageError(f"stage {stage.key}: engagement_id {inputs.get('engagement_id')} is not the voyage's " + f"{self.voyage.engagement_id}") + inputs["engagement_id"] = self.voyage.engagement_id + generated = helpers.new_ids() + inputs["run_id"] = generated["run_id"] + for name, doc_path in stage.documents.items(): + inputs[name] = resolve_path(doc_path).read_text() + + if context_doc is not None: + has_upstream = any(inputs.get(n) for n in helpers.UPSTREAM_INPUTS) + problem = helpers.customer_problem(context_doc, has_upstream=has_upstream) + if problem: + raise VoyageError(f"stage {stage.key}: {stage.context}: {problem}") + header = ("# Added by the Shakedown runner when the run started. Never edit these by hand.\n" + "generated:\n" + "".join(f" {k}: {v}\n" for k, v in generated.items()) + "\n") + path = resolve_path(stage.context) + if wf.context == "upload": + context_text = header + path.read_text() + files = [{"name": path.name, "content": self._data_url(path, context_text.encode())}] + for extra in helpers.context_uploads(path): + files.append({"name": extra.name, "content": self._data_url(extra, extra.read_bytes())}) + inputs["file"] = files + elif wf.context == "document": + resolved = dict(context_doc) + resolved["workflow_inputs"] = {k: inputs[k] for k in (context_doc.get("workflow_inputs") or {}) + if k in inputs} + context_text = header + yaml.safe_dump(resolved, sort_keys=False, allow_unicode=True) + inputs[wf.document_input] = context_text + + leftover = sorted(k for k, v in inputs.items() if isinstance(v, str) and PLACEHOLDER.match(v.strip())) + if leftover: + raise VoyageError(f"stage {stage.key}: still a placeholder: {', '.join(leftover)}") + sample = self.kab.api_sample(workflow_id) + missing = sorted(f"{i['step_id']}.{i['name']}" for i in sample.get("required", []) + if i["name"] != "file" and i["name"] not in inputs) + if missing: + raise VoyageError(f"stage {stage.key}: missing inputs {', '.join(missing)}") + checkable = {k: v for k, v in inputs.items() if not (isinstance(v, str) and v.startswith("(at run time: "))} + try: + helpers.check_engagement(self.kab, checkable) + except SystemExit as refused: + raise VoyageError(f"stage {stage.key}: {refused}") from None + return inputs, context_text + + @staticmethod + def _data_url(path: Path, data: bytes) -> str: + kind = cp1_agents.run_workflow().upload_type(path) + return f"data:{kind};base64,{base64.b64encode(data).decode()}" + + # ---- follow -------------------------------------------------------------------------------------------------- + def _follow(self, stage: Stage, st: dict) -> None: + eid = st["execution_id"] + deadline = self.clock() + self.settings.stage_timeout_s + while True: + status = self.kab.status(eid) + state = str(status.get("status", "")).lower() + if state in DONE_OK or state in DONE_BAD: + if st.get("state") == "needs_person": # a person decided and the run finished between two polls + self._person_decided(stage, st) + st["run_status"] = state + self._save() + return + pending = (self.kab.reviews(eid) or {}).get("pending") or [] + if pending: + self._at_gate(stage, st, status, pending) + elif st.get("state") == "needs_person": + self._person_decided(stage, st) + if self.clock() > deadline: + raise Stop(EXIT_TIMEOUT, f"stage {stage.key}: {eid} still {state} after " + f"{self.settings.stage_timeout_s // 60} minutes") + self.sleep(self.settings.poll_seconds) + + def _at_gate(self, stage: Stage, st: dict, status: dict, pending: list) -> None: + eid = st["execution_id"] + review = pending[0] + checkpoint = str(review.get("checkpoint_id")) + if any(d.get("checkpoint_id") == checkpoint and d.get("decision") == "continue" for d in st["decisions"]): + return # decided; KAB hasn't moved on yet + system_may_decide = self.voyage.project in self.settings.gate_projects and len(pending) == 1 + if not system_may_decide: + if st.get("state") != "needs_person": + st.update(state="needs_person", waiting_since=self.clock(), checkpoint_id=checkpoint) + self._save() + why = "more than one pause" if len(pending) > 1 else "this project isn't one of the runner's gate_projects" + self.log(f"[{stage.key}] {eid} waits at a review point for a person in KAB ({why}).") + elif self.clock() - st.get("waiting_since", self.clock()) > self.settings.person_wait_s: + raise Stop(EXIT_PERSON_TIMEOUT, f"stage {stage.key}: nobody decided {eid}'s review point in " + f"{self.settings.person_wait_s // 3600} hours") + return + steps = self.kab.steps(eid) + verdict = gate.decide(status, steps, review.get("step_id")) + at = utc_iso(self.clock()) + record = {"checkpoint_id": checkpoint, "step_id": review.get("step_id"), "at": at, "by": "system", + "verdict": verdict.verdict, "verdict_step": verdict.verdict_step, "reason": verdict.reason} + if not verdict.proceed: + record["decision"] = "left for a fix" + st["decisions"].append(record) + st["state"] = "needs_fix" + self.ledger.write("steps-at-gate.json", steps, stage.key) + self._save() + raise Stop(EXIT_STOPPED, f"stage {stage.key}: not continued at the review point: {verdict.reason}. " + f"{eid} stays paused for a person or a fix.") + note = (f"Decided by the system (Shakedown runner, KAB automation identity), not a person: continue. " + f"{verdict.reason}. Execution {eid}, run {st.get('run_id') or '?'}, voyage {self.voyage.name}, {at}.") + self.kab.decide(eid, checkpoint, "continue", note) + record.update(decision="continue", note=note) + st["decisions"].append(record) + st.update(state="started", approval_ref=f"Shakedown runner (system) continued {eid} at {at}: " + f"critic verdict pass at {verdict.verdict_step}") + self._save() + self.log(f"[{stage.key}] continued {eid}: {verdict.reason}") + + def _person_decided(self, stage: Stage, st: dict) -> None: + """The pause is gone and the run moves on: a person decided in KAB. Record who, as best KAB says.""" + eid = st["execution_id"] + reviews = self.kab.reviews(eid) or {} + decided = [r for key in ("history", "decided", "completed", "reviews") for r in (reviews.get(key) or []) + if isinstance(r, dict) and str(r.get("checkpoint_id")) == str(st.get("checkpoint_id"))] + last = decided[-1] if decided else {} + who = last.get("decided_by") or last.get("reviewer") or last.get("user") or "a person in KAB" + when = last.get("decided_at") or last.get("updated_at") or utc_iso(self.clock()) + decision = last.get("decision") or "continue" + st["decisions"].append({"checkpoint_id": st.get("checkpoint_id"), "by": "person", "decision": decision, + "who": who, "at": when}) + st.update(state="started", approval_ref=f"{who} decided {decision} on {eid} at {when}") + st.pop("waiting_since", None) + self._save() + self.log(f"[{stage.key}] {who} decided {decision} on {eid}") + + # ---- finish -------------------------------------------------------------------------------------------------- + def _finish(self, stage: Stage, wf: Workflow, st: dict) -> None: + eid = st["execution_id"] + status = self.kab.status(eid) + steps = self.kab.steps(eid) + self.ledger.write("status.json", status, stage.key) + self.ledger.write("steps.json", steps, stage.key) + run_status = str(status.get("status", "")).lower() + if run_status not in DONE_OK: + st["state"] = "failed" + self._save() + raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} ended {run_status or 'with no status'}") + artifact = self._package_artifact(eid, status, steps, st.get("workflow_id")) + if artifact is None: + st["state"] = "failed" + self._save() + raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} completed with no package (no output has a manifest " + "with a package_version)") + st["package_artifact_ids"] = [artifact] + if not st.get("approval_ref"): + st["approval_ref"] = f"{eid} completed with no review point decided by the runner" + if stage.case: + case = yaml.safe_load(resolve_path(stage.case).read_text()) + rows, failed = self.evaluator(wf.feature, case, steps) + evaluation = {"case": stage.case, "failed": failed, + "checks": [dict(zip(("status", "severity", "where", "id", "detail"), r)) for r in rows]} + self.ledger.write("evaluation.json", evaluation, stage.key) + st["evaluation"] = {"failed": failed, "checks": len(rows)} + if failed.get("critical"): + st["state"] = "checks_failed" + self._save() + raise Stop(EXIT_STOPPED, f"stage {stage.key}: {failed['critical']} critical check(s) failed on {eid} " + f"(see {stage.key}/evaluation.json)") + st.update(state="completed", completed_at=utc_iso(self.clock())) + self._save() + self.log(f"[{stage.key}] completed {eid}; package {artifact}") + + def _package_artifact(self, eid: str, status: dict, steps: Any, workflow_id: Optional[str]) -> Optional[str]: + """`_step_`, n the package step's 1-based place in the executed template.""" + outputs = gate.step_outputs(steps) + holder = next((k for k, v in outputs.items() if isinstance(v, dict) and isinstance(v.get("manifest"), dict) + and "package_version" in v["manifest"]), None) + if holder is None: + return None + order = gate.template_order(status) + if holder not in order and workflow_id: + workflow = self.kab.workflow(workflow_id) + order = [str(s.get("id")) for s in ((workflow.get("template") or workflow).get("steps") or [])] + if holder not in order: + return None + return f"{eid}_step_{order.index(holder) + 1}" diff --git a/shakedown_lib/voyage.py b/shakedown_lib/voyage.py new file mode 100644 index 0000000..70b8f78 --- /dev/null +++ b/shakedown_lib/voyage.py @@ -0,0 +1,116 @@ +"""Voyages (one engagement's chain of use cases) and the workflow registry, as loaded from this repository. + +A path is relative to this repository, or to the cp1-agents clone when it starts with `cp1-agents:` (the committed +F01 to F03 cases stay there and are read in place, plan §3). + +A stage input is a literal, or a reference to an earlier stage's result: `{from: , value: }`, where +the name is `execution_id`, `approval_ref` or `package_artifact_ids` (`["_step_"]` of its package +step). References are resolved when the stage starts; the files themselves are never edited. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +import yaml + +from . import cp1_agents + +REPO = Path(__file__).resolve().parents[1] +CONTEXT_ADAPTERS = {"upload", "document", "none"} +STAGE_VALUES = {"execution_id", "approval_ref", "package_artifact_ids"} + + +class VoyageError(ValueError): + pass + + +def resolve_path(text: str) -> Path: + if text.startswith("cp1-agents:"): + return cp1_agents.clone_dir() / text.split(":", 1)[1] + path = Path(text) + return path if path.is_absolute() else REPO / path + + +@dataclass +class Workflow: + key: str + feature: str + names: List[str] + context: str + document_input: Optional[str] = None + workflow_id: Optional[str] = None + + +def load_workflow(key: str) -> Workflow: + path = REPO / "workflows" / f"{key}.yaml" + if not path.is_file(): + raise VoyageError(f"workflow {key!r} is not registered: no {path.relative_to(REPO)}") + doc = yaml.safe_load(path.read_text()) or {} + wf = Workflow(key=doc.get("key", key), feature=doc["feature"], names=list(doc["names"]), context=doc["context"], + document_input=doc.get("document_input"), workflow_id=doc.get("workflow_id")) + if wf.context not in CONTEXT_ADAPTERS: + raise VoyageError(f"{path.name}: context must be one of {sorted(CONTEXT_ADAPTERS)}") + if wf.context == "document" and not wf.document_input: + raise VoyageError(f"{path.name}: a document context names its run input in document_input") + return wf + + +@dataclass +class Stage: + key: str + workflow: str + workflow_id: Optional[str] = None + context: Optional[str] = None + case: Optional[str] = None + inputs: Dict[str, Any] = field(default_factory=dict) + documents: Dict[str, str] = field(default_factory=dict) + use_execution: Optional[str] = None + approval_ref: Optional[str] = None + + +@dataclass +class Voyage: + name: str + engagement_id: str + project: str + stages: List[Stage] + capability: Optional[str] = None + source: Optional[Path] = None + + +def load_voyage(path: Path) -> Voyage: + doc = yaml.safe_load(Path(path).read_text()) or {} + for required in ("voyage", "engagement_id", "project", "stages"): + if not doc.get(required): + raise VoyageError(f"{path}: `{required}` is required") + stages, seen = [], set() + for raw in doc["stages"]: + stage = Stage(key=raw["key"], workflow=raw["workflow"], workflow_id=raw.get("workflow_id"), + context=raw.get("context"), case=raw.get("case"), inputs=dict(raw.get("inputs") or {}), + documents=dict(raw.get("documents") or {}), use_execution=raw.get("use_execution"), + approval_ref=raw.get("approval_ref")) + if stage.key in seen: + raise VoyageError(f"{path}: stage key {stage.key!r} appears twice") + if stage.use_execution and not stage.approval_ref: + raise VoyageError(f"{path}: stage {stage.key} uses execution {stage.use_execution}: give its approval_ref " + "(who continued it at its review point, and when)") + for name, value in stage.inputs.items(): + if isinstance(value, dict): + if value.get("from") not in seen or value.get("value") not in STAGE_VALUES: + raise VoyageError(f"{path}: stage {stage.key} input {name}: `from` must name an earlier stage and " + f"`value` one of {sorted(STAGE_VALUES)}") + seen.add(stage.key) + stages.append(stage) + return Voyage(name=doc["voyage"], engagement_id=doc["engagement_id"], project=doc["project"], stages=stages, + capability=doc.get("capability"), source=Path(path)) + + +def load_config(path: Path | None = None) -> dict: + """Runner settings, from a committed file so a change to them is reviewed. The environment can't widen them.""" + path = path or REPO / "config" / "runner.yaml" + doc = yaml.safe_load(path.read_text()) or {} + doc["gate_projects"] = {str(p) for p in (doc.get("gate_projects") or [])} + return doc diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_runner.py b/tests/test_runner.py new file mode 100644 index 0000000..7f4e186 --- /dev/null +++ b/tests/test_runner.py @@ -0,0 +1,320 @@ +"""The runner against a fake KAB: python3 -m unittest discover -s tests (needs the cp1-agents clone beside this repo, +or CP1_AGENTS_DIR).""" + +from __future__ import annotations + +import json +import tempfile +import textwrap +import unittest +from pathlib import Path +from urllib.parse import urlparse + +import yaml + +from shakedown_lib import gate +from shakedown_lib.auth import AuthError, ClientCredentials +from shakedown_lib.kab import Kab +from shakedown_lib.ledger import Ledger +from shakedown_lib.runner import EXIT_CONFIG, EXIT_OK, EXIT_STOPPED, Runner, Settings +from shakedown_lib.voyage import load_voyage + +PROJECT = "proj-1" +ENG = "ENG-TEST-001" +NAMES = {"wf-f01": "CP1 - Agentic Landing Zone - Discovery", "wf-f02": "CP1 - Agentic Landing Zone - Assessment", + "wf-f03": "CP1 - Agentic Landing Zone - Strategy Synthesis"} + + +def template(name: str, upload: bool) -> dict: + first = "u1" if upload else "a1" + return {"name": name, "version": "1.0", "steps": [{"id": first}] + [{"id": f"s{i}"} for i in range(1, 7)] + [{"id": "f1"}]} + + +class FakeKab: + """Runs move: running -> paused at s5 (one pending review) -> after `continue`, completed with a package at s6.""" + + def __init__(self, verdict: str = "pass", polls_before_pause: int = 1, person_after: int | None = None, + names: dict | None = None) -> None: + self.verdict = verdict + self.polls_before_pause = polls_before_pause + self.person_after = person_after + self.names = names or NAMES + self.runs: dict = {"exec_f01_done": {"workflow": "wf-f01", "inputs": {"engagement_id": ENG}, "phase": "done", + "polls": 0, "history": []}} + self.executed: list = [] + self.decisions: list = [] + + def transport(self, method, url, headers, data): + assert headers["Authorization"].startswith("Bearer "), "no token sent" + path = urlparse(url).path + body = json.loads(data) if data else None + reply = self.route(method, path, body) + return 200, json.dumps(reply).encode() + + def _status(self, eid): + run = self.runs[eid] + state = {"running": "running", "paused": "attention", "done": "completed"}[run["phase"]] + return {"execution_id": eid, "status": state, "inputs": run["inputs"], + "template": template(self.names[run["workflow"]], run["workflow"] == "wf-f01")} + + def _steps(self, eid): + run = self.runs[eid] + steps = {"s4": {"response": json.dumps({"phase": "critique", "status": "completed", + "verdict": {"verdict": self.verdict}})}} + if run["phase"] == "done": + steps["s6"] = {"output": {"engagement_id": ENG, "manifest": {"package_version": "1.0"}}} + return {"execution_id": eid, "steps": steps} + + def route(self, method, path, body): + if path.startswith("/api/v1/workflow-catalog/workflows/"): + wid = path.rsplit("/", 1)[1] + return {"workflow": {"template": template(self.names[wid], wid == "wf-f01")}} + if path.endswith("/api-sample"): + return {"required": []} + if path.startswith("/api/v1/projects/"): + return {"project": {"name": "Test project"}} + if path == "/api/workflows/execute": + wid = body["template"]["metadata"]["workflow_id"] + eid = f"exec_{len(self.executed) + 1}" + self.executed.append({"workflow": wid, "inputs": body["inputs"], "template": body["template"]}) + self.runs[eid] = {"workflow": wid, "inputs": body["inputs"], "phase": "running", "polls": 0, "history": []} + return {"execution_id": eid, "status": "running"} + if path.startswith("/api/workflows/status/"): + eid = path.rsplit("/", 1)[1] + run = self.runs[eid] + run["polls"] += 1 + if run["phase"] == "running" and run["polls"] > self.polls_before_pause: + run["phase"] = "paused" + run["paused_at"] = run["polls"] + if run["phase"] == "paused" and self.person_after is not None and run["polls"] - run["paused_at"] >= self.person_after: + run["phase"] = "done" + run["history"].append({"checkpoint_id": f"cp-{eid}", "decision": "continue", + "decided_by": "lead@example.com", "decided_at": "2026-11-16T10:00:00Z"}) + return self._status(eid) + if path.endswith("/steps"): + return self._steps(path.split("/")[-2]) + if path.endswith("/reviews") and method == "GET": + eid = path.split("/")[-2] + run = self.runs[eid] + pending = [{"checkpoint_id": f"cp-{eid}", "step_id": "s5"}] if run["phase"] == "paused" else [] + return {"pending": pending, "history": run["history"]} + if "/reviews/" in path and method == "POST": + eid = path.split("/")[-3] + self.decisions.append({"execution": eid, **body}) + if body["decision"] == "continue": + self.runs[eid]["phase"] = "done" + return {"ok": True} + raise AssertionError(f"unexpected {method} {path}") + + +class Tokens: + def __init__(self): + self.calls = 0 + + def token(self, force=False): + self.calls += 1 + return "t" + + +class Clock: + def __init__(self): + self.now = 1_800_000_000.0 + + def __call__(self): + return self.now + + def sleep(self, seconds): + self.now += seconds + + +def write_voyage(folder: Path, f01_adopted: bool = True) -> Path: + (folder / "context.yaml").write_text(textwrap.dedent(f"""\ + workflow_inputs: + engagement_id: {ENG} + provider: sample + include_cmdb: yes + customer: {{id: CUS-TEST, name: Test}} + """)) + (folder / "run-context.yaml").write_text(textwrap.dedent(f"""\ + workflow_inputs: + engagement_id: {ENG} + assessment_execution_id: + assessment_approval_ref: "" + context: + customer: {{id: CUS-TEST}} + """)) + (folder / "rate-card.json").write_text('{"rates": []}') + (folder / "case.yaml").write_text("id: test\nchecks: []\n") + f01 = {"key": "f01", "workflow": "alz-f01-discovery", "workflow_id": "wf-f01", "context": f"{folder}/context.yaml"} + if f01_adopted: + f01.update(use_execution="exec_f01_done", approval_ref="approved by a person") + doc = {"voyage": ENG, "engagement_id": ENG, "project": PROJECT, "stages": [ + f01, + {"key": "f02", "workflow": "alz-f02-assessment", "workflow_id": "wf-f02", + "inputs": {"discovery_execution_id": {"from": "f01", "value": "execution_id"}, + "discovery_approval_ref": {"from": "f01", "value": "approval_ref"}, + "artifact_ids": {"from": "f01", "value": "package_artifact_ids"}}, + "documents": {"rate_card_document": f"{folder}/rate-card.json"}, "case": f"{folder}/case.yaml"}, + {"key": "f03", "workflow": "alz-f03-strategy", "workflow_id": "wf-f03", "context": f"{folder}/run-context.yaml", + "inputs": {"assessment_execution_id": {"from": "f02", "value": "execution_id"}, + "assessment_approval_ref": {"from": "f02", "value": "approval_ref"}, + "artifact_ids": {"from": "f02", "value": "package_artifact_ids"}}}]} + voyage = folder / "voyage.yaml" + voyage.write_text(yaml.safe_dump(doc, sort_keys=False)) + return voyage + + +class RunnerTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.dir = Path(self.tmp.name) + self.clock = Clock() + self.evaluated = [] + + def tearDown(self): + self.tmp.cleanup() + + def runner(self, fake: FakeKab, gate_projects=(PROJECT,), ledger: Ledger | None = None, adopted=True, **settings): + voyage = load_voyage(write_voyage(self.dir, adopted)) + kab = Kab(Tokens(), api="https://kab.test", tenant="t", project=PROJECT, transport=fake.transport, + sleep=lambda s: None) + + def evaluator(feature, case, steps): + self.evaluated.append(feature) + return [("pass", "critical", "manifest", "c1", "")], {"critical": 0, "major": 0} + + return Runner(kab=kab, ledger=ledger or Ledger(self.dir / "ledger"), voyage=voyage, + settings=Settings(gate_projects=set(gate_projects), poll_seconds=30, **settings), + evaluator=evaluator, clock=self.clock, sleep=self.clock.sleep, log=lambda m: None) + + def test_whole_voyage_decides_gates_in_its_own_project(self): + fake = FakeKab() + runner = self.runner(fake) + self.assertEqual(runner.run(), EXIT_OK) + self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"]) + f02, f03 = fake.executed + self.assertEqual(f02["inputs"]["discovery_execution_id"], "exec_f01_done") + self.assertEqual(f02["inputs"]["artifact_ids"], ["exec_f01_done_step_7"]) + self.assertEqual(f02["inputs"]["rate_card_document"], '{"rates": []}') + self.assertEqual(f02["template"]["metadata"]["project_id"], PROJECT) + self.assertEqual(f03["inputs"]["artifact_ids"], ["exec_1_step_7"]) + self.assertIn("Shakedown runner (system) continued exec_1", f03["inputs"]["assessment_approval_ref"]) + document = f03["inputs"]["run_context_document"] + self.assertIn("assessment_execution_id: exec_1", document) + self.assertNotIn("", document) + self.assertEqual([d["decision"] for d in fake.decisions], ["continue", "continue"]) + self.assertTrue(all("not a person" in d["note"] for d in fake.decisions)) + state = json.loads((self.dir / "ledger" / "state.json").read_text()) + self.assertEqual({k: v["state"] for k, v in state["stages"].items()}, + {"f01": "completed", "f02": "completed", "f03": "completed"}) + self.assertEqual(self.evaluated, ["f02"]) + self.assertTrue((self.dir / "ledger" / "f03" / "context.yaml").is_file()) + + def test_critic_fail_is_never_continued(self): + fake = FakeKab(verdict="fail") + self.assertEqual(self.runner(fake).run(), EXIT_STOPPED) + self.assertEqual(fake.decisions, []) + state = json.loads((self.dir / "ledger" / "state.json").read_text()) + self.assertEqual(state["stages"]["f02"]["state"], "needs_fix") + self.assertIn("critic verdict is 'fail'", state["stopped"]["reason"]) + + def test_other_projects_wait_for_a_person(self): + fake = FakeKab(person_after=3) + self.assertEqual(self.runner(fake, gate_projects=()).run(), EXIT_OK) + self.assertEqual(fake.decisions, []) + f03 = fake.executed[1] + self.assertEqual(f03["inputs"]["assessment_approval_ref"], + "lead@example.com decided continue on exec_1 at 2026-11-16T10:00:00Z") + + def test_nobody_deciding_times_out(self): + fake = FakeKab() + code = self.runner(fake, gate_projects=(), person_wait_s=120).run() + self.assertEqual(code, 4) + self.assertEqual(fake.decisions, []) + + def test_a_stale_workflow_id_is_refused(self): + fake = FakeKab(names={**NAMES, "wf-f02": "CP1 - Agentic Landing Zone - Assessment (individual agents)"}) + self.assertEqual(self.runner(fake).run(), EXIT_CONFIG) + self.assertEqual(fake.executed, []) + + def test_resume_never_starts_a_run_twice(self): + fake = FakeKab(verdict="fail") + ledger = Ledger(self.dir / "ledger") + self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_STOPPED) + fake.verdict = "pass" # say a fix was deployed and the stage rerun in KAB + self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_OK) + self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"]) + + def test_dry_run_starts_nothing(self): + fake = FakeKab() + self.assertEqual(self.runner(fake, adopted=False).run(dry_run=True), EXIT_OK) + self.assertEqual(fake.executed, []) + shown = json.loads((self.dir / "ledger" / "f02" / "inputs.json").read_text()) + self.assertEqual(shown["discovery_execution_id"], "(at run time: f01.execution_id)") + f01 = json.loads((self.dir / "ledger" / "f01" / "inputs.json").read_text()) + self.assertEqual(f01["file"], ["context.yaml"]) + self.assertEqual(f01["include_cmdb"], "yes") + + +class GateTest(unittest.TestCase): + def status(self): + return {"status": "attention", "template": {"steps": [{"id": i} for i in ("a1", "s1", "s2", "s3", "s4", "s5", "s6")]}} + + def test_pass(self): + steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": "pass"}}}} + self.assertTrue(gate.decide(self.status(), steps, "s5").proceed) + + def test_a_later_fail_overrides_an_earlier_pass(self): + steps = {"steps": {"s3": {"phase": "x", "verdict": "pass"}, "s4": {"phase": "critique", "verdict": {"verdict": "fail"}}}} + self.assertFalse(gate.decide(self.status(), steps, "s5").proceed) + + def test_no_verdict(self): + self.assertFalse(gate.decide(self.status(), {"steps": {"s1": {"phase": "plan"}}}, "s5").proceed) + + def test_a_failed_step(self): + steps = {"steps": {"s2": {"phase": "collect", "status": "failed"}, "s4": {"phase": "c", "verdict": "pass"}}} + self.assertFalse(gate.decide(self.status(), steps, "s5").proceed) + + def test_unparseable_verdict(self): + steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": None}}}} + self.assertFalse(gate.decide(self.status(), steps, "s5").proceed) + + +class AuthTest(unittest.TestCase): + def test_renews_before_expiry_and_hides_the_secret(self): + clock = Clock() + issued = [] + + def post(url, form): + issued.append(form["grant_type"]) + return {"access_token": f"tok{len(issued)}", "expires_in": 300} + + tokens = ClientCredentials("kab-automation", "s3cret", post=post, clock=clock) + self.assertEqual(tokens.token(), "tok1") + clock.now += 200 + self.assertEqual(tokens.token(), "tok1") + clock.now += 50 # inside the last minute + self.assertEqual(tokens.token(), "tok2") + self.assertEqual(issued, ["client_credentials", "client_credentials"]) + self.assertNotIn("s3cret", repr(tokens)) + + def test_no_secret(self): + with self.assertRaises(AuthError): + ClientCredentials("kab-automation", "") + + def test_a_401_renews_the_token_once(self): + calls = [] + + class T: + def token(self, force=False): + calls.append(force) + return "x" + + replies = iter([(401, b""), (200, b'{"ok": 1}')]) + kab = Kab(T(), transport=lambda *a: next(replies), sleep=lambda s: None) + self.assertEqual(kab.call("GET", "/x"), {"ok": 1}) + self.assertEqual(calls, [False, True]) + + +if __name__ == "__main__": + unittest.main() diff --git a/workflows/alz-f01-discovery.yaml b/workflows/alz-f01-discovery.yaml new file mode 100644 index 0000000..79afd82 --- /dev/null +++ b/workflows/alz-f01-discovery.yaml @@ -0,0 +1,8 @@ +# ALZ-F01 Discovery (ADO Feature 3214441). Registry entry: plan §3c. +key: alz-f01-discovery +pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents` +feature: f01 # the feature name cp1-agents scripts/evaluate_case.py takes +# Workflow names KAB may show for it. A run's workflow id is checked against these before it starts, because ids +# change on every re-import (same names as the docs portal's registry/workflows.yaml). +names: ["CP1 - Agentic Landing Zone - Discovery"] +context: upload # the run context is uploaded to u1 (file); its workflow_inputs become run inputs diff --git a/workflows/alz-f02-assessment.yaml b/workflows/alz-f02-assessment.yaml new file mode 100644 index 0000000..6f5888b --- /dev/null +++ b/workflows/alz-f02-assessment.yaml @@ -0,0 +1,8 @@ +# ALZ-F02 Assessment & Disposition (ADO Feature 3214442). Registry entry: plan §3c. +key: alz-f02-assessment +pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents` +feature: f02 +# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change +# on every re-import). +names: ["CP1 - Agentic Landing Zone - Assessment"] +context: none # no context file: inputs come from the voyage (an alignment finding, plan §3b) diff --git a/workflows/alz-f03-strategy.yaml b/workflows/alz-f03-strategy.yaml new file mode 100644 index 0000000..411aa46 --- /dev/null +++ b/workflows/alz-f03-strategy.yaml @@ -0,0 +1,9 @@ +# ALZ-F03 Strategy Synthesis (ADO Feature 3214443). Registry entry: plan §3c. +key: alz-f03-strategy +pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents` +feature: f03 +# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change +# on every re-import). +names: ["CP1 - Agentic Landing Zone - Strategy Synthesis"] +context: document # the run context is sent as text in one run input (F03 ADR 0002) +document_input: run_context_document -- 2.49.1 From fb65ef95150ccaa57f50f7d713af3f5e5990cce7 Mon Sep 17 00:00:00 2001 From: Jonathan Boniface Date: Mon, 16 Nov 2026 05:14:00 +0000 Subject: [PATCH 4/4] Runner: the system decides review points in the Cymbal Bank project The owner named the KAB project "Cymbal Bank" (5a2da83b-...; voyage ENG-CYMBAL-001) as one of Shakedown's own. There, the runner continues a review point only on a critic pass with no failed step (plan 5b). Every other project still waits for a person. Co-Authored-By: Claude Opus 5.5 --- config/runner.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/config/runner.yaml b/config/runner.yaml index d2bae79..d59a210 100644 --- a/config/runner.yaml +++ b/config/runner.yaml @@ -4,9 +4,9 @@ # projects. In any other project it waits for a person. It continues a gate only on a critic `pass` with no failed # step (shakedown_lib/gate.py), never on a `fail`, and its note says the system decided. # -# Empty until the owner names the projects. For the Cymbal voyage that would be the "Cymbal Bank" project: -# - 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 -gate_projects: [] +# Each entry was named by the owner. +gate_projects: + - 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 # "Cymbal Bank" (fictional customer; voyage ENG-CYMBAL-001), owner 2026-11-16 poll_seconds: 30 stage_timeout_minutes: 360 # one stage, start to package -- 2.49.1