Compare commits

4 Commits

Author SHA1 Message Date
fb65ef9515 Runner: the system decides review points in the Cymbal Bank project
The owner named the KAB project "Cymbal Bank" (5a2da83b-...; voyage
ENG-CYMBAL-001) as one of Shakedown's own. There, the runner continues a
review point only on a critic pass with no failed step (plan 5b). Every
other project still waits for a person.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 05:14:00 +00:00
099f9a87e8 Runner: run a voyage remotely as the kab-automation identity
- shakedown_lib: runs F01 -> F02 -> F03 for one engagement on KAB with no
  personal token. Signs in as the kab-automation client (client credentials,
  provisioned by Stellwerk from crucible-platform-deploy) and renews the token
  itself.
- Each stage checks its workflow id against the registered sub-agent name,
  builds inputs from the case and upstream stages, starts the run, follows it
  and judges it with cp1-agents evaluate_case.py.
- Gates (plan 5b): in config/runner.yaml gate_projects only, continue on a
  critic pass with no failed step and a note saying the system decided.
  Never on a fail. Elsewhere, wait for a person and record who decided.
  gate_projects is empty for now.
- Ledger per voyage run, with --resume; nothing secret is written.
- Cymbal voyage ENG-CYMBAL-001; workflow registry; Dockerfile and an
  unapplied Job template; 15 tests against a fake KAB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 05:08:31 +00:00
07060dbb37 Plan §5b: end-to-end use cases that fix what fails (owner, 2026-11-15)
Name a customer: the system generates its use case and runs Discovery,
Assessment and Strategy remotely. It decides gates only in its own projects
and only on a critic pass, fixes failures on an agentic/<customer>-<date>
branch, builds and deploys over the shared agents (one branch at a time,
rolled back if the PR is closed), and raises one PR to main per repository.
AC8, §4, §5, §5a, §9 and §10 amended to match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 21:03:50 +00:00
0890bc5c31 Plan: one UI in the KAB Docs Portal, the nightly cycle, skills
Amended 2026-11-12 with the owner: Shakedown has no UI of its own; the KAB
Docs Portal is the one UI and Shakedown its visualisation layer for workflow
case outputs, so Shakedown serves an API that accepts the portal's realm-kab
tokens. Adds the nightly cycle (5a: harvest, generate, execute, qualify,
prove, raise pull requests automatically for next-morning review, within a
standing daily budget; never merge or deploy), skills Bosun's sub-agents work
through (5), phase 7, and the engagement-id rule from cp1-agents.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:28:36 +00:00
22 changed files with 1676 additions and 28 deletions

5
.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
ledger/
.env
__pycache__/
*.pyc
build/

72
README.md Normal file
View File

@@ -0,0 +1,72 @@
# Shakedown
Runs, rates and experiments on KAB workflows. The plan is [docs/implementation-plan.md](docs/implementation-plan.md).
What is built so far is the **runner**, which runs a voyage remotely with no person's token: F01 Discovery → F02
Assessment → F03 Strategy for one engagement.
## The runner
```sh
export KAB_CLIENT_SECRET=... # the kab-automation client's secret; never in a file
python3 -m shakedown_lib token-check
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml
python3 -m shakedown_lib run <voyage> --resume ledger/ENG-CYMBAL-001/<started> # pick up where it stopped
python3 -m unittest discover -s tests -t . # tests, against a fake KAB
```
**Identity.** The runner signs in as the confidential client `kab-automation` of realm `kab`, using client
credentials. Its tokens carry the automation user's email, which KAB makes an Editor. The Stellwerk orchestrator
provisions the client and its secret: crucible-platform-deploy, `kab-cluster/dev` component 85. The secret is held in
Key Vault `crucible-kab-dev-kv` as `kab-automation-client-secret`. A token is renewed a minute before it expires, so a
run that lasts hours needs nobody.
**What one stage does:**
1. **Check the workflow.** The stage's workflow id must still name its registered workflow (`workflows/*.yaml`, sub-agent pattern). Ids change on every re-import.
2. **Build the inputs.** They come from the case's context file and the voyage. Upstream execution ids, approval
references and package artifact ids (`<execution id>_step_<n>`) come from earlier stages. Each context
adapter delivers the file differently:
- `upload` (F01): uploaded as the run's file;
- `document` (F03): sent as text in one run input;
- `none` (F02): inputs come from the voyage.
3. **Check and start.** Before starting, it checks the required inputs, the engagement id (one per chain) and the
customer block, using cp1-agents' own helpers.
4. **Follow the run.** At a review point:
- in a project listed in `config/runner.yaml` `gate_projects`, the runner continues **only** when the critic's
verdict is `pass` and no step has failed. Its note says the system decided, not a person. Otherwise the pause is
left as it is and the voyage stops for a fix;
- in any other project, it waits for a person to decide in KAB and records who did.
5. **Finish.** It pulls the outputs, finds the package and judges the run against its case with cp1-agents
`scripts/evaluate_case.py`. A failed run, a missing package or a failed critical check stops the voyage.
**Ledger.** Each voyage run gets its own folder, `ledger/<voyage>/<started>/` (or `$SHAKEDOWN_LEDGER`). It holds:
- `state.json`;
- for each stage: its inputs (uploads by name only), the context as sent, the status, the steps and the evaluation.
No token or secret is written there.
**Exit status:**
| Code | Meaning |
|---|---|
| 0 | Done |
| 2 | Configuration error, or KAB refused |
| 3 | Stopped for a fix |
| 4 | Nobody decided a review point in time |
| 5 | A stage ran too long |
**cp1-agents** is read from a clone beside this repository, or from `$CP1_AGENTS_DIR`, and its commit is recorded
with every run. Nothing in cp1-agents is changed.
**On the cluster:** `deploy/Dockerfile` and `deploy/runner-job.yaml`, a Job template with placeholders. Nothing is
applied without approval.
## Not built yet
These come from the plan:
- Bosun (the deep agent);
- fixes on `agentic/*` branches;
- deploying from a branch;
- experiments and variations;
- the portal views;
- the nightly cycle.

View File

@@ -0,0 +1,40 @@
# Cymbal Bank (fictional; estate served by alz-discovery-sample): Discovery -> Assessment -> Strategy.
# The cases stay in cp1-agents and are read in place. Upstream ids are filled from the earlier stages' runs.
# The workflow ids are unverified: each is checked against its registry name (sub-agent pattern) before a stage
# starts, so a stale or individual-pattern id is refused. `--dry-run` checks them all without starting anything.
voyage: ENG-CYMBAL-001
capability: cp1-agentic-landing-zone
engagement_id: ENG-CYMBAL-001
project: 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 # KAB project "Cymbal Bank" (kab-dev)
stages:
- key: f01
workflow: alz-f01-discovery
workflow_id: 6aea4d516bc9323ca8d4bd12
# Run exec_e60e23f0a4ac passed its case and was approved on 2026-11-15, so it is adopted, not run again.
# Remove these two lines to run Discovery afresh from the context below.
use_execution: exec_e60e23f0a4ac
approval_ref: "Continued by a person in KAB on 2026-11-15 (Cymbal Discovery run approved)"
context: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/context.yaml"
case: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/case.yaml"
- key: f02
workflow: alz-f02-assessment
workflow_id: 6aecac756bc9323ca8d4be1c # from the case README; checked against the registry names
inputs:
discovery_execution_id: {from: f01, value: execution_id}
discovery_approval_ref: {from: f01, value: approval_ref}
artifact_ids: {from: f01, value: package_artifact_ids}
documents:
rate_card_document: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/rate-card.json"
case: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/case.yaml"
- key: f03
workflow: alz-f03-strategy
workflow_id: wf_0f80f01c-e423-4553-9ec0-afa21b3bbb52 # from the docs portal registry; checked by name
context: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/run-context.yaml"
inputs:
assessment_execution_id: {from: f02, value: execution_id}
assessment_approval_ref: {from: f02, value: approval_ref}
artifact_ids: {from: f02, value: package_artifact_ids}
case: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/case.yaml"

13
config/runner.yaml Normal file
View File

@@ -0,0 +1,13 @@
# Shakedown runner settings. Committed, so every change is reviewed; the environment can't widen them.
# The KAB projects whose review points the runner may decide itself (plan §5b): only Shakedown's own use-case
# projects. In any other project it waits for a person. It continues a gate only on a critic `pass` with no failed
# step (shakedown_lib/gate.py), never on a `fail`, and its note says the system decided.
#
# Each entry was named by the owner.
gate_projects:
- 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 # "Cymbal Bank" (fictional customer; voyage ENG-CYMBAL-001), owner 2026-11-16
poll_seconds: 30
stage_timeout_minutes: 360 # one stage, start to package
person_wait_hours: 24 # the workflows' own review timeout (hitl.timeout_hours)

16
deploy/Dockerfile Normal file
View File

@@ -0,0 +1,16 @@
# The Shakedown runner. Build from the repository root:
# buildah bud -f deploy/Dockerfile -t cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/runner:<version> .
ARG BASE=python:3.12-slim
FROM ${BASE}
RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY shakedown_lib/ shakedown_lib/
COPY config/ config/
COPY workflows/ workflows/
COPY capabilities/ capabilities/
RUN useradd -u 10001 -m shakedown && mkdir -p /ledger && chown shakedown /ledger
USER 10001
ENV CP1_AGENTS_DIR=/cp1-agents SHAKEDOWN_LEDGER=/ledger PYTHONUNBUFFERED=1
ENTRYPOINT ["python3", "-m", "shakedown_lib"]

52
deploy/runner-job.yaml Normal file
View File

@@ -0,0 +1,52 @@
# One voyage run as a Kubernetes Job in namespace `shakedown`. A template: placeholders are filled at render time,
# and nothing here is applied without the owner's approval.
#
# Provisioned outside this repository by the Stellwerk orchestrator (crucible-platform-deploy, kab-cluster/dev):
# - the `shakedown` namespace and the PVC `shakedown-ledger`;
# - Secret `shakedown-kab-automation`, key `client-secret`: the kab-automation client's secret, from Key Vault
# crucible-kab-dev-kv `kab-automation-client-secret`. The runner reads it from the environment only;
# - Secret `shakedown-gitea-read`, key `token`: read access to agents/cp1-agents, for the pinned clone.
apiVersion: batch/v1
kind: Job
metadata:
generateName: shakedown-voyage-
namespace: shakedown
labels: {app.kubernetes.io/name: shakedown-runner, app.kubernetes.io/part-of: cp1-shakedown}
spec:
backoffLimit: 0 # a failed run is resumed on purpose (--resume), never retried blind
activeDeadlineSeconds: 172800 # two days: three stages plus waiting for people at review points
template:
metadata:
labels: {app.kubernetes.io/name: shakedown-runner}
spec:
restartPolicy: Never
securityContext: {runAsNonRoot: true, runAsUser: 10001, fsGroup: 10001}
initContainers:
- name: clone-cp1-agents
image: <placeholder_for_runner_image>
command: ["sh", "-c"]
args:
- >-
git -c http.extraHeader="Authorization: token $GITEA_TOKEN" clone --quiet
https://gitea.kyndemo.live/agents/cp1-agents.git /cp1-agents &&
git -C /cp1-agents checkout --quiet <placeholder_for_cp1_agents_commit>
env:
- name: GITEA_TOKEN
valueFrom: {secretKeyRef: {name: shakedown-gitea-read, key: token}}
volumeMounts: [{name: cp1-agents, mountPath: /cp1-agents}]
containers:
- name: runner
image: <placeholder_for_runner_image>
args: ["run", "<placeholder_for_voyage_file>"]
env:
- name: KAB_CLIENT_SECRET
valueFrom: {secretKeyRef: {name: shakedown-kab-automation, key: client-secret}}
resources:
requests: {cpu: 50m, memory: 128Mi}
limits: {cpu: 500m, memory: 512Mi}
volumeMounts:
- {name: cp1-agents, mountPath: /cp1-agents, readOnly: true}
- {name: ledger, mountPath: /ledger}
volumes:
- {name: cp1-agents, emptyDir: {}}
- {name: ledger, persistentVolumeClaim: {claimName: shakedown-ledger}}

View File

@@ -6,8 +6,9 @@ enters service. This app does the same for every workflow before its outputs rea
| | |
|---|---|
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
| Status | Draft. Nothing is built yet |
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) |
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
| Status | Draft. Built so far (2026-11-16): the voyage **runner** (README), which runs F01 → F02 → F03 remotely as the `kab-automation` identity provisioned by Stellwerk, with the §5b gate rule in code |
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
| Runs on | The `crucible-kab-dev-aks` cluster |
| Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource |
@@ -40,7 +41,7 @@ without anyone pointing it at them.
## 2. What Shakedown does
Shakedown does four things, each usable through the UI and by the deep agent (§5):
Shakedown does four things, each usable through the portal's Shakedown views and by the deep agent (§5):
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
@@ -81,7 +82,8 @@ experiment's record links to its runs by execution id.
```
cp1-shakedown/
├── CLAUDE.md, README.md, .agents/PLAN.md, docs/ (this plan, adr/)
├── app/ # FastAPI API + UI
├── app/ # FastAPI API only: the views live in the KAB Docs Portal (portal ADR 0005)
├── skills/<skill>/SKILL.md # versioned skills Bosun's sub-agents work through (§5, portal ADR 0010)
├── bosun/ # the deep agent
│ ├── agent.py # create_deep_agent(...): tools, sub-agents, interrupt_on, backend
│ ├── subagents/ # one module per sub-agent: prompt + tool list
@@ -299,7 +301,7 @@ Shakedown checks each link four ways:
| AC5 Independent critic | Each stage's critic runs on a different model from its executors (from `agents.json`), or the deliverable says it doesn't |
| AC6 One binding budget | One token ceiling binds each run, checked against `usage.json` |
| AC7 Truthful deliverables | The provider, critic verdict, change count and run status each report states match the run record |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output |
| AC8 Human decisions stay human | Every gate decision in the ledger comes from a person's review action, never from an agent's output. *Amended 2026-11-15:* in Shakedown's own use-case projects, the system may decide a gate under the rule in §5b, and the ledger records that decision as the system's, never as a person's |
**Rules for linked workflows:**
- A voyage uses one binding for each stage, all from the same pattern. Individual F01 feeds F02; the sub-agent F01 is
@@ -313,18 +315,18 @@ Shakedown checks each link four ways:
**Other capabilities** onboard the same way: a `capability.yaml` built from the Epic, its workflows registered and its
links declared. CP4 Epics are next after the landing zone.
**In the UI,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
**In the portal's Shakedown views,** the capability view is the home page. It shows the chain, each stage's state and binding, the health of
each link, the last voyage for each engagement, and coverage: the use cases and expectations for each stage, and the
Epic and Feature acceptance criteria that have no check yet.
## 4. Architecture on the cluster
```
you (browser) ── Keycloak sign-in (realm kab) ──┐
▼
you (browser) ── KAB Docs Portal (UI + BFF session, Keycloak realm kab) ──┐
bearer token (you) ▼
┌──────────────────── namespace: shakedown ──────────────────────────────────────┐
│ shakedown-app (FastAPI + UI, one replica) │
│ ├─ session: your KAB tokens, held server-side in memory, renewed by refresh │
│ shakedown-api (FastAPI, no UI; scheduler for the nightly cycle, §5a) │
│ ├─ acts as you: the portal's token, validated per request; never stored │
│ ├─ Bosun deep agent (deepagents / LangGraph), checkpointed to the ledger │
│ │ └─ Claude Sonnet 5.5 ──► Kyndryl Azure resource (key in a Secret) │
│ ├─ git: read-only clones of cp1-agents and shakedown; PRs via gitea API │
@@ -334,19 +336,22 @@ Epic and Feature acceptance criteria that have no check yet.
│ HTTPS: api.kab-dev… │ A2A :8080: agents.svc │ read-only RBAC: pods/log in `agents`
```
- **Sign-in and tokens:** Shakedown is an OIDC client of KAB's Keycloak (`id.kab-dev.kyndemo.live`, realm `kab`).
The access and refresh tokens are held server-side in memory for the session, never written to disk, the PVC or the
ledger, and never shown to the browser or to Bosun. Bosun's KAB tool calls go through the session, acting as you.
This replaces `kab_token.py` for everything Shakedown does.
- **Sign-in and tokens (amended 2026-11-12):** people sign in once, in the KAB Docs Portal. Its BFF holds the
session (portal ADR 0007) and calls Shakedown's API with the signed-in person's access token. Shakedown validates
that token against realm `kab` on every request and uses it, acting as that person, for KAB calls. It never stores
it, writes it to the PVC or the ledger, or shows it to Bosun. The nightly cycle (§5a) runs under a service identity
of its own with KAB rights limited to the cycle's projects. This replaces `kab_token.py` for everything Shakedown
does.
- **Reading the cluster:** a ServiceAccount with read-only RBAC (`get`/`list` on pods and `pods/log` in `agents`)
replaces `collect_run_usage.py`'s kubeconfig. It gets no `secrets` access.
- **The UI is exposed through an ingress from Phase 0**, with Keycloak sign-in in front of it. There's no
port-forward stage, so the Keycloak client is Phase 0's first dependency on the platform team.
- **The API is exposed through an ingress from Phase 0**, accepting only realm-`kab` bearer tokens issued to the
portal's client. There's no Shakedown UI to expose: the portal shows Shakedown's views.
- **Shadow agents:** these are applied to the `shakedown` namespace, beside Shakedown and apart from the live agents
in `agents`. They
reach the gateways through the same `deploy/` profile values. Shakedown's ServiceAccount **cannot** apply
manifests: it renders the overlay, and a person applies it after approving that change. This keeps cp1-agents'
rule that each cluster change is approved.
rule that each cluster change is approved. *Amended 2026-11-15:* end-to-end use cases (§5b) are the exception.
There the system builds agents from its branch and deploys them over the shared agents in `agents`.
- **Model:** Claude Sonnet 5.5 is called on the Kyndryl Azure resource directly, not through the cluster's LiteLLM,
which doesn't serve it. The key is in Secret `shakedown-secrets`, created by hand. Phase 0 proves the call path
from LangChain to the Azure-hosted Claude endpoint before anything else is built on it.
@@ -372,10 +377,10 @@ them as follows:
| Deep Agents feature | Used for |
|---|---|
| Planning tool | Navigator's experiment plan, shown live in the UI |
| Planning tool | Navigator's experiment plan, shown live in the portal |
| File workspace (backend) | A composite backend: scratch state per thread, plus read-only routes onto the two repository clones and the ledger folder of the run being studied |
| Sub-agents | The six roles below, each with only the tools its job needs, which keeps the main agent's context small |
| `interrupt_on` | Approval in the UI before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant` |
| `interrupt_on` | Approval in the portal before `start_run`, `start_replay_batch`, `open_pull_request` and `render_variant`, except inside the nightly cycle's standing approval (§5a) |
| LangGraph checkpointer | A long experiment survives a pod restart, and its whole trail can be audited afterwards |
| Sub-agent | Job | Tools | Writes |
@@ -390,13 +395,30 @@ them as follows:
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
the judge is visible in its scores.
**Skills, not prompt rules (amended 2026-11-12, portal ADR 0010).** Sub-agents do repeatable work through versioned
skills in `skills/` (a `SKILL.md` with the procedure, plus scripts for anything checkable in code), so best practice,
quality controls and integration specifics are applied the same way every time:
| Family | Skills (first set) | Used by |
|---|---|---|
| Quality controls | case authoring (one variable, expected direction); qualification (checks → invariants → rubric, with quoted evidence); spread before a finding; pull-request evidence pack | Chartmaker, Surveyor, Navigator, Shipwright |
| Integration specifics | KAB API (tenant and project headers, gates paused, never answered); run-context and uploads for each workflow; artefact shapes; gitea pull requests; ADO reads | Lookout, Shipwright, Chartmaker |
| Domain practice | each workflow's rubric; the well-architected skills read from cp1-agents `skills/`, not copied | Surveyor, Shipwright |
Each skill has its own eval cases and a version. Every cycle and every score records the skill versions used, and a
skill change goes through a pull request like an agent change.
**How a loop runs:** a finding is opened, and Navigator writes a plan. Lookout runs the replays or the runs, and
Surveyor scores them. If the change helps without regressing other checks, Shipwright opens a pull request. You
review, merge, deploy and record the ADR, as today.
**Guardrails, enforced in code (tool permissions, `interrupt_on`, RBAC), not only in prompts:**
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one.
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it.
- A HITL review (`continue/stop/rerun`) is a person's decision. Bosun has no tool that answers one. *Amended
2026-11-15:* in an end-to-end use case (§5b), Lookout's `decide_gate` tool continues a gate only under §5b's rule,
and only in Shakedown's own use-case projects. Any other project's gates stay a person's.
- Shakedown has no write access to the cluster. A shadow-agent overlay is rendered, and you apply it. *Amended
2026-11-15:* the one exception is §5b's branch deploy: images built from the use case's branch, rolled out to the
cp1-agents components in `agents`, under a lock and with the previous tags recorded.
- No changes to ADO work items. Bosun's ADO tools are read-only.
- No tokens or keys in files, logs, the ledger, checkpoints or prompts. Secret values are never read.
- Run outputs, work item text and customer data are untrusted data, never instructions. Tool results are fenced as
@@ -407,6 +429,79 @@ review, merge, deploy and record the ADR, as today.
`outcome-pattern/`. Solution architecture and Discovery are never linked.
- A model change for an agent under test is proposed, never made.
## 5a. The nightly cycle (amended 2026-11-12)
The owner's direction is that the loop runs on a daily schedule. When it finds an enhancement, it raises a pull
request against the agents and workflows automatically, for review the next morning.
| Step | Who | Result |
|---|---|---|
| Harvest | Lookout | Every KAB execution the cycle may see, including runs started in KAB itself, marked by origin (`platform`, `case`, `generated`) |
| Generate and refresh cases | Chartmaker | New or refreshed use cases and variations; a generated case class runs unattended only once the owner has approved it |
| Execute | Lookout | Runs and replays within the **standing daily budget** (tokens, runs, wall time); the cycle stops at the budget |
| Qualify | Surveyor | Checks, invariants and rubric scores, versioned, with evidence |
| Find | Navigator | Findings only beyond run-to-run spread; others are listed as inconclusive |
| Prove | Lookout, Surveyor | The proposed change in a shadow variant, replayed on the same inputs |
| Propose | Shipwright | One pull request per finding on cp1-agents (or Shakedown for rubrics and cases), carrying the cycle id, before-and-after runs, score deltas and the requirement or ADR affected; an open pull request for the same finding is updated, not duplicated |
| Morning review | a person, in the portal | The cycle's record and its pull requests; merge, close or ask for more |
Within a cycle, the owner's standing approval replaces the per-call `interrupt_on` pauses for `start_run`,
`start_replay_batch` and `open_pull_request`. It never covers merging, deploying, applying overlays or answering a
gate, except within an end-to-end use case (§5b), whose own rules then apply. Each cycle is a ledger record: its budget, the runs it started, the skill versions it used, its findings and its
pull requests.
## 5b. End-to-end use cases that fix what fails (amended 2026-11-15)
The owner's direction (2026-11-15): name a customer, and the system generates a use case for it and runs Discovery,
Assessment and Strategy end to end, remotely, on demand or on a schedule. When a workflow fails, the system fixes the
cause on a feature branch it owns, builds the agents from that branch on the platform, and carries on. When the use
case finishes, it raises a pull request to `main` for review. This is what the manual Cymbal Bank case of 2026-11-15
took a person a day to do (§1).
**Owner decisions (2026-11-15):**
| Decision | Choice |
|---|---|
| Gates | The system decides them, **only** in its own use-case projects. It continues a gate when the run completed, the stage's critic verdict is `pass` and the code checks hold. Otherwise it stops the gate and opens a fix. It never continues a gate on a critic `fail` |
| Agents built from the branch | Built on the platform and deployed **over the shared agents** in `agents`, not as shadow agents |
| Branches | **One per use case**, `agentic/<customer-slug>-<yyyymmdd>`, owned by the system's gitea identity; one pull request to `main` per repository when the use case ends |
| Home | Shakedown (Bosun): Chartmaker, Lookout, Surveyor and Shipwright gain the steps below |
**How a use case runs:**
| Step | Who | What |
|---|---|---|
| 1. Generate | Chartmaker | From the customer's name and brief: the F01 context, F02 rate card and F03 run context, one `engagement_id` (`ENG-<SEGMENTS>`, F01 ADR 0022), and the Discovery source. The source is either one that exists (a live cloud connection, or an estate the sample server already serves) or a new estate added to `alz-discovery-sample` as data on the branch (F01 ADR 0025). Expectations are written before the first run. The case goes to `cp1-shakedown` on the branch |
| 2. Set up | Lookout | A KAB project for the customer in CP1, the three workflows associated with it, and the case registered in the portal's `bindings.yaml` (on a branch of the portal) |
| 3. Run | Lookout | F01, then F02 from the approved F01 package, then F03 from the approved F02 package; each started from its context file (`--context`) |
| 4. Decide each gate | Lookout | By the gate rule above. Each decision carries a note naming the system, the stage's critic verdict and the run that was checked |
| 5. Fix on failure | Surveyor, then Shipwright | Surveyor sorts the failure into one kind: a step refused its input (schema); the critic misread a rule; an agent broke a rule (wrong or missing citations); a data or server bug; the platform (auth, quota, an agent not reachable). Shipwright changes the prompt, schema, hook or server data on the branch, with a release note and version bump, and runs `check_agents.py` and the agent's tests. It commits with the failing run's id and the reason |
| 6. Build and deploy | Lookout | Builds the changed agents' images from the branch, in parallel, and rolls them out over the shared agents. Then it reruns the stage that failed, not the whole chain: an approved upstream stage is kept |
| 7. Prove | Surveyor | A fix holds only when the rerun passes. Because models vary from run to run, a fix to a critic rule needs two passing runs |
| 8. Finish | Shipwright | One pull request to `main` per repository (`cp1-agents`, `cp1-shakedown`, the portal if bindings changed). It lists every failure, its kind, the commit that fixed it, and the run that proved it; the versions now deployed; and anything left unresolved |
**Limits and safety:**
- **Three fix attempts per stage,** and the daily budget (§5a). When either runs out, the use case stops and the pull
request is raised with the failure marked unresolved.
- **One branch deploys at a time.** Shared agents run one branch's code, so a lock in the ledger allows one use
case to deploy at a time. Others wait or run without deploying.
- **The shared agents run branch code until the pull request is decided.** The portal shows which agents are
deployed from which branch. If the pull request is closed unmerged, Lookout rolls those agents back to the image
tags it recorded before the first deploy. Merging puts `main` back in line with what's deployed.
- **What stays a person's:** merging a pull request; anything in another project, including gates; kamp-demo
(read-only); platform-admin actions (claims, registrations owned by another tenant); a model change for an agent;
accepting an ADR.
- **What the system never changes:** secrets (it reads none); `main`; ADO work items.
**What it needs from the platform (blocks the build):**
1. A **service identity** in realm `kab` (client credentials) with CP1 rights to create projects, start runs and
decide gates in its own projects. Personal tokens can't run unattended.
2. A **gitea bot** that can push `agentic/*` branches and open pull requests on `cp1-agents`, `cp1-shakedown` and
the portal, and nothing more.
3. **Cluster rights** for Shakedown's ServiceAccount: push to `cruciblecoredevacr.azurecr.io/kab/cp1/*`, and
`patch` on the Deployments of cp1-agents components in `agents` (image only). No `secrets` access, nothing else.
4. **In-cluster image builds** (buildah or kaniko Jobs), because there is no laptop in the loop.
## 6. Business value: rating quality, not volume
Schema validity and passing checks prove that an output is well-formed. They don't prove that anyone can act on it.
@@ -437,13 +532,14 @@ repository exists.
| # | Phase | Delivers | Verified by |
|---|---|---|---|
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; API ingress hostname registered, accepting the portal's realm-`kab` tokens; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
| 1 | **Ledger and rating** | Workflow registry with F01, F02 and F03 (§3c); backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail served by the API and shown in the portal | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
| 4 | **Capability, links, value and variations** | `cp1-agentic-landing-zone` capability synced from Epic 3214440; links F01 → F02 → F03 checked statically, for drift and for use; voyages with ids resolved from upstream runs; chain-level checks from Epic AC1 to AC8; ripple variations; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F01 → F02 → F03 with no id typed by hand, and its AC1 walk traces an F03 decision back to a Discovery record or lists where the chain breaks; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
| 6 | **Scale out** | F04 to F07 as they are built, each linked into the capability; the sub-agent pattern as a separate binding; then CP4 Epics as capabilities of their own | A new workflow is onboarded by adding config only (its registry file, use cases and rubric), and a new capability by its `capability.yaml` and `links.yaml` |
| 7 | **Nightly cycle** (§5a) | Scheduler and cycle records; standing daily budget; skills library with eval cases; harvest of platform runs; automatic pull requests with evidence packs; morning review in the portal | Seven nights in a row each end within budget, with a cycle record; every pull request raised carries its evidence and cites its cycle; none is merged without a person |
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
the experimentation loop pays back.
@@ -482,13 +578,19 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
| Hosting | The crucible cluster |
| Shadow agents | The `shakedown` namespace |
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
| UI access | ~~Ingress with Keycloak sign-in~~ Amended 2026-11-12: no UI of its own. The KAB Docs Portal is the one UI and Shakedown its case-output visualisation layer (portal ADR 0005); Shakedown serves an API |
| Nightly cycle (2026-11-12) | Daily schedule; a confirmed enhancement raises a pull request automatically for next-morning review; never merged or deployed automatically (§5a, portal ADR 0009) |
| Skills (2026-11-12) | Sub-agents work through versioned skills for quality controls, integration specifics and domain practice (§5, portal ADR 0010) |
| End-to-end use cases (2026-11-15) | Generate a customer's use case and run F01 → F02 → F03 remotely; the system decides gates in its own projects on a critic pass, fixes failures on its own `agentic/<customer>-<date>` branch, builds and deploys over the shared agents, and raises one pull request to `main` at the end (§5b) |
| Engagement ids (2026-11-12) | One `engagement_id` per customer engagement from F01 on, enforced in cp1-agents (F01 ADR 0022, F02 ADR 0016, F03 ADR 0009); voyages key by it |
**Still open:**
1. **Keycloak client and hostname (platform ask, blocks Phase 0):** the KAB platform team registers a `cp1-shakedown`
OIDC client in realm `kab`, with the ingress hostname as its redirect URL. That also needs a DNS name and a TLS
certificate under `kyndemo.live`.
1. **Keycloak client and hostname (platform ask):** one client for the portal (its BFF) in realm `kab`, whose tokens
Shakedown's API accepts, plus a service identity for the nightly cycle. That needs DNS names and TLS certificates
under `kyndemo.live` for the API (and the portal once it is shared).
5. **Nightly cycle settings:** the daily budget figures, the time window, which generated case classes run
unattended, and where the skills library lives (here, or shared with cp1-agents).
2. **Gitea bot account:** a gitea token that Bosun uses to open pull requests on `cp1-shakedown` and `cp1-agents`,
held in a Secret, with rights limited to opening pull requests.
3. **Variant agent keys (to verify with KAB):** whether a variant in `shakedown` may reuse the agent's
@@ -498,7 +600,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
## 10. Out of scope
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests.
- Approving HITL reviews, deploying production agents, applying cluster changes, or merging pull requests, except as
§5b allows for end-to-end use cases in Shakedown's own projects. Merging stays a person's in every case.
- Writing to ADO work items.
- Replacing the critic agents inside the workflows. Shakedown evaluates the critic along with everything else.
- Moving the existing cases out of cp1-agents.

2
requirements.txt Normal file
View File

@@ -0,0 +1,2 @@
pyyaml>=6
jsonschema>=4 # cp1-agents scripts/check_data_contract.py, used by evaluate_case.py

View File

@@ -0,0 +1 @@
"""Shakedown: runs, rates and experiments on KAB workflows (docs/implementation-plan.md). This package holds the runner."""

76
shakedown_lib/__main__.py Normal file
View File

@@ -0,0 +1,76 @@
"""Shakedown runner: run a voyage's workflows on KAB remotely, as the kab-automation identity.
python3 -m shakedown_lib token-check
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run
python3 -m shakedown_lib run <voyage.yaml> [--only f02 f03]
python3 -m shakedown_lib run <voyage.yaml> --resume ledger/ENG-CYMBAL-001/<started>
Environment: KAB_CLIENT_SECRET (required), KAB_CLIENT_ID (kab-automation), KAB_TOKEN_URL, KAB_API, KAB_TENANT,
CP1_AGENTS_DIR (the cp1-agents clone, default ../cp1-agents), SHAKEDOWN_LEDGER (default ./ledger).
Which projects the runner may decide gates in is set only in config/runner.yaml.
Exit status: 0 done, 2 configuration or KAB refused, 3 stopped for a fix, 4 nobody decided a review point in time,
5 a stage ran too long.
"""
from __future__ import annotations
import argparse
import base64
import json
import os
import sys
from pathlib import Path
from .auth import AuthError, ClientCredentials
from .kab import CP1_TENANT, DEFAULT_API, Kab
from .ledger import Ledger, default_root
from .runner import EXIT_CONFIG, Runner, Settings
from .voyage import VoyageError, load_config, load_voyage
def _claims(token: str) -> dict:
try:
payload = token.split(".")[1]
return json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4)))
except (IndexError, ValueError):
return {}
def token_check() -> int:
tokens = ClientCredentials.from_env()
claims = _claims(tokens.token())
shown = {k: claims.get(k) for k in ("iss", "azp", "email", "preferred_username", "exp")}
print(f"Token issued to {tokens.client_id}, valid for {tokens.expires_in()} s. Claims: {json.dumps(shown)}")
return 0
def run(args) -> int:
voyage = load_voyage(Path(args.voyage))
settings = Settings.from_config(load_config())
kab = Kab(ClientCredentials.from_env(), api=os.getenv("KAB_API", DEFAULT_API),
tenant=os.getenv("KAB_TENANT", CP1_TENANT), project=voyage.project)
ledger = Ledger(Path(args.resume)) if args.resume else Ledger.new(default_root(), voyage.name)
return Runner(kab=kab, ledger=ledger, voyage=voyage, settings=settings).run(dry_run=args.dry_run, only=args.only)
def main(argv=None) -> int:
parser = argparse.ArgumentParser(prog="shakedown_lib", description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
commands = parser.add_subparsers(dest="command", required=True)
commands.add_parser("token-check", help="get a token as kab-automation and show who it names")
start = commands.add_parser("run", help="run a voyage")
start.add_argument("voyage")
start.add_argument("--dry-run", action="store_true", help="build and check every stage's inputs; start nothing")
start.add_argument("--resume", help="a ledger folder of an earlier run of this voyage")
start.add_argument("--only", nargs="+", help="run only these stages (earlier ones must be completed)")
args = parser.parse_args(argv)
try:
return token_check() if args.command == "token-check" else run(args)
except (AuthError, VoyageError, FileNotFoundError, RuntimeError) as error:
print(f"Refused: {error}", file=sys.stderr)
return EXIT_CONFIG
if __name__ == "__main__":
sys.exit(main())

80
shakedown_lib/auth.py Normal file
View File

@@ -0,0 +1,80 @@
"""KAB access tokens for unattended runs: the `kab-automation` client of realm `kab`, by client credentials.
The client and its secret are provisioned by the Stellwerk orchestrator (crucible-platform-deploy, `kab-cluster/dev`
component 85; module `kab-identity`). The secret is in Key Vault `crucible-kab-dev-kv` as
`kab-automation-client-secret` and reaches the runner as the environment variable KAB_CLIENT_SECRET (a mounted
Secret on the cluster). Tokens carry the fixed email claim KAB knows the automation user by.
The secret and every token stay in memory: never written to a file, the ledger or a log, and never printed.
"""
from __future__ import annotations
import json
import os
import time
import urllib.error
import urllib.parse
import urllib.request
from typing import Callable
DEFAULT_TOKEN_URL = "https://id.kab-dev.kyndemo.live/realms/kab/protocol/openid-connect/token"
DEFAULT_CLIENT_ID = "kab-automation"
REFRESH_MARGIN_S = 60
class AuthError(RuntimeError):
pass
def _post_form(url: str, form: dict, timeout: int = 30) -> dict:
body = urllib.parse.urlencode(form).encode()
request = urllib.request.Request(url, data=body, headers={"Content-Type": "application/x-www-form-urlencoded"})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.load(response)
except urllib.error.HTTPError as error:
try:
reason = json.loads(error.read() or b"{}").get("error_description") or "no reason given"
except ValueError:
reason = "no reason given"
raise AuthError(f"token request refused: HTTP {error.code}: {reason}") from None
except urllib.error.URLError as error:
raise AuthError(f"token endpoint unreachable: {error.reason}") from None
class ClientCredentials:
"""A token source that renews itself a minute before the token expires."""
def __init__(self, client_id: str, secret: str, token_url: str = DEFAULT_TOKEN_URL,
post: Callable[[str, dict], dict] = _post_form, clock: Callable[[], float] = time.time) -> None:
if not secret:
raise AuthError("no client secret: set KAB_CLIENT_SECRET (the kab-automation client's secret)")
self.client_id = client_id
self._secret = secret
self.token_url = token_url
self._post = post
self._clock = clock
self._token: str | None = None
self._expires_at = 0.0
@classmethod
def from_env(cls) -> "ClientCredentials":
return cls(os.getenv("KAB_CLIENT_ID", DEFAULT_CLIENT_ID), os.getenv("KAB_CLIENT_SECRET", ""),
os.getenv("KAB_TOKEN_URL", DEFAULT_TOKEN_URL))
def __repr__(self) -> str: # never show the secret or the token
return f"ClientCredentials(client_id={self.client_id!r}, token_url={self.token_url!r})"
def token(self, force: bool = False) -> str:
if force or self._token is None or self._clock() >= self._expires_at - REFRESH_MARGIN_S:
reply = self._post(self.token_url, {"grant_type": "client_credentials", "client_id": self.client_id,
"client_secret": self._secret})
if not reply.get("access_token"):
raise AuthError("token reply has no access_token")
self._token = reply["access_token"]
self._expires_at = self._clock() + float(reply.get("expires_in") or 300)
return self._token
def expires_in(self) -> int:
return max(0, int(self._expires_at - self._clock()))

View File

@@ -0,0 +1,59 @@
"""The cp1-agents scripts Shakedown reuses, imported by path from a clone (plan §3: no change to cp1-agents).
`kab_run_workflow.py` gives the run-context helpers (load_context, new_ids, context_uploads, upload_type,
customer_problem, with_run_metadata, check_engagement) and `evaluate_case.py` judges a run against a case. The clone's
commit is recorded with every run, so a result can be reproduced.
"""
from __future__ import annotations
import importlib.util
import os
import subprocess
import sys
from pathlib import Path
from types import ModuleType
DEFAULT_DIR = Path(__file__).resolve().parents[2] / "cp1-agents"
def clone_dir() -> Path:
path = Path(os.getenv("CP1_AGENTS_DIR") or DEFAULT_DIR)
if not (path / "scripts" / "kab_run_workflow.py").is_file():
raise RuntimeError(f"no cp1-agents clone at {path}: set CP1_AGENTS_DIR")
return path
def _load(name: str) -> ModuleType:
scripts = clone_dir() / "scripts"
if str(scripts) not in sys.path:
sys.path.insert(0, str(scripts)) # evaluate_case imports check_data_contract beside it
spec = importlib.util.spec_from_file_location(f"cp1_agents_{name}", scripts / f"{name}.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
_cache: dict[str, ModuleType] = {}
def module(name: str) -> ModuleType:
if name not in _cache:
_cache[name] = _load(name)
return _cache[name]
def run_workflow() -> ModuleType:
return module("kab_run_workflow")
def evaluate_case() -> ModuleType:
return module("evaluate_case")
def commit() -> str | None:
try:
return subprocess.run(["git", "-C", str(clone_dir()), "rev-parse", "HEAD"], capture_output=True, text=True,
check=True).stdout.strip()
except (subprocess.CalledProcessError, FileNotFoundError, RuntimeError):
return None

97
shakedown_lib/gate.py Normal file
View File

@@ -0,0 +1,97 @@
"""The gate rule for end-to-end use cases (plan §5b), decided in code from what the run recorded.
At a review pause the system may continue only when, for the steps up to the pause:
- no step reported `failed` or `error`, and
- the stage's latest critic verdict is exactly `pass`.
Anything else (a `fail`, an unparseable or missing verdict, a failed step) is never continued: the pause is left for a
person and the use case stops for a fix. Whether the system may decide at all depends on the project (the runner's
`gate_projects`), not on this module.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
FAILED = {"failed", "error"}
def parsed(value: Any) -> Any:
if isinstance(value, str) and value[:1] in "{[":
try:
return json.loads(value)
except ValueError:
return value
return value
def unwrap(value: Any) -> Any:
"""A step's own output, out of KAB's wrappers (`response`, `output`, `result`, `data`), JSON strings decoded."""
value = parsed(value)
for _ in range(4):
if isinstance(value, dict) and "phase" not in value and "manifest" not in value:
inner = next((value[k] for k in ("response", "output", "result", "data") if k in value), None)
if inner is None:
break
value = parsed(inner)
else:
break
return value
def step_outputs(steps: Any) -> Dict[str, Any]:
"""step id -> its output, from `/executions/{id}/steps` (a mapping by step id, or a list of step records)."""
body = steps.get("steps", steps) if isinstance(steps, dict) else steps
if isinstance(body, dict):
return {str(k): unwrap(v) for k, v in body.items() if k != "execution_id"}
out: Dict[str, Any] = {}
for item in body or []:
if isinstance(item, dict):
key = item.get("step_id") or item.get("id")
if key is not None:
out[str(key)] = unwrap(item.get("output", item.get("result", item)))
return out
def template_order(status: dict) -> List[str]:
return [str(s.get("id")) for s in ((status.get("template") or {}).get("steps") or [])]
def _verdict(value: Any) -> Optional[str]:
if isinstance(value, dict):
value = value.get("verdict")
return value if isinstance(value, str) else None
@dataclass
class Decision:
proceed: bool
reason: str
verdict: Optional[str]
verdict_step: Optional[str]
def decide(status: dict, steps: Any, paused_step: Optional[str]) -> Decision:
if str(status.get("status", "")).lower() in FAILED:
return Decision(False, f"run status is {status.get('status')}", None, None)
outputs = step_outputs(steps)
order = template_order(status) or list(outputs)
if paused_step in order:
order = order[: order.index(paused_step) + 1]
verdict, verdict_step = None, None
for step_id in order:
out = outputs.get(step_id)
if not isinstance(out, dict):
continue
if str(out.get("status", "")).lower() in FAILED:
return Decision(False, f"step {step_id} reported {out.get('status')}", verdict, verdict_step)
found = _verdict(out.get("verdict"))
if found is not None:
verdict, verdict_step = found, step_id
if verdict is None:
return Decision(False, "no critic verdict before the review point", None, None)
if verdict != "pass":
return Decision(False, f"critic verdict is {verdict!r} (step {verdict_step})", verdict, verdict_step)
return Decision(True, f"critic verdict pass (step {verdict_step}), no failed step", verdict, verdict_step)

104
shakedown_lib/kab.py Normal file
View File

@@ -0,0 +1,104 @@
"""The KAB API, as the automation identity: the same endpoints `cp1-agents/scripts/kab_run_workflow.py` uses.
Every call takes a fresh token from the token source, so a run that lasts hours never needs a person. A 401 renews
the token once; 429 and 5xx are retried with back-off. Errors raise KabError; nothing calls sys.exit.
"""
from __future__ import annotations
import json
import time
import urllib.error
import urllib.request
from typing import Any, Callable, Protocol
DEFAULT_API = "https://api.kab-dev.kyndemo.live"
CP1_TENANT = "6ae37d13999438461a529409"
RETRIES = 4
class TokenSource(Protocol):
def token(self, force: bool = False) -> str: ...
class KabError(RuntimeError):
def __init__(self, message: str, status: int | None = None) -> None:
super().__init__(message)
self.status = status
Transport = Callable[[str, str, dict, bytes | None], tuple[int, bytes]]
def _urllib_transport(method: str, url: str, headers: dict, data: bytes | None) -> tuple[int, bytes]:
request = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(request, timeout=120) as response:
return response.status, response.read()
except urllib.error.HTTPError as error:
return error.code, error.read() or b""
except urllib.error.URLError as error:
return 599, str(error.reason).encode()
class Kab:
def __init__(self, tokens: TokenSource, api: str = DEFAULT_API, tenant: str = CP1_TENANT,
project: str | None = None, transport: Transport = _urllib_transport,
sleep: Callable[[float], None] = time.sleep) -> None:
self.tokens = tokens
self.api = api.rstrip("/")
self.tenant = tenant
self.project = project
self._transport = transport
self._sleep = sleep
def call(self, method: str, path: str, body: dict | None = None) -> Any:
data = json.dumps(body).encode() if body is not None else None
force = False
for attempt in range(RETRIES + 1):
headers = {"Authorization": f"Bearer {self.tokens.token(force=force)}", "X-Tenant-ID": self.tenant,
"Content-Type": "application/json"}
if self.project:
headers["X-Project-ID"] = self.project
status, raw = self._transport(method, f"{self.api}{path}", headers, data)
if 200 <= status < 300:
return json.loads(raw) if raw else {}
if status == 401 and not force:
force = True
continue
if (status == 429 or status >= 500) and attempt < RETRIES:
self._sleep(min(60, 5 * 2 ** attempt))
continue
raise KabError(f"{method} {path}: HTTP {status}: {raw[:400].decode(errors='replace')}", status)
raise KabError(f"{method} {path}: gave up after {RETRIES} retries")
# The calls the runner makes.
def workflow(self, workflow_id: str) -> dict:
found = self.call("GET", f"/api/v1/workflow-catalog/workflows/{workflow_id}")
return found.get("workflow", found)
def api_sample(self, workflow_id: str) -> dict:
return self.call("GET", f"/api/workflows/{workflow_id}/api-sample")
def project_name(self, project_id: str) -> str | None:
try:
found = self.call("GET", f"/api/v1/projects/{project_id}")
except KabError:
return None
return (found.get("project") or found).get("name")
def execute(self, template: dict, inputs: dict) -> dict:
return self.call("POST", "/api/workflows/execute", {"template": template, "inputs": inputs})
def status(self, execution_id: str) -> dict:
return self.call("GET", f"/api/workflows/status/{execution_id}")
def steps(self, execution_id: str) -> Any:
return self.call("GET", f"/api/workflows/executions/{execution_id}/steps")
def reviews(self, execution_id: str) -> dict:
return self.call("GET", f"/api/workflows/executions/{execution_id}/reviews")
def decide(self, execution_id: str, checkpoint_id: str, decision: str, note: str) -> dict:
return self.call("POST", f"/api/workflows/executions/{execution_id}/reviews/{checkpoint_id}",
{"decision": decision, "note": note})

78
shakedown_lib/ledger.py Normal file
View File

@@ -0,0 +1,78 @@
"""The run ledger: one folder per voyage run, and one per stage inside it (plan §4).
<root>/<voyage>/<started UTC>/state.json every stage's state, ids, gate decisions; rewritten on each change
<root>/<voyage>/<started UTC>/<stage>/inputs.json the run inputs as sent, uploads shown by name only
<root>/<voyage>/<started UTC>/<stage>/context.yaml the context file as uploaded or sent (with `generated:`)
<root>/<voyage>/<started UTC>/<stage>/status.json, steps.json, evaluation.json
No token or secret is ever written here. state.json is what a resumed run reads.
"""
from __future__ import annotations
import json
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
def utc_stamp(now: float | None = None) -> str:
moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc)
return moment.strftime("%Y%m%dT%H%M%SZ")
def utc_iso(now: float | None = None) -> str:
moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc)
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
def default_root() -> Path:
return Path(os.getenv("SHAKEDOWN_LEDGER") or Path(__file__).resolve().parents[1] / "ledger")
class Ledger:
def __init__(self, folder: Path) -> None:
self.folder = Path(folder)
self.folder.mkdir(parents=True, exist_ok=True)
@classmethod
def new(cls, root: Path, voyage: str, now: float | None = None) -> "Ledger":
return cls(Path(root) / voyage / utc_stamp(now))
def path(self, *parts: str) -> Path:
target = self.folder.joinpath(*parts)
target.parent.mkdir(parents=True, exist_ok=True)
return target
def write(self, name: str, data: Any, stage: str | None = None) -> Path:
target = self.path(*([stage] if stage else []), name)
tmp = target.with_suffix(target.suffix + ".tmp")
if isinstance(data, (bytes, bytearray)):
tmp.write_bytes(data)
elif isinstance(data, str):
tmp.write_text(data)
else:
tmp.write_text(json.dumps(data, indent=1, default=str))
tmp.replace(target)
return target
def load_state(self) -> dict:
path = self.folder / "state.json"
return json.loads(path.read_text()) if path.is_file() else {}
def save_state(self, state: dict) -> None:
self.write("state.json", state)
def shown_inputs(inputs: dict) -> dict:
"""The inputs as recorded: uploads by name, long documents by length (their text is kept beside, in the stage)."""
out = {}
for key, value in inputs.items():
if key == "file" and isinstance(value, list):
out[key] = [item.get("name") for item in value if isinstance(item, dict)]
elif isinstance(value, str) and len(value) > 200:
out[key] = f"<{len(value)} characters>"
else:
out[key] = value
return out

389
shakedown_lib/runner.py Normal file
View File

@@ -0,0 +1,389 @@
"""Run a voyage remotely, stage by stage, with no person in the loop except where the rules keep one.
For each stage, in order:
1. **Start**: check the workflow id still names the registered workflow; build the inputs from the context file and
the voyage (upstream ids resolved); check required inputs and the engagement id; start the run.
A stage with `use_execution` adopts an existing, finished run instead.
2. **Follow**: poll the run. At a review point:
- in a project listed in `config/runner.yaml` `gate_projects`, the gate rule (gate.py) decides: continue only on a
critic `pass` with no failed step, with a note saying the system decided. Otherwise the pause is left for a
person and the voyage stops for a fix;
- in any other project, wait for a person to decide in KAB.
3. **Finish**: pull the outputs, find the package and its artifact id, judge the run against its case
(cp1-agents `evaluate_case.py`), and stop the voyage on a failed run, a missing package or a failed critical check.
The ledger is written on every change, so `--resume` picks a voyage up where it stopped (a started run is followed,
never started twice).
"""
from __future__ import annotations
import base64
import re
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional
import yaml
from . import cp1_agents, gate
from .kab import Kab, KabError
from .ledger import Ledger, shown_inputs, utc_iso
from .voyage import Stage, Voyage, VoyageError, Workflow, load_workflow, resolve_path
DONE_OK = {"completed", "succeeded", "success"}
DONE_BAD = {"failed", "error", "cancelled", "canceled", "aborted", "stopped", "rejected", "timed_out", "timeout"}
PLACEHOLDER = re.compile(r"^<[^>]+>$")
EXIT_OK, EXIT_CONFIG, EXIT_STOPPED, EXIT_PERSON_TIMEOUT, EXIT_TIMEOUT = 0, 2, 3, 4, 5
class Stop(Exception):
def __init__(self, code: int, reason: str) -> None:
super().__init__(reason)
self.code = code
self.reason = reason
@dataclass
class Settings:
gate_projects: set
poll_seconds: int = 30
stage_timeout_s: int = 6 * 3600
person_wait_s: int = 24 * 3600
@classmethod
def from_config(cls, config: dict) -> "Settings":
return cls(gate_projects=set(config.get("gate_projects") or set()),
poll_seconds=int(config.get("poll_seconds", 30)),
stage_timeout_s=int(config.get("stage_timeout_minutes", 360)) * 60,
person_wait_s=int(config.get("person_wait_hours", 24)) * 3600)
Evaluator = Callable[[str, dict, Any], tuple]
def default_evaluator(feature: str, case: dict, steps: Any) -> tuple:
return cp1_agents.evaluate_case().evaluate(feature, case, steps)
@dataclass
class Runner:
kab: Kab
ledger: Ledger
voyage: Voyage
settings: Settings
evaluator: Evaluator = default_evaluator
clock: Callable[[], float] = time.time
sleep: Callable[[float], None] = time.sleep
log: Callable[[str], None] = print
state: dict = field(default_factory=dict)
# ---- state -------------------------------------------------------------------------------------------------
def _load(self) -> None:
self.state = self.ledger.load_state() or {
"voyage": self.voyage.name, "engagement_id": self.voyage.engagement_id, "project": self.voyage.project,
"voyage_file": str(self.voyage.source) if self.voyage.source else None, "started_at": utc_iso(self.clock()),
"cp1_agents_commit": cp1_agents.commit(), "decided_by": "kab-automation (Shakedown runner)",
"stages": {}}
def _save(self) -> None:
self.state["updated_at"] = utc_iso(self.clock())
self.ledger.save_state(self.state)
def _stage_state(self, key: str) -> dict:
return self.state["stages"].setdefault(key, {"state": "pending", "decisions": []})
# ---- the voyage --------------------------------------------------------------------------------------------
def run(self, dry_run: bool = False, only: Optional[List[str]] = None) -> int:
self._load()
if self.state.get("project") != self.voyage.project:
self.log(f"refused: the ledger is for project {self.state.get('project')}, the voyage for {self.voyage.project}")
return EXIT_CONFIG
gates = "decided by the system" if self.voyage.project in self.settings.gate_projects else "left to a person"
self.log(f"Voyage {self.voyage.name} ({self.voyage.engagement_id}) in project {self.voyage.project}; gates {gates}.")
self.log(f"Ledger: {self.ledger.folder}")
try:
for stage in self.voyage.stages:
if only and stage.key not in only and self._stage_state(stage.key).get("state") != "completed":
if any(s.key in only for s in self.voyage.stages[self.voyage.stages.index(stage) + 1:]):
raise Stop(EXIT_CONFIG, f"stage {stage.key} isn't completed, and a later stage was asked for")
continue
self._stage(stage, dry_run)
except Stop as stop:
self.state["stopped"] = {"code": stop.code, "reason": stop.reason, "at": utc_iso(self.clock())}
self._save()
self.log(f"Stopped: {stop.reason}")
return stop.code
except (KabError, VoyageError) as error:
self.state["stopped"] = {"code": EXIT_CONFIG, "reason": str(error), "at": utc_iso(self.clock())}
self._save()
self.log(f"Stopped: {error}")
return EXIT_CONFIG
self.state.pop("stopped", None)
self.state["finished_at"] = utc_iso(self.clock())
self._save()
self.log("Dry run: nothing started." if dry_run else "Voyage complete.")
return EXIT_OK
def _stage(self, stage: Stage, dry_run: bool) -> None:
st = self._stage_state(stage.key)
if st.get("state") == "completed":
self.log(f"[{stage.key}] already completed: {st.get('execution_id')}")
return
wf = load_workflow(stage.workflow)
workflow_id = stage.workflow_id or wf.workflow_id
if stage.use_execution:
st.update(execution_id=stage.use_execution, adopted=True, approval_ref=stage.approval_ref,
workflow=wf.key, state="started")
self._save()
if not st.get("execution_id"):
if not workflow_id:
raise VoyageError(f"stage {stage.key}: no workflow_id in the voyage or workflows/{wf.key}.yaml")
template = self._checked_template(stage, wf, workflow_id)
inputs, context_text = self._inputs(stage, wf, workflow_id)
st.update(workflow=wf.key, workflow_id=workflow_id, workflow_name=template.get("name"),
workflow_version=template.get("version"), run_id=inputs.get("run_id"))
self.ledger.write("inputs.json", shown_inputs(inputs), stage.key)
if context_text is not None:
self.ledger.write("context.yaml", context_text, stage.key)
if dry_run:
st["state"] = "planned"
self._save()
self.log(f"[{stage.key}] would start {template.get('name')} ({workflow_id}) with "
f"{shown_inputs(inputs)}")
return
name = self.kab.project_name(self.voyage.project)
body = cp1_agents.run_workflow().with_run_metadata(template, workflow_id, self.voyage.project, name)
result = self.kab.execute(body, inputs)
if not result.get("execution_id"):
raise Stop(EXIT_STOPPED, f"stage {stage.key}: KAB started no execution: {str(result)[:300]}")
st.update(execution_id=result["execution_id"], state="started", started_at=utc_iso(self.clock()))
self._save()
self.log(f"[{stage.key}] started {result['execution_id']} ({template.get('name')})")
self._follow(stage, st)
self._finish(stage, wf, st)
# ---- start ---------------------------------------------------------------------------------------------------
def _checked_template(self, stage: Stage, wf: Workflow, workflow_id: str) -> dict:
workflow = self.kab.workflow(workflow_id)
template = workflow.get("template") or workflow
if template.get("name") not in wf.names:
raise VoyageError(f"stage {stage.key}: workflow {workflow_id} is {template.get('name')!r}, not one of "
f"{wf.names} (ids change on re-import: update the voyage)")
return template
def _resolve(self, stage: Stage, value: Any) -> Any:
if not isinstance(value, dict):
return value
upstream = self.state["stages"].get(value["from"]) or {}
if upstream.get("state") == "planned": # a dry run: the upstream run doesn't exist yet
deferred = f"(at run time: {value['from']}.{value['value']})"
return [deferred] if value["value"] == "package_artifact_ids" else deferred
if upstream.get("state") != "completed":
raise VoyageError(f"stage {stage.key} needs stage {value['from']} completed first")
found = upstream.get(value["value"])
if found in (None, "", []):
raise VoyageError(f"stage {stage.key}: stage {value['from']} has no {value['value']}")
return found
def _inputs(self, stage: Stage, wf: Workflow, workflow_id: str) -> tuple[dict, Optional[str]]:
helpers = cp1_agents.run_workflow()
given = {name: self._resolve(stage, value) for name, value in stage.inputs.items()}
inputs: Dict[str, Any] = {}
context_doc, context_text = None, None
if stage.context:
path = resolve_path(stage.context)
context_doc = yaml.safe_load(path.read_text()) or {}
from_file = (context_doc.get("workflow_inputs") or {}) if isinstance(context_doc, dict) else {}
for key, value in from_file.items():
value = "yes" if value is True else "no" if value is False else value
if key in given and not (isinstance(value, str) and PLACEHOLDER.match(value)) and str(value) != str(given[key]):
raise VoyageError(f"stage {stage.key}: {key} is {value!r} in {path.name} and {given[key]!r} in "
"the voyage: keep one")
inputs[key] = value if not isinstance(value, (int, float)) else str(value)
inputs.update(given)
if "run_id" in inputs:
raise VoyageError(f"stage {stage.key}: run_id is generated for every run; remove it")
if inputs.get("engagement_id") not in (None, self.voyage.engagement_id):
raise VoyageError(f"stage {stage.key}: engagement_id {inputs.get('engagement_id')} is not the voyage's "
f"{self.voyage.engagement_id}")
inputs["engagement_id"] = self.voyage.engagement_id
generated = helpers.new_ids()
inputs["run_id"] = generated["run_id"]
for name, doc_path in stage.documents.items():
inputs[name] = resolve_path(doc_path).read_text()
if context_doc is not None:
has_upstream = any(inputs.get(n) for n in helpers.UPSTREAM_INPUTS)
problem = helpers.customer_problem(context_doc, has_upstream=has_upstream)
if problem:
raise VoyageError(f"stage {stage.key}: {stage.context}: {problem}")
header = ("# Added by the Shakedown runner when the run started. Never edit these by hand.\n"
"generated:\n" + "".join(f" {k}: {v}\n" for k, v in generated.items()) + "\n")
path = resolve_path(stage.context)
if wf.context == "upload":
context_text = header + path.read_text()
files = [{"name": path.name, "content": self._data_url(path, context_text.encode())}]
for extra in helpers.context_uploads(path):
files.append({"name": extra.name, "content": self._data_url(extra, extra.read_bytes())})
inputs["file"] = files
elif wf.context == "document":
resolved = dict(context_doc)
resolved["workflow_inputs"] = {k: inputs[k] for k in (context_doc.get("workflow_inputs") or {})
if k in inputs}
context_text = header + yaml.safe_dump(resolved, sort_keys=False, allow_unicode=True)
inputs[wf.document_input] = context_text
leftover = sorted(k for k, v in inputs.items() if isinstance(v, str) and PLACEHOLDER.match(v.strip()))
if leftover:
raise VoyageError(f"stage {stage.key}: still a placeholder: {', '.join(leftover)}")
sample = self.kab.api_sample(workflow_id)
missing = sorted(f"{i['step_id']}.{i['name']}" for i in sample.get("required", [])
if i["name"] != "file" and i["name"] not in inputs)
if missing:
raise VoyageError(f"stage {stage.key}: missing inputs {', '.join(missing)}")
checkable = {k: v for k, v in inputs.items() if not (isinstance(v, str) and v.startswith("(at run time: "))}
try:
helpers.check_engagement(self.kab, checkable)
except SystemExit as refused:
raise VoyageError(f"stage {stage.key}: {refused}") from None
return inputs, context_text
@staticmethod
def _data_url(path: Path, data: bytes) -> str:
kind = cp1_agents.run_workflow().upload_type(path)
return f"data:{kind};base64,{base64.b64encode(data).decode()}"
# ---- follow --------------------------------------------------------------------------------------------------
def _follow(self, stage: Stage, st: dict) -> None:
eid = st["execution_id"]
deadline = self.clock() + self.settings.stage_timeout_s
while True:
status = self.kab.status(eid)
state = str(status.get("status", "")).lower()
if state in DONE_OK or state in DONE_BAD:
if st.get("state") == "needs_person": # a person decided and the run finished between two polls
self._person_decided(stage, st)
st["run_status"] = state
self._save()
return
pending = (self.kab.reviews(eid) or {}).get("pending") or []
if pending:
self._at_gate(stage, st, status, pending)
elif st.get("state") == "needs_person":
self._person_decided(stage, st)
if self.clock() > deadline:
raise Stop(EXIT_TIMEOUT, f"stage {stage.key}: {eid} still {state} after "
f"{self.settings.stage_timeout_s // 60} minutes")
self.sleep(self.settings.poll_seconds)
def _at_gate(self, stage: Stage, st: dict, status: dict, pending: list) -> None:
eid = st["execution_id"]
review = pending[0]
checkpoint = str(review.get("checkpoint_id"))
if any(d.get("checkpoint_id") == checkpoint and d.get("decision") == "continue" for d in st["decisions"]):
return # decided; KAB hasn't moved on yet
system_may_decide = self.voyage.project in self.settings.gate_projects and len(pending) == 1
if not system_may_decide:
if st.get("state") != "needs_person":
st.update(state="needs_person", waiting_since=self.clock(), checkpoint_id=checkpoint)
self._save()
why = "more than one pause" if len(pending) > 1 else "this project isn't one of the runner's gate_projects"
self.log(f"[{stage.key}] {eid} waits at a review point for a person in KAB ({why}).")
elif self.clock() - st.get("waiting_since", self.clock()) > self.settings.person_wait_s:
raise Stop(EXIT_PERSON_TIMEOUT, f"stage {stage.key}: nobody decided {eid}'s review point in "
f"{self.settings.person_wait_s // 3600} hours")
return
steps = self.kab.steps(eid)
verdict = gate.decide(status, steps, review.get("step_id"))
at = utc_iso(self.clock())
record = {"checkpoint_id": checkpoint, "step_id": review.get("step_id"), "at": at, "by": "system",
"verdict": verdict.verdict, "verdict_step": verdict.verdict_step, "reason": verdict.reason}
if not verdict.proceed:
record["decision"] = "left for a fix"
st["decisions"].append(record)
st["state"] = "needs_fix"
self.ledger.write("steps-at-gate.json", steps, stage.key)
self._save()
raise Stop(EXIT_STOPPED, f"stage {stage.key}: not continued at the review point: {verdict.reason}. "
f"{eid} stays paused for a person or a fix.")
note = (f"Decided by the system (Shakedown runner, KAB automation identity), not a person: continue. "
f"{verdict.reason}. Execution {eid}, run {st.get('run_id') or '?'}, voyage {self.voyage.name}, {at}.")
self.kab.decide(eid, checkpoint, "continue", note)
record.update(decision="continue", note=note)
st["decisions"].append(record)
st.update(state="started", approval_ref=f"Shakedown runner (system) continued {eid} at {at}: "
f"critic verdict pass at {verdict.verdict_step}")
self._save()
self.log(f"[{stage.key}] continued {eid}: {verdict.reason}")
def _person_decided(self, stage: Stage, st: dict) -> None:
"""The pause is gone and the run moves on: a person decided in KAB. Record who, as best KAB says."""
eid = st["execution_id"]
reviews = self.kab.reviews(eid) or {}
decided = [r for key in ("history", "decided", "completed", "reviews") for r in (reviews.get(key) or [])
if isinstance(r, dict) and str(r.get("checkpoint_id")) == str(st.get("checkpoint_id"))]
last = decided[-1] if decided else {}
who = last.get("decided_by") or last.get("reviewer") or last.get("user") or "a person in KAB"
when = last.get("decided_at") or last.get("updated_at") or utc_iso(self.clock())
decision = last.get("decision") or "continue"
st["decisions"].append({"checkpoint_id": st.get("checkpoint_id"), "by": "person", "decision": decision,
"who": who, "at": when})
st.update(state="started", approval_ref=f"{who} decided {decision} on {eid} at {when}")
st.pop("waiting_since", None)
self._save()
self.log(f"[{stage.key}] {who} decided {decision} on {eid}")
# ---- finish --------------------------------------------------------------------------------------------------
def _finish(self, stage: Stage, wf: Workflow, st: dict) -> None:
eid = st["execution_id"]
status = self.kab.status(eid)
steps = self.kab.steps(eid)
self.ledger.write("status.json", status, stage.key)
self.ledger.write("steps.json", steps, stage.key)
run_status = str(status.get("status", "")).lower()
if run_status not in DONE_OK:
st["state"] = "failed"
self._save()
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} ended {run_status or 'with no status'}")
artifact = self._package_artifact(eid, status, steps, st.get("workflow_id"))
if artifact is None:
st["state"] = "failed"
self._save()
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} completed with no package (no output has a manifest "
"with a package_version)")
st["package_artifact_ids"] = [artifact]
if not st.get("approval_ref"):
st["approval_ref"] = f"{eid} completed with no review point decided by the runner"
if stage.case:
case = yaml.safe_load(resolve_path(stage.case).read_text())
rows, failed = self.evaluator(wf.feature, case, steps)
evaluation = {"case": stage.case, "failed": failed,
"checks": [dict(zip(("status", "severity", "where", "id", "detail"), r)) for r in rows]}
self.ledger.write("evaluation.json", evaluation, stage.key)
st["evaluation"] = {"failed": failed, "checks": len(rows)}
if failed.get("critical"):
st["state"] = "checks_failed"
self._save()
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {failed['critical']} critical check(s) failed on {eid} "
f"(see {stage.key}/evaluation.json)")
st.update(state="completed", completed_at=utc_iso(self.clock()))
self._save()
self.log(f"[{stage.key}] completed {eid}; package {artifact}")
def _package_artifact(self, eid: str, status: dict, steps: Any, workflow_id: Optional[str]) -> Optional[str]:
"""`<execution id>_step_<n>`, n the package step's 1-based place in the executed template."""
outputs = gate.step_outputs(steps)
holder = next((k for k, v in outputs.items() if isinstance(v, dict) and isinstance(v.get("manifest"), dict)
and "package_version" in v["manifest"]), None)
if holder is None:
return None
order = gate.template_order(status)
if holder not in order and workflow_id:
workflow = self.kab.workflow(workflow_id)
order = [str(s.get("id")) for s in ((workflow.get("template") or workflow).get("steps") or [])]
if holder not in order:
return None
return f"{eid}_step_{order.index(holder) + 1}"

116
shakedown_lib/voyage.py Normal file
View File

@@ -0,0 +1,116 @@
"""Voyages (one engagement's chain of use cases) and the workflow registry, as loaded from this repository.
A path is relative to this repository, or to the cp1-agents clone when it starts with `cp1-agents:` (the committed
F01 to F03 cases stay there and are read in place, plan §3).
A stage input is a literal, or a reference to an earlier stage's result: `{from: <stage key>, value: <name>}`, where
the name is `execution_id`, `approval_ref` or `package_artifact_ids` (`["<execution id>_step_<n>"]` of its package
step). References are resolved when the stage starts; the files themselves are never edited.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional
import yaml
from . import cp1_agents
REPO = Path(__file__).resolve().parents[1]
CONTEXT_ADAPTERS = {"upload", "document", "none"}
STAGE_VALUES = {"execution_id", "approval_ref", "package_artifact_ids"}
class VoyageError(ValueError):
pass
def resolve_path(text: str) -> Path:
if text.startswith("cp1-agents:"):
return cp1_agents.clone_dir() / text.split(":", 1)[1]
path = Path(text)
return path if path.is_absolute() else REPO / path
@dataclass
class Workflow:
key: str
feature: str
names: List[str]
context: str
document_input: Optional[str] = None
workflow_id: Optional[str] = None
def load_workflow(key: str) -> Workflow:
path = REPO / "workflows" / f"{key}.yaml"
if not path.is_file():
raise VoyageError(f"workflow {key!r} is not registered: no {path.relative_to(REPO)}")
doc = yaml.safe_load(path.read_text()) or {}
wf = Workflow(key=doc.get("key", key), feature=doc["feature"], names=list(doc["names"]), context=doc["context"],
document_input=doc.get("document_input"), workflow_id=doc.get("workflow_id"))
if wf.context not in CONTEXT_ADAPTERS:
raise VoyageError(f"{path.name}: context must be one of {sorted(CONTEXT_ADAPTERS)}")
if wf.context == "document" and not wf.document_input:
raise VoyageError(f"{path.name}: a document context names its run input in document_input")
return wf
@dataclass
class Stage:
key: str
workflow: str
workflow_id: Optional[str] = None
context: Optional[str] = None
case: Optional[str] = None
inputs: Dict[str, Any] = field(default_factory=dict)
documents: Dict[str, str] = field(default_factory=dict)
use_execution: Optional[str] = None
approval_ref: Optional[str] = None
@dataclass
class Voyage:
name: str
engagement_id: str
project: str
stages: List[Stage]
capability: Optional[str] = None
source: Optional[Path] = None
def load_voyage(path: Path) -> Voyage:
doc = yaml.safe_load(Path(path).read_text()) or {}
for required in ("voyage", "engagement_id", "project", "stages"):
if not doc.get(required):
raise VoyageError(f"{path}: `{required}` is required")
stages, seen = [], set()
for raw in doc["stages"]:
stage = Stage(key=raw["key"], workflow=raw["workflow"], workflow_id=raw.get("workflow_id"),
context=raw.get("context"), case=raw.get("case"), inputs=dict(raw.get("inputs") or {}),
documents=dict(raw.get("documents") or {}), use_execution=raw.get("use_execution"),
approval_ref=raw.get("approval_ref"))
if stage.key in seen:
raise VoyageError(f"{path}: stage key {stage.key!r} appears twice")
if stage.use_execution and not stage.approval_ref:
raise VoyageError(f"{path}: stage {stage.key} uses execution {stage.use_execution}: give its approval_ref "
"(who continued it at its review point, and when)")
for name, value in stage.inputs.items():
if isinstance(value, dict):
if value.get("from") not in seen or value.get("value") not in STAGE_VALUES:
raise VoyageError(f"{path}: stage {stage.key} input {name}: `from` must name an earlier stage and "
f"`value` one of {sorted(STAGE_VALUES)}")
seen.add(stage.key)
stages.append(stage)
return Voyage(name=doc["voyage"], engagement_id=doc["engagement_id"], project=doc["project"], stages=stages,
capability=doc.get("capability"), source=Path(path))
def load_config(path: Path | None = None) -> dict:
"""Runner settings, from a committed file so a change to them is reviewed. The environment can't widen them."""
path = path or REPO / "config" / "runner.yaml"
doc = yaml.safe_load(path.read_text()) or {}
doc["gate_projects"] = {str(p) for p in (doc.get("gate_projects") or [])}
return doc

0
tests/__init__.py Normal file
View File

320
tests/test_runner.py Normal file
View File

@@ -0,0 +1,320 @@
"""The runner against a fake KAB: python3 -m unittest discover -s tests (needs the cp1-agents clone beside this repo,
or CP1_AGENTS_DIR)."""
from __future__ import annotations
import json
import tempfile
import textwrap
import unittest
from pathlib import Path
from urllib.parse import urlparse
import yaml
from shakedown_lib import gate
from shakedown_lib.auth import AuthError, ClientCredentials
from shakedown_lib.kab import Kab
from shakedown_lib.ledger import Ledger
from shakedown_lib.runner import EXIT_CONFIG, EXIT_OK, EXIT_STOPPED, Runner, Settings
from shakedown_lib.voyage import load_voyage
PROJECT = "proj-1"
ENG = "ENG-TEST-001"
NAMES = {"wf-f01": "CP1 - Agentic Landing Zone - Discovery", "wf-f02": "CP1 - Agentic Landing Zone - Assessment",
"wf-f03": "CP1 - Agentic Landing Zone - Strategy Synthesis"}
def template(name: str, upload: bool) -> dict:
first = "u1" if upload else "a1"
return {"name": name, "version": "1.0", "steps": [{"id": first}] + [{"id": f"s{i}"} for i in range(1, 7)] + [{"id": "f1"}]}
class FakeKab:
"""Runs move: running -> paused at s5 (one pending review) -> after `continue`, completed with a package at s6."""
def __init__(self, verdict: str = "pass", polls_before_pause: int = 1, person_after: int | None = None,
names: dict | None = None) -> None:
self.verdict = verdict
self.polls_before_pause = polls_before_pause
self.person_after = person_after
self.names = names or NAMES
self.runs: dict = {"exec_f01_done": {"workflow": "wf-f01", "inputs": {"engagement_id": ENG}, "phase": "done",
"polls": 0, "history": []}}
self.executed: list = []
self.decisions: list = []
def transport(self, method, url, headers, data):
assert headers["Authorization"].startswith("Bearer "), "no token sent"
path = urlparse(url).path
body = json.loads(data) if data else None
reply = self.route(method, path, body)
return 200, json.dumps(reply).encode()
def _status(self, eid):
run = self.runs[eid]
state = {"running": "running", "paused": "attention", "done": "completed"}[run["phase"]]
return {"execution_id": eid, "status": state, "inputs": run["inputs"],
"template": template(self.names[run["workflow"]], run["workflow"] == "wf-f01")}
def _steps(self, eid):
run = self.runs[eid]
steps = {"s4": {"response": json.dumps({"phase": "critique", "status": "completed",
"verdict": {"verdict": self.verdict}})}}
if run["phase"] == "done":
steps["s6"] = {"output": {"engagement_id": ENG, "manifest": {"package_version": "1.0"}}}
return {"execution_id": eid, "steps": steps}
def route(self, method, path, body):
if path.startswith("/api/v1/workflow-catalog/workflows/"):
wid = path.rsplit("/", 1)[1]
return {"workflow": {"template": template(self.names[wid], wid == "wf-f01")}}
if path.endswith("/api-sample"):
return {"required": []}
if path.startswith("/api/v1/projects/"):
return {"project": {"name": "Test project"}}
if path == "/api/workflows/execute":
wid = body["template"]["metadata"]["workflow_id"]
eid = f"exec_{len(self.executed) + 1}"
self.executed.append({"workflow": wid, "inputs": body["inputs"], "template": body["template"]})
self.runs[eid] = {"workflow": wid, "inputs": body["inputs"], "phase": "running", "polls": 0, "history": []}
return {"execution_id": eid, "status": "running"}
if path.startswith("/api/workflows/status/"):
eid = path.rsplit("/", 1)[1]
run = self.runs[eid]
run["polls"] += 1
if run["phase"] == "running" and run["polls"] > self.polls_before_pause:
run["phase"] = "paused"
run["paused_at"] = run["polls"]
if run["phase"] == "paused" and self.person_after is not None and run["polls"] - run["paused_at"] >= self.person_after:
run["phase"] = "done"
run["history"].append({"checkpoint_id": f"cp-{eid}", "decision": "continue",
"decided_by": "lead@example.com", "decided_at": "2026-11-16T10:00:00Z"})
return self._status(eid)
if path.endswith("/steps"):
return self._steps(path.split("/")[-2])
if path.endswith("/reviews") and method == "GET":
eid = path.split("/")[-2]
run = self.runs[eid]
pending = [{"checkpoint_id": f"cp-{eid}", "step_id": "s5"}] if run["phase"] == "paused" else []
return {"pending": pending, "history": run["history"]}
if "/reviews/" in path and method == "POST":
eid = path.split("/")[-3]
self.decisions.append({"execution": eid, **body})
if body["decision"] == "continue":
self.runs[eid]["phase"] = "done"
return {"ok": True}
raise AssertionError(f"unexpected {method} {path}")
class Tokens:
def __init__(self):
self.calls = 0
def token(self, force=False):
self.calls += 1
return "t"
class Clock:
def __init__(self):
self.now = 1_800_000_000.0
def __call__(self):
return self.now
def sleep(self, seconds):
self.now += seconds
def write_voyage(folder: Path, f01_adopted: bool = True) -> Path:
(folder / "context.yaml").write_text(textwrap.dedent(f"""\
workflow_inputs:
engagement_id: {ENG}
provider: sample
include_cmdb: yes
customer: {{id: CUS-TEST, name: Test}}
"""))
(folder / "run-context.yaml").write_text(textwrap.dedent(f"""\
workflow_inputs:
engagement_id: {ENG}
assessment_execution_id: <assessment execution id>
assessment_approval_ref: "<who continued it>"
context:
customer: {{id: CUS-TEST}}
"""))
(folder / "rate-card.json").write_text('{"rates": []}')
(folder / "case.yaml").write_text("id: test\nchecks: []\n")
f01 = {"key": "f01", "workflow": "alz-f01-discovery", "workflow_id": "wf-f01", "context": f"{folder}/context.yaml"}
if f01_adopted:
f01.update(use_execution="exec_f01_done", approval_ref="approved by a person")
doc = {"voyage": ENG, "engagement_id": ENG, "project": PROJECT, "stages": [
f01,
{"key": "f02", "workflow": "alz-f02-assessment", "workflow_id": "wf-f02",
"inputs": {"discovery_execution_id": {"from": "f01", "value": "execution_id"},
"discovery_approval_ref": {"from": "f01", "value": "approval_ref"},
"artifact_ids": {"from": "f01", "value": "package_artifact_ids"}},
"documents": {"rate_card_document": f"{folder}/rate-card.json"}, "case": f"{folder}/case.yaml"},
{"key": "f03", "workflow": "alz-f03-strategy", "workflow_id": "wf-f03", "context": f"{folder}/run-context.yaml",
"inputs": {"assessment_execution_id": {"from": "f02", "value": "execution_id"},
"assessment_approval_ref": {"from": "f02", "value": "approval_ref"},
"artifact_ids": {"from": "f02", "value": "package_artifact_ids"}}}]}
voyage = folder / "voyage.yaml"
voyage.write_text(yaml.safe_dump(doc, sort_keys=False))
return voyage
class RunnerTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.dir = Path(self.tmp.name)
self.clock = Clock()
self.evaluated = []
def tearDown(self):
self.tmp.cleanup()
def runner(self, fake: FakeKab, gate_projects=(PROJECT,), ledger: Ledger | None = None, adopted=True, **settings):
voyage = load_voyage(write_voyage(self.dir, adopted))
kab = Kab(Tokens(), api="https://kab.test", tenant="t", project=PROJECT, transport=fake.transport,
sleep=lambda s: None)
def evaluator(feature, case, steps):
self.evaluated.append(feature)
return [("pass", "critical", "manifest", "c1", "")], {"critical": 0, "major": 0}
return Runner(kab=kab, ledger=ledger or Ledger(self.dir / "ledger"), voyage=voyage,
settings=Settings(gate_projects=set(gate_projects), poll_seconds=30, **settings),
evaluator=evaluator, clock=self.clock, sleep=self.clock.sleep, log=lambda m: None)
def test_whole_voyage_decides_gates_in_its_own_project(self):
fake = FakeKab()
runner = self.runner(fake)
self.assertEqual(runner.run(), EXIT_OK)
self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"])
f02, f03 = fake.executed
self.assertEqual(f02["inputs"]["discovery_execution_id"], "exec_f01_done")
self.assertEqual(f02["inputs"]["artifact_ids"], ["exec_f01_done_step_7"])
self.assertEqual(f02["inputs"]["rate_card_document"], '{"rates": []}')
self.assertEqual(f02["template"]["metadata"]["project_id"], PROJECT)
self.assertEqual(f03["inputs"]["artifact_ids"], ["exec_1_step_7"])
self.assertIn("Shakedown runner (system) continued exec_1", f03["inputs"]["assessment_approval_ref"])
document = f03["inputs"]["run_context_document"]
self.assertIn("assessment_execution_id: exec_1", document)
self.assertNotIn("<assessment execution id>", document)
self.assertEqual([d["decision"] for d in fake.decisions], ["continue", "continue"])
self.assertTrue(all("not a person" in d["note"] for d in fake.decisions))
state = json.loads((self.dir / "ledger" / "state.json").read_text())
self.assertEqual({k: v["state"] for k, v in state["stages"].items()},
{"f01": "completed", "f02": "completed", "f03": "completed"})
self.assertEqual(self.evaluated, ["f02"])
self.assertTrue((self.dir / "ledger" / "f03" / "context.yaml").is_file())
def test_critic_fail_is_never_continued(self):
fake = FakeKab(verdict="fail")
self.assertEqual(self.runner(fake).run(), EXIT_STOPPED)
self.assertEqual(fake.decisions, [])
state = json.loads((self.dir / "ledger" / "state.json").read_text())
self.assertEqual(state["stages"]["f02"]["state"], "needs_fix")
self.assertIn("critic verdict is 'fail'", state["stopped"]["reason"])
def test_other_projects_wait_for_a_person(self):
fake = FakeKab(person_after=3)
self.assertEqual(self.runner(fake, gate_projects=()).run(), EXIT_OK)
self.assertEqual(fake.decisions, [])
f03 = fake.executed[1]
self.assertEqual(f03["inputs"]["assessment_approval_ref"],
"lead@example.com decided continue on exec_1 at 2026-11-16T10:00:00Z")
def test_nobody_deciding_times_out(self):
fake = FakeKab()
code = self.runner(fake, gate_projects=(), person_wait_s=120).run()
self.assertEqual(code, 4)
self.assertEqual(fake.decisions, [])
def test_a_stale_workflow_id_is_refused(self):
fake = FakeKab(names={**NAMES, "wf-f02": "CP1 - Agentic Landing Zone - Assessment (individual agents)"})
self.assertEqual(self.runner(fake).run(), EXIT_CONFIG)
self.assertEqual(fake.executed, [])
def test_resume_never_starts_a_run_twice(self):
fake = FakeKab(verdict="fail")
ledger = Ledger(self.dir / "ledger")
self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_STOPPED)
fake.verdict = "pass" # say a fix was deployed and the stage rerun in KAB
self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_OK)
self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"])
def test_dry_run_starts_nothing(self):
fake = FakeKab()
self.assertEqual(self.runner(fake, adopted=False).run(dry_run=True), EXIT_OK)
self.assertEqual(fake.executed, [])
shown = json.loads((self.dir / "ledger" / "f02" / "inputs.json").read_text())
self.assertEqual(shown["discovery_execution_id"], "(at run time: f01.execution_id)")
f01 = json.loads((self.dir / "ledger" / "f01" / "inputs.json").read_text())
self.assertEqual(f01["file"], ["context.yaml"])
self.assertEqual(f01["include_cmdb"], "yes")
class GateTest(unittest.TestCase):
def status(self):
return {"status": "attention", "template": {"steps": [{"id": i} for i in ("a1", "s1", "s2", "s3", "s4", "s5", "s6")]}}
def test_pass(self):
steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": "pass"}}}}
self.assertTrue(gate.decide(self.status(), steps, "s5").proceed)
def test_a_later_fail_overrides_an_earlier_pass(self):
steps = {"steps": {"s3": {"phase": "x", "verdict": "pass"}, "s4": {"phase": "critique", "verdict": {"verdict": "fail"}}}}
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
def test_no_verdict(self):
self.assertFalse(gate.decide(self.status(), {"steps": {"s1": {"phase": "plan"}}}, "s5").proceed)
def test_a_failed_step(self):
steps = {"steps": {"s2": {"phase": "collect", "status": "failed"}, "s4": {"phase": "c", "verdict": "pass"}}}
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
def test_unparseable_verdict(self):
steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": None}}}}
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
class AuthTest(unittest.TestCase):
def test_renews_before_expiry_and_hides_the_secret(self):
clock = Clock()
issued = []
def post(url, form):
issued.append(form["grant_type"])
return {"access_token": f"tok{len(issued)}", "expires_in": 300}
tokens = ClientCredentials("kab-automation", "s3cret", post=post, clock=clock)
self.assertEqual(tokens.token(), "tok1")
clock.now += 200
self.assertEqual(tokens.token(), "tok1")
clock.now += 50 # inside the last minute
self.assertEqual(tokens.token(), "tok2")
self.assertEqual(issued, ["client_credentials", "client_credentials"])
self.assertNotIn("s3cret", repr(tokens))
def test_no_secret(self):
with self.assertRaises(AuthError):
ClientCredentials("kab-automation", "")
def test_a_401_renews_the_token_once(self):
calls = []
class T:
def token(self, force=False):
calls.append(force)
return "x"
replies = iter([(401, b""), (200, b'{"ok": 1}')])
kab = Kab(T(), transport=lambda *a: next(replies), sleep=lambda s: None)
self.assertEqual(kab.call("GET", "/x"), {"ok": 1})
self.assertEqual(calls, [False, True])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,8 @@
# ALZ-F01 Discovery (ADO Feature 3214441). Registry entry: plan §3c.
key: alz-f01-discovery
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
feature: f01 # the feature name cp1-agents scripts/evaluate_case.py takes
# Workflow names KAB may show for it. A run's workflow id is checked against these before it starts, because ids
# change on every re-import (same names as the docs portal's registry/workflows.yaml).
names: ["CP1 - Agentic Landing Zone - Discovery"]
context: upload # the run context is uploaded to u1 (file); its workflow_inputs become run inputs

View File

@@ -0,0 +1,8 @@
# ALZ-F02 Assessment & Disposition (ADO Feature 3214442). Registry entry: plan §3c.
key: alz-f02-assessment
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
feature: f02
# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change
# on every re-import).
names: ["CP1 - Agentic Landing Zone - Assessment"]
context: none # no context file: inputs come from the voyage (an alignment finding, plan §3b)

View File

@@ -0,0 +1,9 @@
# ALZ-F03 Strategy Synthesis (ADO Feature 3214443). Registry entry: plan §3c.
key: alz-f03-strategy
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
feature: f03
# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change
# on every re-import).
names: ["CP1 - Agentic Landing Zone - Strategy Synthesis"]
context: document # the run context is sent as text in one run input (F03 ADR 0002)
document_input: run_context_document