- shakedown_lib: runs F01 -> F02 -> F03 for one engagement on KAB with no personal token. Signs in as the kab-automation client (client credentials, provisioned by Stellwerk from crucible-platform-deploy) and renews the token itself. - Each stage checks its workflow id against the registered sub-agent name, builds inputs from the case and upstream stages, starts the run, follows it and judges it with cp1-agents evaluate_case.py. - Gates (plan 5b): in config/runner.yaml gate_projects only, continue on a critic pass with no failed step and a note saying the system decided. Never on a fail. Elsewhere, wait for a person and record who decided. gate_projects is empty for now. - Ledger per voyage run, with --resume; nothing secret is written. - Cymbal voyage ENG-CYMBAL-001; workflow registry; Dockerfile and an unapplied Job template; 15 tests against a fake KAB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
73 lines
3.7 KiB
Markdown
73 lines
3.7 KiB
Markdown
# Shakedown
|
|
|
|
Runs, rates and experiments on KAB workflows. The plan is [docs/implementation-plan.md](docs/implementation-plan.md).
|
|
What is built so far is the **runner**, which runs a voyage remotely with no person's token: F01 Discovery → F02
|
|
Assessment → F03 Strategy for one engagement.
|
|
|
|
## The runner
|
|
|
|
```sh
|
|
export KAB_CLIENT_SECRET=... # the kab-automation client's secret; never in a file
|
|
python3 -m shakedown_lib token-check
|
|
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run
|
|
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml
|
|
python3 -m shakedown_lib run <voyage> --resume ledger/ENG-CYMBAL-001/<started> # pick up where it stopped
|
|
python3 -m unittest discover -s tests -t . # tests, against a fake KAB
|
|
```
|
|
|
|
**Identity.** The runner signs in as the confidential client `kab-automation` of realm `kab`, using client
|
|
credentials. Its tokens carry the automation user's email, which KAB makes an Editor. The Stellwerk orchestrator
|
|
provisions the client and its secret: crucible-platform-deploy, `kab-cluster/dev` component 85. The secret is held in
|
|
Key Vault `crucible-kab-dev-kv` as `kab-automation-client-secret`. A token is renewed a minute before it expires, so a
|
|
run that lasts hours needs nobody.
|
|
|
|
**What one stage does:**
|
|
1. **Check the workflow.** The stage's workflow id must still name its registered workflow (`workflows/*.yaml`, sub-agent pattern). Ids change on every re-import.
|
|
2. **Build the inputs.** They come from the case's context file and the voyage. Upstream execution ids, approval
|
|
references and package artifact ids (`<execution id>_step_<n>`) come from earlier stages. Each context
|
|
adapter delivers the file differently:
|
|
- `upload` (F01): uploaded as the run's file;
|
|
- `document` (F03): sent as text in one run input;
|
|
- `none` (F02): inputs come from the voyage.
|
|
3. **Check and start.** Before starting, it checks the required inputs, the engagement id (one per chain) and the
|
|
customer block, using cp1-agents' own helpers.
|
|
4. **Follow the run.** At a review point:
|
|
- in a project listed in `config/runner.yaml` `gate_projects`, the runner continues **only** when the critic's
|
|
verdict is `pass` and no step has failed. Its note says the system decided, not a person. Otherwise the pause is
|
|
left as it is and the voyage stops for a fix;
|
|
- in any other project, it waits for a person to decide in KAB and records who did.
|
|
5. **Finish.** It pulls the outputs, finds the package and judges the run against its case with cp1-agents
|
|
`scripts/evaluate_case.py`. A failed run, a missing package or a failed critical check stops the voyage.
|
|
|
|
**Ledger.** Each voyage run gets its own folder, `ledger/<voyage>/<started>/` (or `$SHAKEDOWN_LEDGER`). It holds:
|
|
- `state.json`;
|
|
- for each stage: its inputs (uploads by name only), the context as sent, the status, the steps and the evaluation.
|
|
|
|
No token or secret is written there.
|
|
|
|
**Exit status:**
|
|
|
|
| Code | Meaning |
|
|
|---|---|
|
|
| 0 | Done |
|
|
| 2 | Configuration error, or KAB refused |
|
|
| 3 | Stopped for a fix |
|
|
| 4 | Nobody decided a review point in time |
|
|
| 5 | A stage ran too long |
|
|
|
|
**cp1-agents** is read from a clone beside this repository, or from `$CP1_AGENTS_DIR`, and its commit is recorded
|
|
with every run. Nothing in cp1-agents is changed.
|
|
|
|
**On the cluster:** `deploy/Dockerfile` and `deploy/runner-job.yaml`, a Job template with placeholders. Nothing is
|
|
applied without approval.
|
|
|
|
## Not built yet
|
|
|
|
These come from the plan:
|
|
- Bosun (the deep agent);
|
|
- fixes on `agentic/*` branches;
|
|
- deploying from a branch;
|
|
- experiments and variations;
|
|
- the portal views;
|
|
- the nightly cycle.
|