Runner: run a voyage remotely as the kab-automation identity
- shakedown_lib: runs F01 -> F02 -> F03 for one engagement on KAB with no personal token. Signs in as the kab-automation client (client credentials, provisioned by Stellwerk from crucible-platform-deploy) and renews the token itself. - Each stage checks its workflow id against the registered sub-agent name, builds inputs from the case and upstream stages, starts the run, follows it and judges it with cp1-agents evaluate_case.py. - Gates (plan 5b): in config/runner.yaml gate_projects only, continue on a critic pass with no failed step and a note saying the system decided. Never on a fail. Elsewhere, wait for a person and record who decided. gate_projects is empty for now. - Ledger per voyage run, with --resume; nothing secret is written. - Cymbal voyage ENG-CYMBAL-001; workflow registry; Dockerfile and an unapplied Job template; 15 tests against a fake KAB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
ledger/
|
||||
.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
build/
|
||||
72
README.md
Normal file
72
README.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Shakedown
|
||||
|
||||
Runs, rates and experiments on KAB workflows. The plan is [docs/implementation-plan.md](docs/implementation-plan.md).
|
||||
What is built so far is the **runner**, which runs a voyage remotely with no person's token: F01 Discovery → F02
|
||||
Assessment → F03 Strategy for one engagement.
|
||||
|
||||
## The runner
|
||||
|
||||
```sh
|
||||
export KAB_CLIENT_SECRET=... # the kab-automation client's secret; never in a file
|
||||
python3 -m shakedown_lib token-check
|
||||
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run
|
||||
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml
|
||||
python3 -m shakedown_lib run <voyage> --resume ledger/ENG-CYMBAL-001/<started> # pick up where it stopped
|
||||
python3 -m unittest discover -s tests -t . # tests, against a fake KAB
|
||||
```
|
||||
|
||||
**Identity.** The runner signs in as the confidential client `kab-automation` of realm `kab`, using client
|
||||
credentials. Its tokens carry the automation user's email, which KAB makes an Editor. The Stellwerk orchestrator
|
||||
provisions the client and its secret: crucible-platform-deploy, `kab-cluster/dev` component 85. The secret is held in
|
||||
Key Vault `crucible-kab-dev-kv` as `kab-automation-client-secret`. A token is renewed a minute before it expires, so a
|
||||
run that lasts hours needs nobody.
|
||||
|
||||
**What one stage does:**
|
||||
1. **Check the workflow.** The stage's workflow id must still name its registered workflow (`workflows/*.yaml`, sub-agent pattern). Ids change on every re-import.
|
||||
2. **Build the inputs.** They come from the case's context file and the voyage. Upstream execution ids, approval
|
||||
references and package artifact ids (`<execution id>_step_<n>`) come from earlier stages. Each context
|
||||
adapter delivers the file differently:
|
||||
- `upload` (F01): uploaded as the run's file;
|
||||
- `document` (F03): sent as text in one run input;
|
||||
- `none` (F02): inputs come from the voyage.
|
||||
3. **Check and start.** Before starting, it checks the required inputs, the engagement id (one per chain) and the
|
||||
customer block, using cp1-agents' own helpers.
|
||||
4. **Follow the run.** At a review point:
|
||||
- in a project listed in `config/runner.yaml` `gate_projects`, the runner continues **only** when the critic's
|
||||
verdict is `pass` and no step has failed. Its note says the system decided, not a person. Otherwise the pause is
|
||||
left as it is and the voyage stops for a fix;
|
||||
- in any other project, it waits for a person to decide in KAB and records who did.
|
||||
5. **Finish.** It pulls the outputs, finds the package and judges the run against its case with cp1-agents
|
||||
`scripts/evaluate_case.py`. A failed run, a missing package or a failed critical check stops the voyage.
|
||||
|
||||
**Ledger.** Each voyage run gets its own folder, `ledger/<voyage>/<started>/` (or `$SHAKEDOWN_LEDGER`). It holds:
|
||||
- `state.json`;
|
||||
- for each stage: its inputs (uploads by name only), the context as sent, the status, the steps and the evaluation.
|
||||
|
||||
No token or secret is written there.
|
||||
|
||||
**Exit status:**
|
||||
|
||||
| Code | Meaning |
|
||||
|---|---|
|
||||
| 0 | Done |
|
||||
| 2 | Configuration error, or KAB refused |
|
||||
| 3 | Stopped for a fix |
|
||||
| 4 | Nobody decided a review point in time |
|
||||
| 5 | A stage ran too long |
|
||||
|
||||
**cp1-agents** is read from a clone beside this repository, or from `$CP1_AGENTS_DIR`, and its commit is recorded
|
||||
with every run. Nothing in cp1-agents is changed.
|
||||
|
||||
**On the cluster:** `deploy/Dockerfile` and `deploy/runner-job.yaml`, a Job template with placeholders. Nothing is
|
||||
applied without approval.
|
||||
|
||||
## Not built yet
|
||||
|
||||
These come from the plan:
|
||||
- Bosun (the deep agent);
|
||||
- fixes on `agentic/*` branches;
|
||||
- deploying from a branch;
|
||||
- experiments and variations;
|
||||
- the portal views;
|
||||
- the nightly cycle.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Cymbal Bank (fictional; estate served by alz-discovery-sample): Discovery -> Assessment -> Strategy.
|
||||
# The cases stay in cp1-agents and are read in place. Upstream ids are filled from the earlier stages' runs.
|
||||
# The workflow ids are unverified: each is checked against its registry name (sub-agent pattern) before a stage
|
||||
# starts, so a stale or individual-pattern id is refused. `--dry-run` checks them all without starting anything.
|
||||
voyage: ENG-CYMBAL-001
|
||||
capability: cp1-agentic-landing-zone
|
||||
engagement_id: ENG-CYMBAL-001
|
||||
project: 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026 # KAB project "Cymbal Bank" (kab-dev)
|
||||
|
||||
stages:
|
||||
- key: f01
|
||||
workflow: alz-f01-discovery
|
||||
workflow_id: 6aea4d516bc9323ca8d4bd12
|
||||
# Run exec_e60e23f0a4ac passed its case and was approved on 2026-11-15, so it is adopted, not run again.
|
||||
# Remove these two lines to run Discovery afresh from the context below.
|
||||
use_execution: exec_e60e23f0a4ac
|
||||
approval_ref: "Continued by a person in KAB on 2026-11-15 (Cymbal Discovery run approved)"
|
||||
context: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/context.yaml"
|
||||
case: "cp1-agents:individual-agents/ALZ-F01: Discovery Workflow Agent/workflows/cases/cymbal-bank/case.yaml"
|
||||
|
||||
- key: f02
|
||||
workflow: alz-f02-assessment
|
||||
workflow_id: 6aecac756bc9323ca8d4be1c # from the case README; checked against the registry names
|
||||
inputs:
|
||||
discovery_execution_id: {from: f01, value: execution_id}
|
||||
discovery_approval_ref: {from: f01, value: approval_ref}
|
||||
artifact_ids: {from: f01, value: package_artifact_ids}
|
||||
documents:
|
||||
rate_card_document: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/rate-card.json"
|
||||
case: "cp1-agents:individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/cymbal-bank-assessment/case.yaml"
|
||||
|
||||
- key: f03
|
||||
workflow: alz-f03-strategy
|
||||
workflow_id: wf_0f80f01c-e423-4553-9ec0-afa21b3bbb52 # from the docs portal registry; checked by name
|
||||
context: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/run-context.yaml"
|
||||
inputs:
|
||||
assessment_execution_id: {from: f02, value: execution_id}
|
||||
assessment_approval_ref: {from: f02, value: approval_ref}
|
||||
artifact_ids: {from: f02, value: package_artifact_ids}
|
||||
case: "cp1-agents:individual-agents/ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/cymbal-bank-strategy/case.yaml"
|
||||
13
config/runner.yaml
Normal file
13
config/runner.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
# Shakedown runner settings. Committed, so every change is reviewed; the environment can't widen them.
|
||||
|
||||
# The KAB projects whose review points the runner may decide itself (plan §5b): only Shakedown's own use-case
|
||||
# projects. In any other project it waits for a person. It continues a gate only on a critic `pass` with no failed
|
||||
# step (shakedown_lib/gate.py), never on a `fail`, and its note says the system decided.
|
||||
#
|
||||
# Empty until the owner names the projects. For the Cymbal voyage that would be the "Cymbal Bank" project:
|
||||
# - 5a2da83b-cf1e-48d8-9b07-0a84f5e5f026
|
||||
gate_projects: []
|
||||
|
||||
poll_seconds: 30
|
||||
stage_timeout_minutes: 360 # one stage, start to package
|
||||
person_wait_hours: 24 # the workflows' own review timeout (hitl.timeout_hours)
|
||||
16
deploy/Dockerfile
Normal file
16
deploy/Dockerfile
Normal file
@@ -0,0 +1,16 @@
|
||||
# The Shakedown runner. Build from the repository root:
|
||||
# buildah bud -f deploy/Dockerfile -t cruciblecoredevacr.azurecr.io/kab/cp1-shakedown/runner:<version> .
|
||||
ARG BASE=python:3.12-slim
|
||||
FROM ${BASE}
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends git ca-certificates && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY shakedown_lib/ shakedown_lib/
|
||||
COPY config/ config/
|
||||
COPY workflows/ workflows/
|
||||
COPY capabilities/ capabilities/
|
||||
RUN useradd -u 10001 -m shakedown && mkdir -p /ledger && chown shakedown /ledger
|
||||
USER 10001
|
||||
ENV CP1_AGENTS_DIR=/cp1-agents SHAKEDOWN_LEDGER=/ledger PYTHONUNBUFFERED=1
|
||||
ENTRYPOINT ["python3", "-m", "shakedown_lib"]
|
||||
52
deploy/runner-job.yaml
Normal file
52
deploy/runner-job.yaml
Normal file
@@ -0,0 +1,52 @@
|
||||
# One voyage run as a Kubernetes Job in namespace `shakedown`. A template: placeholders are filled at render time,
|
||||
# and nothing here is applied without the owner's approval.
|
||||
#
|
||||
# Provisioned outside this repository by the Stellwerk orchestrator (crucible-platform-deploy, kab-cluster/dev):
|
||||
# - the `shakedown` namespace and the PVC `shakedown-ledger`;
|
||||
# - Secret `shakedown-kab-automation`, key `client-secret`: the kab-automation client's secret, from Key Vault
|
||||
# crucible-kab-dev-kv `kab-automation-client-secret`. The runner reads it from the environment only;
|
||||
# - Secret `shakedown-gitea-read`, key `token`: read access to agents/cp1-agents, for the pinned clone.
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
generateName: shakedown-voyage-
|
||||
namespace: shakedown
|
||||
labels: {app.kubernetes.io/name: shakedown-runner, app.kubernetes.io/part-of: cp1-shakedown}
|
||||
spec:
|
||||
backoffLimit: 0 # a failed run is resumed on purpose (--resume), never retried blind
|
||||
activeDeadlineSeconds: 172800 # two days: three stages plus waiting for people at review points
|
||||
template:
|
||||
metadata:
|
||||
labels: {app.kubernetes.io/name: shakedown-runner}
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
securityContext: {runAsNonRoot: true, runAsUser: 10001, fsGroup: 10001}
|
||||
initContainers:
|
||||
- name: clone-cp1-agents
|
||||
image: <placeholder_for_runner_image>
|
||||
command: ["sh", "-c"]
|
||||
args:
|
||||
- >-
|
||||
git -c http.extraHeader="Authorization: token $GITEA_TOKEN" clone --quiet
|
||||
https://gitea.kyndemo.live/agents/cp1-agents.git /cp1-agents &&
|
||||
git -C /cp1-agents checkout --quiet <placeholder_for_cp1_agents_commit>
|
||||
env:
|
||||
- name: GITEA_TOKEN
|
||||
valueFrom: {secretKeyRef: {name: shakedown-gitea-read, key: token}}
|
||||
volumeMounts: [{name: cp1-agents, mountPath: /cp1-agents}]
|
||||
containers:
|
||||
- name: runner
|
||||
image: <placeholder_for_runner_image>
|
||||
args: ["run", "<placeholder_for_voyage_file>"]
|
||||
env:
|
||||
- name: KAB_CLIENT_SECRET
|
||||
valueFrom: {secretKeyRef: {name: shakedown-kab-automation, key: client-secret}}
|
||||
resources:
|
||||
requests: {cpu: 50m, memory: 128Mi}
|
||||
limits: {cpu: 500m, memory: 512Mi}
|
||||
volumeMounts:
|
||||
- {name: cp1-agents, mountPath: /cp1-agents, readOnly: true}
|
||||
- {name: ledger, mountPath: /ledger}
|
||||
volumes:
|
||||
- {name: cp1-agents, emptyDir: {}}
|
||||
- {name: ledger, persistentVolumeClaim: {claimName: shakedown-ledger}}
|
||||
@@ -8,7 +8,7 @@ enters service. This app does the same for every workflow before its outputs rea
|
||||
|---|---|
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9; amended 2026-11-12: one UI in the portal, the nightly cycle, skills, §5a, §9; amended 2026-11-15: end-to-end use cases that fix what fails, §5b) |
|
||||
| User interface | the KAB Docs Portal (`engineering_hub_responsiblities/kab-docs-portal`). Shakedown is that UI's visualisation layer for workflow case outputs (portal ADR 0005); Shakedown itself serves an API |
|
||||
| Status | Draft. Nothing is built yet |
|
||||
| Status | Draft. Built so far (2026-11-16): the voyage **runner** (README), which runs F01 → F02 → F03 remotely as the `kab-automation` identity provisioned by Stellwerk, with the §5b gate rule in code |
|
||||
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
|
||||
| Runs on | The `crucible-kab-dev-aks` cluster |
|
||||
| Deep agent | LangChain **Deep Agents** (`deepagents`, on LangGraph), driven by **Claude Sonnet 5.5** on the Kyndryl Azure resource |
|
||||
|
||||
2
requirements.txt
Normal file
2
requirements.txt
Normal file
@@ -0,0 +1,2 @@
|
||||
pyyaml>=6
|
||||
jsonschema>=4 # cp1-agents scripts/check_data_contract.py, used by evaluate_case.py
|
||||
1
shakedown_lib/__init__.py
Normal file
1
shakedown_lib/__init__.py
Normal file
@@ -0,0 +1 @@
|
||||
"""Shakedown: runs, rates and experiments on KAB workflows (docs/implementation-plan.md). This package holds the runner."""
|
||||
76
shakedown_lib/__main__.py
Normal file
76
shakedown_lib/__main__.py
Normal file
@@ -0,0 +1,76 @@
|
||||
"""Shakedown runner: run a voyage's workflows on KAB remotely, as the kab-automation identity.
|
||||
|
||||
python3 -m shakedown_lib token-check
|
||||
python3 -m shakedown_lib run capabilities/cp1-agentic-landing-zone/voyages/ENG-CYMBAL-001.yaml --dry-run
|
||||
python3 -m shakedown_lib run <voyage.yaml> [--only f02 f03]
|
||||
python3 -m shakedown_lib run <voyage.yaml> --resume ledger/ENG-CYMBAL-001/<started>
|
||||
|
||||
Environment: KAB_CLIENT_SECRET (required), KAB_CLIENT_ID (kab-automation), KAB_TOKEN_URL, KAB_API, KAB_TENANT,
|
||||
CP1_AGENTS_DIR (the cp1-agents clone, default ../cp1-agents), SHAKEDOWN_LEDGER (default ./ledger).
|
||||
Which projects the runner may decide gates in is set only in config/runner.yaml.
|
||||
|
||||
Exit status: 0 done, 2 configuration or KAB refused, 3 stopped for a fix, 4 nobody decided a review point in time,
|
||||
5 a stage ran too long.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from .auth import AuthError, ClientCredentials
|
||||
from .kab import CP1_TENANT, DEFAULT_API, Kab
|
||||
from .ledger import Ledger, default_root
|
||||
from .runner import EXIT_CONFIG, Runner, Settings
|
||||
from .voyage import VoyageError, load_config, load_voyage
|
||||
|
||||
|
||||
def _claims(token: str) -> dict:
|
||||
try:
|
||||
payload = token.split(".")[1]
|
||||
return json.loads(base64.urlsafe_b64decode(payload + "=" * (-len(payload) % 4)))
|
||||
except (IndexError, ValueError):
|
||||
return {}
|
||||
|
||||
|
||||
def token_check() -> int:
|
||||
tokens = ClientCredentials.from_env()
|
||||
claims = _claims(tokens.token())
|
||||
shown = {k: claims.get(k) for k in ("iss", "azp", "email", "preferred_username", "exp")}
|
||||
print(f"Token issued to {tokens.client_id}, valid for {tokens.expires_in()} s. Claims: {json.dumps(shown)}")
|
||||
return 0
|
||||
|
||||
|
||||
def run(args) -> int:
|
||||
voyage = load_voyage(Path(args.voyage))
|
||||
settings = Settings.from_config(load_config())
|
||||
kab = Kab(ClientCredentials.from_env(), api=os.getenv("KAB_API", DEFAULT_API),
|
||||
tenant=os.getenv("KAB_TENANT", CP1_TENANT), project=voyage.project)
|
||||
ledger = Ledger(Path(args.resume)) if args.resume else Ledger.new(default_root(), voyage.name)
|
||||
return Runner(kab=kab, ledger=ledger, voyage=voyage, settings=settings).run(dry_run=args.dry_run, only=args.only)
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
parser = argparse.ArgumentParser(prog="shakedown_lib", description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
commands.add_parser("token-check", help="get a token as kab-automation and show who it names")
|
||||
start = commands.add_parser("run", help="run a voyage")
|
||||
start.add_argument("voyage")
|
||||
start.add_argument("--dry-run", action="store_true", help="build and check every stage's inputs; start nothing")
|
||||
start.add_argument("--resume", help="a ledger folder of an earlier run of this voyage")
|
||||
start.add_argument("--only", nargs="+", help="run only these stages (earlier ones must be completed)")
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
return token_check() if args.command == "token-check" else run(args)
|
||||
except (AuthError, VoyageError, FileNotFoundError, RuntimeError) as error:
|
||||
print(f"Refused: {error}", file=sys.stderr)
|
||||
return EXIT_CONFIG
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
80
shakedown_lib/auth.py
Normal file
80
shakedown_lib/auth.py
Normal file
@@ -0,0 +1,80 @@
|
||||
"""KAB access tokens for unattended runs: the `kab-automation` client of realm `kab`, by client credentials.
|
||||
|
||||
The client and its secret are provisioned by the Stellwerk orchestrator (crucible-platform-deploy, `kab-cluster/dev`
|
||||
component 85; module `kab-identity`). The secret is in Key Vault `crucible-kab-dev-kv` as
|
||||
`kab-automation-client-secret` and reaches the runner as the environment variable KAB_CLIENT_SECRET (a mounted
|
||||
Secret on the cluster). Tokens carry the fixed email claim KAB knows the automation user by.
|
||||
|
||||
The secret and every token stay in memory: never written to a file, the ledger or a log, and never printed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Callable
|
||||
|
||||
DEFAULT_TOKEN_URL = "https://id.kab-dev.kyndemo.live/realms/kab/protocol/openid-connect/token"
|
||||
DEFAULT_CLIENT_ID = "kab-automation"
|
||||
REFRESH_MARGIN_S = 60
|
||||
|
||||
|
||||
class AuthError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _post_form(url: str, form: dict, timeout: int = 30) -> dict:
|
||||
body = urllib.parse.urlencode(form).encode()
|
||||
request = urllib.request.Request(url, data=body, headers={"Content-Type": "application/x-www-form-urlencoded"})
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
return json.load(response)
|
||||
except urllib.error.HTTPError as error:
|
||||
try:
|
||||
reason = json.loads(error.read() or b"{}").get("error_description") or "no reason given"
|
||||
except ValueError:
|
||||
reason = "no reason given"
|
||||
raise AuthError(f"token request refused: HTTP {error.code}: {reason}") from None
|
||||
except urllib.error.URLError as error:
|
||||
raise AuthError(f"token endpoint unreachable: {error.reason}") from None
|
||||
|
||||
|
||||
class ClientCredentials:
|
||||
"""A token source that renews itself a minute before the token expires."""
|
||||
|
||||
def __init__(self, client_id: str, secret: str, token_url: str = DEFAULT_TOKEN_URL,
|
||||
post: Callable[[str, dict], dict] = _post_form, clock: Callable[[], float] = time.time) -> None:
|
||||
if not secret:
|
||||
raise AuthError("no client secret: set KAB_CLIENT_SECRET (the kab-automation client's secret)")
|
||||
self.client_id = client_id
|
||||
self._secret = secret
|
||||
self.token_url = token_url
|
||||
self._post = post
|
||||
self._clock = clock
|
||||
self._token: str | None = None
|
||||
self._expires_at = 0.0
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "ClientCredentials":
|
||||
return cls(os.getenv("KAB_CLIENT_ID", DEFAULT_CLIENT_ID), os.getenv("KAB_CLIENT_SECRET", ""),
|
||||
os.getenv("KAB_TOKEN_URL", DEFAULT_TOKEN_URL))
|
||||
|
||||
def __repr__(self) -> str: # never show the secret or the token
|
||||
return f"ClientCredentials(client_id={self.client_id!r}, token_url={self.token_url!r})"
|
||||
|
||||
def token(self, force: bool = False) -> str:
|
||||
if force or self._token is None or self._clock() >= self._expires_at - REFRESH_MARGIN_S:
|
||||
reply = self._post(self.token_url, {"grant_type": "client_credentials", "client_id": self.client_id,
|
||||
"client_secret": self._secret})
|
||||
if not reply.get("access_token"):
|
||||
raise AuthError("token reply has no access_token")
|
||||
self._token = reply["access_token"]
|
||||
self._expires_at = self._clock() + float(reply.get("expires_in") or 300)
|
||||
return self._token
|
||||
|
||||
def expires_in(self) -> int:
|
||||
return max(0, int(self._expires_at - self._clock()))
|
||||
59
shakedown_lib/cp1_agents.py
Normal file
59
shakedown_lib/cp1_agents.py
Normal file
@@ -0,0 +1,59 @@
|
||||
"""The cp1-agents scripts Shakedown reuses, imported by path from a clone (plan §3: no change to cp1-agents).
|
||||
|
||||
`kab_run_workflow.py` gives the run-context helpers (load_context, new_ids, context_uploads, upload_type,
|
||||
customer_problem, with_run_metadata, check_engagement) and `evaluate_case.py` judges a run against a case. The clone's
|
||||
commit is recorded with every run, so a result can be reproduced.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import ModuleType
|
||||
|
||||
DEFAULT_DIR = Path(__file__).resolve().parents[2] / "cp1-agents"
|
||||
|
||||
|
||||
def clone_dir() -> Path:
|
||||
path = Path(os.getenv("CP1_AGENTS_DIR") or DEFAULT_DIR)
|
||||
if not (path / "scripts" / "kab_run_workflow.py").is_file():
|
||||
raise RuntimeError(f"no cp1-agents clone at {path}: set CP1_AGENTS_DIR")
|
||||
return path
|
||||
|
||||
|
||||
def _load(name: str) -> ModuleType:
|
||||
scripts = clone_dir() / "scripts"
|
||||
if str(scripts) not in sys.path:
|
||||
sys.path.insert(0, str(scripts)) # evaluate_case imports check_data_contract beside it
|
||||
spec = importlib.util.spec_from_file_location(f"cp1_agents_{name}", scripts / f"{name}.py")
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
_cache: dict[str, ModuleType] = {}
|
||||
|
||||
|
||||
def module(name: str) -> ModuleType:
|
||||
if name not in _cache:
|
||||
_cache[name] = _load(name)
|
||||
return _cache[name]
|
||||
|
||||
|
||||
def run_workflow() -> ModuleType:
|
||||
return module("kab_run_workflow")
|
||||
|
||||
|
||||
def evaluate_case() -> ModuleType:
|
||||
return module("evaluate_case")
|
||||
|
||||
|
||||
def commit() -> str | None:
|
||||
try:
|
||||
return subprocess.run(["git", "-C", str(clone_dir()), "rev-parse", "HEAD"], capture_output=True, text=True,
|
||||
check=True).stdout.strip()
|
||||
except (subprocess.CalledProcessError, FileNotFoundError, RuntimeError):
|
||||
return None
|
||||
97
shakedown_lib/gate.py
Normal file
97
shakedown_lib/gate.py
Normal file
@@ -0,0 +1,97 @@
|
||||
"""The gate rule for end-to-end use cases (plan §5b), decided in code from what the run recorded.
|
||||
|
||||
At a review pause the system may continue only when, for the steps up to the pause:
|
||||
- no step reported `failed` or `error`, and
|
||||
- the stage's latest critic verdict is exactly `pass`.
|
||||
|
||||
Anything else (a `fail`, an unparseable or missing verdict, a failed step) is never continued: the pause is left for a
|
||||
person and the use case stops for a fix. Whether the system may decide at all depends on the project (the runner's
|
||||
`gate_projects`), not on this module.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
FAILED = {"failed", "error"}
|
||||
|
||||
|
||||
def parsed(value: Any) -> Any:
|
||||
if isinstance(value, str) and value[:1] in "{[":
|
||||
try:
|
||||
return json.loads(value)
|
||||
except ValueError:
|
||||
return value
|
||||
return value
|
||||
|
||||
|
||||
def unwrap(value: Any) -> Any:
|
||||
"""A step's own output, out of KAB's wrappers (`response`, `output`, `result`, `data`), JSON strings decoded."""
|
||||
value = parsed(value)
|
||||
for _ in range(4):
|
||||
if isinstance(value, dict) and "phase" not in value and "manifest" not in value:
|
||||
inner = next((value[k] for k in ("response", "output", "result", "data") if k in value), None)
|
||||
if inner is None:
|
||||
break
|
||||
value = parsed(inner)
|
||||
else:
|
||||
break
|
||||
return value
|
||||
|
||||
|
||||
def step_outputs(steps: Any) -> Dict[str, Any]:
|
||||
"""step id -> its output, from `/executions/{id}/steps` (a mapping by step id, or a list of step records)."""
|
||||
body = steps.get("steps", steps) if isinstance(steps, dict) else steps
|
||||
if isinstance(body, dict):
|
||||
return {str(k): unwrap(v) for k, v in body.items() if k != "execution_id"}
|
||||
out: Dict[str, Any] = {}
|
||||
for item in body or []:
|
||||
if isinstance(item, dict):
|
||||
key = item.get("step_id") or item.get("id")
|
||||
if key is not None:
|
||||
out[str(key)] = unwrap(item.get("output", item.get("result", item)))
|
||||
return out
|
||||
|
||||
|
||||
def template_order(status: dict) -> List[str]:
|
||||
return [str(s.get("id")) for s in ((status.get("template") or {}).get("steps") or [])]
|
||||
|
||||
|
||||
def _verdict(value: Any) -> Optional[str]:
|
||||
if isinstance(value, dict):
|
||||
value = value.get("verdict")
|
||||
return value if isinstance(value, str) else None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Decision:
|
||||
proceed: bool
|
||||
reason: str
|
||||
verdict: Optional[str]
|
||||
verdict_step: Optional[str]
|
||||
|
||||
|
||||
def decide(status: dict, steps: Any, paused_step: Optional[str]) -> Decision:
|
||||
if str(status.get("status", "")).lower() in FAILED:
|
||||
return Decision(False, f"run status is {status.get('status')}", None, None)
|
||||
outputs = step_outputs(steps)
|
||||
order = template_order(status) or list(outputs)
|
||||
if paused_step in order:
|
||||
order = order[: order.index(paused_step) + 1]
|
||||
verdict, verdict_step = None, None
|
||||
for step_id in order:
|
||||
out = outputs.get(step_id)
|
||||
if not isinstance(out, dict):
|
||||
continue
|
||||
if str(out.get("status", "")).lower() in FAILED:
|
||||
return Decision(False, f"step {step_id} reported {out.get('status')}", verdict, verdict_step)
|
||||
found = _verdict(out.get("verdict"))
|
||||
if found is not None:
|
||||
verdict, verdict_step = found, step_id
|
||||
if verdict is None:
|
||||
return Decision(False, "no critic verdict before the review point", None, None)
|
||||
if verdict != "pass":
|
||||
return Decision(False, f"critic verdict is {verdict!r} (step {verdict_step})", verdict, verdict_step)
|
||||
return Decision(True, f"critic verdict pass (step {verdict_step}), no failed step", verdict, verdict_step)
|
||||
104
shakedown_lib/kab.py
Normal file
104
shakedown_lib/kab.py
Normal file
@@ -0,0 +1,104 @@
|
||||
"""The KAB API, as the automation identity: the same endpoints `cp1-agents/scripts/kab_run_workflow.py` uses.
|
||||
|
||||
Every call takes a fresh token from the token source, so a run that lasts hours never needs a person. A 401 renews
|
||||
the token once; 429 and 5xx are retried with back-off. Errors raise KabError; nothing calls sys.exit.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from typing import Any, Callable, Protocol
|
||||
|
||||
DEFAULT_API = "https://api.kab-dev.kyndemo.live"
|
||||
CP1_TENANT = "6ae37d13999438461a529409"
|
||||
RETRIES = 4
|
||||
|
||||
|
||||
class TokenSource(Protocol):
|
||||
def token(self, force: bool = False) -> str: ...
|
||||
|
||||
|
||||
class KabError(RuntimeError):
|
||||
def __init__(self, message: str, status: int | None = None) -> None:
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
|
||||
|
||||
Transport = Callable[[str, str, dict, bytes | None], tuple[int, bytes]]
|
||||
|
||||
|
||||
def _urllib_transport(method: str, url: str, headers: dict, data: bytes | None) -> tuple[int, bytes]:
|
||||
request = urllib.request.Request(url, data=data, headers=headers, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=120) as response:
|
||||
return response.status, response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, error.read() or b""
|
||||
except urllib.error.URLError as error:
|
||||
return 599, str(error.reason).encode()
|
||||
|
||||
|
||||
class Kab:
|
||||
def __init__(self, tokens: TokenSource, api: str = DEFAULT_API, tenant: str = CP1_TENANT,
|
||||
project: str | None = None, transport: Transport = _urllib_transport,
|
||||
sleep: Callable[[float], None] = time.sleep) -> None:
|
||||
self.tokens = tokens
|
||||
self.api = api.rstrip("/")
|
||||
self.tenant = tenant
|
||||
self.project = project
|
||||
self._transport = transport
|
||||
self._sleep = sleep
|
||||
|
||||
def call(self, method: str, path: str, body: dict | None = None) -> Any:
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
force = False
|
||||
for attempt in range(RETRIES + 1):
|
||||
headers = {"Authorization": f"Bearer {self.tokens.token(force=force)}", "X-Tenant-ID": self.tenant,
|
||||
"Content-Type": "application/json"}
|
||||
if self.project:
|
||||
headers["X-Project-ID"] = self.project
|
||||
status, raw = self._transport(method, f"{self.api}{path}", headers, data)
|
||||
if 200 <= status < 300:
|
||||
return json.loads(raw) if raw else {}
|
||||
if status == 401 and not force:
|
||||
force = True
|
||||
continue
|
||||
if (status == 429 or status >= 500) and attempt < RETRIES:
|
||||
self._sleep(min(60, 5 * 2 ** attempt))
|
||||
continue
|
||||
raise KabError(f"{method} {path}: HTTP {status}: {raw[:400].decode(errors='replace')}", status)
|
||||
raise KabError(f"{method} {path}: gave up after {RETRIES} retries")
|
||||
|
||||
# The calls the runner makes.
|
||||
def workflow(self, workflow_id: str) -> dict:
|
||||
found = self.call("GET", f"/api/v1/workflow-catalog/workflows/{workflow_id}")
|
||||
return found.get("workflow", found)
|
||||
|
||||
def api_sample(self, workflow_id: str) -> dict:
|
||||
return self.call("GET", f"/api/workflows/{workflow_id}/api-sample")
|
||||
|
||||
def project_name(self, project_id: str) -> str | None:
|
||||
try:
|
||||
found = self.call("GET", f"/api/v1/projects/{project_id}")
|
||||
except KabError:
|
||||
return None
|
||||
return (found.get("project") or found).get("name")
|
||||
|
||||
def execute(self, template: dict, inputs: dict) -> dict:
|
||||
return self.call("POST", "/api/workflows/execute", {"template": template, "inputs": inputs})
|
||||
|
||||
def status(self, execution_id: str) -> dict:
|
||||
return self.call("GET", f"/api/workflows/status/{execution_id}")
|
||||
|
||||
def steps(self, execution_id: str) -> Any:
|
||||
return self.call("GET", f"/api/workflows/executions/{execution_id}/steps")
|
||||
|
||||
def reviews(self, execution_id: str) -> dict:
|
||||
return self.call("GET", f"/api/workflows/executions/{execution_id}/reviews")
|
||||
|
||||
def decide(self, execution_id: str, checkpoint_id: str, decision: str, note: str) -> dict:
|
||||
return self.call("POST", f"/api/workflows/executions/{execution_id}/reviews/{checkpoint_id}",
|
||||
{"decision": decision, "note": note})
|
||||
78
shakedown_lib/ledger.py
Normal file
78
shakedown_lib/ledger.py
Normal file
@@ -0,0 +1,78 @@
|
||||
"""The run ledger: one folder per voyage run, and one per stage inside it (plan §4).
|
||||
|
||||
<root>/<voyage>/<started UTC>/state.json every stage's state, ids, gate decisions; rewritten on each change
|
||||
<root>/<voyage>/<started UTC>/<stage>/inputs.json the run inputs as sent, uploads shown by name only
|
||||
<root>/<voyage>/<started UTC>/<stage>/context.yaml the context file as uploaded or sent (with `generated:`)
|
||||
<root>/<voyage>/<started UTC>/<stage>/status.json, steps.json, evaluation.json
|
||||
|
||||
No token or secret is ever written here. state.json is what a resumed run reads.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def utc_stamp(now: float | None = None) -> str:
|
||||
moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc)
|
||||
return moment.strftime("%Y%m%dT%H%M%SZ")
|
||||
|
||||
|
||||
def utc_iso(now: float | None = None) -> str:
|
||||
moment = datetime.fromtimestamp(now, timezone.utc) if now is not None else datetime.now(timezone.utc)
|
||||
return moment.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
|
||||
def default_root() -> Path:
|
||||
return Path(os.getenv("SHAKEDOWN_LEDGER") or Path(__file__).resolve().parents[1] / "ledger")
|
||||
|
||||
|
||||
class Ledger:
|
||||
def __init__(self, folder: Path) -> None:
|
||||
self.folder = Path(folder)
|
||||
self.folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
@classmethod
|
||||
def new(cls, root: Path, voyage: str, now: float | None = None) -> "Ledger":
|
||||
return cls(Path(root) / voyage / utc_stamp(now))
|
||||
|
||||
def path(self, *parts: str) -> Path:
|
||||
target = self.folder.joinpath(*parts)
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
return target
|
||||
|
||||
def write(self, name: str, data: Any, stage: str | None = None) -> Path:
|
||||
target = self.path(*([stage] if stage else []), name)
|
||||
tmp = target.with_suffix(target.suffix + ".tmp")
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
tmp.write_bytes(data)
|
||||
elif isinstance(data, str):
|
||||
tmp.write_text(data)
|
||||
else:
|
||||
tmp.write_text(json.dumps(data, indent=1, default=str))
|
||||
tmp.replace(target)
|
||||
return target
|
||||
|
||||
def load_state(self) -> dict:
|
||||
path = self.folder / "state.json"
|
||||
return json.loads(path.read_text()) if path.is_file() else {}
|
||||
|
||||
def save_state(self, state: dict) -> None:
|
||||
self.write("state.json", state)
|
||||
|
||||
|
||||
def shown_inputs(inputs: dict) -> dict:
|
||||
"""The inputs as recorded: uploads by name, long documents by length (their text is kept beside, in the stage)."""
|
||||
out = {}
|
||||
for key, value in inputs.items():
|
||||
if key == "file" and isinstance(value, list):
|
||||
out[key] = [item.get("name") for item in value if isinstance(item, dict)]
|
||||
elif isinstance(value, str) and len(value) > 200:
|
||||
out[key] = f"<{len(value)} characters>"
|
||||
else:
|
||||
out[key] = value
|
||||
return out
|
||||
389
shakedown_lib/runner.py
Normal file
389
shakedown_lib/runner.py
Normal file
@@ -0,0 +1,389 @@
|
||||
"""Run a voyage remotely, stage by stage, with no person in the loop except where the rules keep one.
|
||||
|
||||
For each stage, in order:
|
||||
1. **Start**: check the workflow id still names the registered workflow; build the inputs from the context file and
|
||||
the voyage (upstream ids resolved); check required inputs and the engagement id; start the run.
|
||||
A stage with `use_execution` adopts an existing, finished run instead.
|
||||
2. **Follow**: poll the run. At a review point:
|
||||
- in a project listed in `config/runner.yaml` `gate_projects`, the gate rule (gate.py) decides: continue only on a
|
||||
critic `pass` with no failed step, with a note saying the system decided. Otherwise the pause is left for a
|
||||
person and the voyage stops for a fix;
|
||||
- in any other project, wait for a person to decide in KAB.
|
||||
3. **Finish**: pull the outputs, find the package and its artifact id, judge the run against its case
|
||||
(cp1-agents `evaluate_case.py`), and stop the voyage on a failed run, a missing package or a failed critical check.
|
||||
|
||||
The ledger is written on every change, so `--resume` picks a voyage up where it stopped (a started run is followed,
|
||||
never started twice).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import re
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
from . import cp1_agents, gate
|
||||
from .kab import Kab, KabError
|
||||
from .ledger import Ledger, shown_inputs, utc_iso
|
||||
from .voyage import Stage, Voyage, VoyageError, Workflow, load_workflow, resolve_path
|
||||
|
||||
DONE_OK = {"completed", "succeeded", "success"}
|
||||
DONE_BAD = {"failed", "error", "cancelled", "canceled", "aborted", "stopped", "rejected", "timed_out", "timeout"}
|
||||
PLACEHOLDER = re.compile(r"^<[^>]+>$")
|
||||
|
||||
EXIT_OK, EXIT_CONFIG, EXIT_STOPPED, EXIT_PERSON_TIMEOUT, EXIT_TIMEOUT = 0, 2, 3, 4, 5
|
||||
|
||||
|
||||
class Stop(Exception):
|
||||
def __init__(self, code: int, reason: str) -> None:
|
||||
super().__init__(reason)
|
||||
self.code = code
|
||||
self.reason = reason
|
||||
|
||||
|
||||
@dataclass
|
||||
class Settings:
|
||||
gate_projects: set
|
||||
poll_seconds: int = 30
|
||||
stage_timeout_s: int = 6 * 3600
|
||||
person_wait_s: int = 24 * 3600
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config: dict) -> "Settings":
|
||||
return cls(gate_projects=set(config.get("gate_projects") or set()),
|
||||
poll_seconds=int(config.get("poll_seconds", 30)),
|
||||
stage_timeout_s=int(config.get("stage_timeout_minutes", 360)) * 60,
|
||||
person_wait_s=int(config.get("person_wait_hours", 24)) * 3600)
|
||||
|
||||
|
||||
Evaluator = Callable[[str, dict, Any], tuple]
|
||||
|
||||
|
||||
def default_evaluator(feature: str, case: dict, steps: Any) -> tuple:
|
||||
return cp1_agents.evaluate_case().evaluate(feature, case, steps)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Runner:
|
||||
kab: Kab
|
||||
ledger: Ledger
|
||||
voyage: Voyage
|
||||
settings: Settings
|
||||
evaluator: Evaluator = default_evaluator
|
||||
clock: Callable[[], float] = time.time
|
||||
sleep: Callable[[float], None] = time.sleep
|
||||
log: Callable[[str], None] = print
|
||||
state: dict = field(default_factory=dict)
|
||||
|
||||
# ---- state -------------------------------------------------------------------------------------------------
|
||||
def _load(self) -> None:
|
||||
self.state = self.ledger.load_state() or {
|
||||
"voyage": self.voyage.name, "engagement_id": self.voyage.engagement_id, "project": self.voyage.project,
|
||||
"voyage_file": str(self.voyage.source) if self.voyage.source else None, "started_at": utc_iso(self.clock()),
|
||||
"cp1_agents_commit": cp1_agents.commit(), "decided_by": "kab-automation (Shakedown runner)",
|
||||
"stages": {}}
|
||||
|
||||
def _save(self) -> None:
|
||||
self.state["updated_at"] = utc_iso(self.clock())
|
||||
self.ledger.save_state(self.state)
|
||||
|
||||
def _stage_state(self, key: str) -> dict:
|
||||
return self.state["stages"].setdefault(key, {"state": "pending", "decisions": []})
|
||||
|
||||
# ---- the voyage --------------------------------------------------------------------------------------------
|
||||
def run(self, dry_run: bool = False, only: Optional[List[str]] = None) -> int:
|
||||
self._load()
|
||||
if self.state.get("project") != self.voyage.project:
|
||||
self.log(f"refused: the ledger is for project {self.state.get('project')}, the voyage for {self.voyage.project}")
|
||||
return EXIT_CONFIG
|
||||
gates = "decided by the system" if self.voyage.project in self.settings.gate_projects else "left to a person"
|
||||
self.log(f"Voyage {self.voyage.name} ({self.voyage.engagement_id}) in project {self.voyage.project}; gates {gates}.")
|
||||
self.log(f"Ledger: {self.ledger.folder}")
|
||||
try:
|
||||
for stage in self.voyage.stages:
|
||||
if only and stage.key not in only and self._stage_state(stage.key).get("state") != "completed":
|
||||
if any(s.key in only for s in self.voyage.stages[self.voyage.stages.index(stage) + 1:]):
|
||||
raise Stop(EXIT_CONFIG, f"stage {stage.key} isn't completed, and a later stage was asked for")
|
||||
continue
|
||||
self._stage(stage, dry_run)
|
||||
except Stop as stop:
|
||||
self.state["stopped"] = {"code": stop.code, "reason": stop.reason, "at": utc_iso(self.clock())}
|
||||
self._save()
|
||||
self.log(f"Stopped: {stop.reason}")
|
||||
return stop.code
|
||||
except (KabError, VoyageError) as error:
|
||||
self.state["stopped"] = {"code": EXIT_CONFIG, "reason": str(error), "at": utc_iso(self.clock())}
|
||||
self._save()
|
||||
self.log(f"Stopped: {error}")
|
||||
return EXIT_CONFIG
|
||||
self.state.pop("stopped", None)
|
||||
self.state["finished_at"] = utc_iso(self.clock())
|
||||
self._save()
|
||||
self.log("Dry run: nothing started." if dry_run else "Voyage complete.")
|
||||
return EXIT_OK
|
||||
|
||||
def _stage(self, stage: Stage, dry_run: bool) -> None:
|
||||
st = self._stage_state(stage.key)
|
||||
if st.get("state") == "completed":
|
||||
self.log(f"[{stage.key}] already completed: {st.get('execution_id')}")
|
||||
return
|
||||
wf = load_workflow(stage.workflow)
|
||||
workflow_id = stage.workflow_id or wf.workflow_id
|
||||
if stage.use_execution:
|
||||
st.update(execution_id=stage.use_execution, adopted=True, approval_ref=stage.approval_ref,
|
||||
workflow=wf.key, state="started")
|
||||
self._save()
|
||||
if not st.get("execution_id"):
|
||||
if not workflow_id:
|
||||
raise VoyageError(f"stage {stage.key}: no workflow_id in the voyage or workflows/{wf.key}.yaml")
|
||||
template = self._checked_template(stage, wf, workflow_id)
|
||||
inputs, context_text = self._inputs(stage, wf, workflow_id)
|
||||
st.update(workflow=wf.key, workflow_id=workflow_id, workflow_name=template.get("name"),
|
||||
workflow_version=template.get("version"), run_id=inputs.get("run_id"))
|
||||
self.ledger.write("inputs.json", shown_inputs(inputs), stage.key)
|
||||
if context_text is not None:
|
||||
self.ledger.write("context.yaml", context_text, stage.key)
|
||||
if dry_run:
|
||||
st["state"] = "planned"
|
||||
self._save()
|
||||
self.log(f"[{stage.key}] would start {template.get('name')} ({workflow_id}) with "
|
||||
f"{shown_inputs(inputs)}")
|
||||
return
|
||||
name = self.kab.project_name(self.voyage.project)
|
||||
body = cp1_agents.run_workflow().with_run_metadata(template, workflow_id, self.voyage.project, name)
|
||||
result = self.kab.execute(body, inputs)
|
||||
if not result.get("execution_id"):
|
||||
raise Stop(EXIT_STOPPED, f"stage {stage.key}: KAB started no execution: {str(result)[:300]}")
|
||||
st.update(execution_id=result["execution_id"], state="started", started_at=utc_iso(self.clock()))
|
||||
self._save()
|
||||
self.log(f"[{stage.key}] started {result['execution_id']} ({template.get('name')})")
|
||||
self._follow(stage, st)
|
||||
self._finish(stage, wf, st)
|
||||
|
||||
# ---- start ---------------------------------------------------------------------------------------------------
|
||||
def _checked_template(self, stage: Stage, wf: Workflow, workflow_id: str) -> dict:
|
||||
workflow = self.kab.workflow(workflow_id)
|
||||
template = workflow.get("template") or workflow
|
||||
if template.get("name") not in wf.names:
|
||||
raise VoyageError(f"stage {stage.key}: workflow {workflow_id} is {template.get('name')!r}, not one of "
|
||||
f"{wf.names} (ids change on re-import: update the voyage)")
|
||||
return template
|
||||
|
||||
def _resolve(self, stage: Stage, value: Any) -> Any:
|
||||
if not isinstance(value, dict):
|
||||
return value
|
||||
upstream = self.state["stages"].get(value["from"]) or {}
|
||||
if upstream.get("state") == "planned": # a dry run: the upstream run doesn't exist yet
|
||||
deferred = f"(at run time: {value['from']}.{value['value']})"
|
||||
return [deferred] if value["value"] == "package_artifact_ids" else deferred
|
||||
if upstream.get("state") != "completed":
|
||||
raise VoyageError(f"stage {stage.key} needs stage {value['from']} completed first")
|
||||
found = upstream.get(value["value"])
|
||||
if found in (None, "", []):
|
||||
raise VoyageError(f"stage {stage.key}: stage {value['from']} has no {value['value']}")
|
||||
return found
|
||||
|
||||
def _inputs(self, stage: Stage, wf: Workflow, workflow_id: str) -> tuple[dict, Optional[str]]:
|
||||
helpers = cp1_agents.run_workflow()
|
||||
given = {name: self._resolve(stage, value) for name, value in stage.inputs.items()}
|
||||
inputs: Dict[str, Any] = {}
|
||||
context_doc, context_text = None, None
|
||||
if stage.context:
|
||||
path = resolve_path(stage.context)
|
||||
context_doc = yaml.safe_load(path.read_text()) or {}
|
||||
from_file = (context_doc.get("workflow_inputs") or {}) if isinstance(context_doc, dict) else {}
|
||||
for key, value in from_file.items():
|
||||
value = "yes" if value is True else "no" if value is False else value
|
||||
if key in given and not (isinstance(value, str) and PLACEHOLDER.match(value)) and str(value) != str(given[key]):
|
||||
raise VoyageError(f"stage {stage.key}: {key} is {value!r} in {path.name} and {given[key]!r} in "
|
||||
"the voyage: keep one")
|
||||
inputs[key] = value if not isinstance(value, (int, float)) else str(value)
|
||||
inputs.update(given)
|
||||
if "run_id" in inputs:
|
||||
raise VoyageError(f"stage {stage.key}: run_id is generated for every run; remove it")
|
||||
if inputs.get("engagement_id") not in (None, self.voyage.engagement_id):
|
||||
raise VoyageError(f"stage {stage.key}: engagement_id {inputs.get('engagement_id')} is not the voyage's "
|
||||
f"{self.voyage.engagement_id}")
|
||||
inputs["engagement_id"] = self.voyage.engagement_id
|
||||
generated = helpers.new_ids()
|
||||
inputs["run_id"] = generated["run_id"]
|
||||
for name, doc_path in stage.documents.items():
|
||||
inputs[name] = resolve_path(doc_path).read_text()
|
||||
|
||||
if context_doc is not None:
|
||||
has_upstream = any(inputs.get(n) for n in helpers.UPSTREAM_INPUTS)
|
||||
problem = helpers.customer_problem(context_doc, has_upstream=has_upstream)
|
||||
if problem:
|
||||
raise VoyageError(f"stage {stage.key}: {stage.context}: {problem}")
|
||||
header = ("# Added by the Shakedown runner when the run started. Never edit these by hand.\n"
|
||||
"generated:\n" + "".join(f" {k}: {v}\n" for k, v in generated.items()) + "\n")
|
||||
path = resolve_path(stage.context)
|
||||
if wf.context == "upload":
|
||||
context_text = header + path.read_text()
|
||||
files = [{"name": path.name, "content": self._data_url(path, context_text.encode())}]
|
||||
for extra in helpers.context_uploads(path):
|
||||
files.append({"name": extra.name, "content": self._data_url(extra, extra.read_bytes())})
|
||||
inputs["file"] = files
|
||||
elif wf.context == "document":
|
||||
resolved = dict(context_doc)
|
||||
resolved["workflow_inputs"] = {k: inputs[k] for k in (context_doc.get("workflow_inputs") or {})
|
||||
if k in inputs}
|
||||
context_text = header + yaml.safe_dump(resolved, sort_keys=False, allow_unicode=True)
|
||||
inputs[wf.document_input] = context_text
|
||||
|
||||
leftover = sorted(k for k, v in inputs.items() if isinstance(v, str) and PLACEHOLDER.match(v.strip()))
|
||||
if leftover:
|
||||
raise VoyageError(f"stage {stage.key}: still a placeholder: {', '.join(leftover)}")
|
||||
sample = self.kab.api_sample(workflow_id)
|
||||
missing = sorted(f"{i['step_id']}.{i['name']}" for i in sample.get("required", [])
|
||||
if i["name"] != "file" and i["name"] not in inputs)
|
||||
if missing:
|
||||
raise VoyageError(f"stage {stage.key}: missing inputs {', '.join(missing)}")
|
||||
checkable = {k: v for k, v in inputs.items() if not (isinstance(v, str) and v.startswith("(at run time: "))}
|
||||
try:
|
||||
helpers.check_engagement(self.kab, checkable)
|
||||
except SystemExit as refused:
|
||||
raise VoyageError(f"stage {stage.key}: {refused}") from None
|
||||
return inputs, context_text
|
||||
|
||||
@staticmethod
|
||||
def _data_url(path: Path, data: bytes) -> str:
|
||||
kind = cp1_agents.run_workflow().upload_type(path)
|
||||
return f"data:{kind};base64,{base64.b64encode(data).decode()}"
|
||||
|
||||
# ---- follow --------------------------------------------------------------------------------------------------
|
||||
def _follow(self, stage: Stage, st: dict) -> None:
|
||||
eid = st["execution_id"]
|
||||
deadline = self.clock() + self.settings.stage_timeout_s
|
||||
while True:
|
||||
status = self.kab.status(eid)
|
||||
state = str(status.get("status", "")).lower()
|
||||
if state in DONE_OK or state in DONE_BAD:
|
||||
if st.get("state") == "needs_person": # a person decided and the run finished between two polls
|
||||
self._person_decided(stage, st)
|
||||
st["run_status"] = state
|
||||
self._save()
|
||||
return
|
||||
pending = (self.kab.reviews(eid) or {}).get("pending") or []
|
||||
if pending:
|
||||
self._at_gate(stage, st, status, pending)
|
||||
elif st.get("state") == "needs_person":
|
||||
self._person_decided(stage, st)
|
||||
if self.clock() > deadline:
|
||||
raise Stop(EXIT_TIMEOUT, f"stage {stage.key}: {eid} still {state} after "
|
||||
f"{self.settings.stage_timeout_s // 60} minutes")
|
||||
self.sleep(self.settings.poll_seconds)
|
||||
|
||||
def _at_gate(self, stage: Stage, st: dict, status: dict, pending: list) -> None:
|
||||
eid = st["execution_id"]
|
||||
review = pending[0]
|
||||
checkpoint = str(review.get("checkpoint_id"))
|
||||
if any(d.get("checkpoint_id") == checkpoint and d.get("decision") == "continue" for d in st["decisions"]):
|
||||
return # decided; KAB hasn't moved on yet
|
||||
system_may_decide = self.voyage.project in self.settings.gate_projects and len(pending) == 1
|
||||
if not system_may_decide:
|
||||
if st.get("state") != "needs_person":
|
||||
st.update(state="needs_person", waiting_since=self.clock(), checkpoint_id=checkpoint)
|
||||
self._save()
|
||||
why = "more than one pause" if len(pending) > 1 else "this project isn't one of the runner's gate_projects"
|
||||
self.log(f"[{stage.key}] {eid} waits at a review point for a person in KAB ({why}).")
|
||||
elif self.clock() - st.get("waiting_since", self.clock()) > self.settings.person_wait_s:
|
||||
raise Stop(EXIT_PERSON_TIMEOUT, f"stage {stage.key}: nobody decided {eid}'s review point in "
|
||||
f"{self.settings.person_wait_s // 3600} hours")
|
||||
return
|
||||
steps = self.kab.steps(eid)
|
||||
verdict = gate.decide(status, steps, review.get("step_id"))
|
||||
at = utc_iso(self.clock())
|
||||
record = {"checkpoint_id": checkpoint, "step_id": review.get("step_id"), "at": at, "by": "system",
|
||||
"verdict": verdict.verdict, "verdict_step": verdict.verdict_step, "reason": verdict.reason}
|
||||
if not verdict.proceed:
|
||||
record["decision"] = "left for a fix"
|
||||
st["decisions"].append(record)
|
||||
st["state"] = "needs_fix"
|
||||
self.ledger.write("steps-at-gate.json", steps, stage.key)
|
||||
self._save()
|
||||
raise Stop(EXIT_STOPPED, f"stage {stage.key}: not continued at the review point: {verdict.reason}. "
|
||||
f"{eid} stays paused for a person or a fix.")
|
||||
note = (f"Decided by the system (Shakedown runner, KAB automation identity), not a person: continue. "
|
||||
f"{verdict.reason}. Execution {eid}, run {st.get('run_id') or '?'}, voyage {self.voyage.name}, {at}.")
|
||||
self.kab.decide(eid, checkpoint, "continue", note)
|
||||
record.update(decision="continue", note=note)
|
||||
st["decisions"].append(record)
|
||||
st.update(state="started", approval_ref=f"Shakedown runner (system) continued {eid} at {at}: "
|
||||
f"critic verdict pass at {verdict.verdict_step}")
|
||||
self._save()
|
||||
self.log(f"[{stage.key}] continued {eid}: {verdict.reason}")
|
||||
|
||||
def _person_decided(self, stage: Stage, st: dict) -> None:
|
||||
"""The pause is gone and the run moves on: a person decided in KAB. Record who, as best KAB says."""
|
||||
eid = st["execution_id"]
|
||||
reviews = self.kab.reviews(eid) or {}
|
||||
decided = [r for key in ("history", "decided", "completed", "reviews") for r in (reviews.get(key) or [])
|
||||
if isinstance(r, dict) and str(r.get("checkpoint_id")) == str(st.get("checkpoint_id"))]
|
||||
last = decided[-1] if decided else {}
|
||||
who = last.get("decided_by") or last.get("reviewer") or last.get("user") or "a person in KAB"
|
||||
when = last.get("decided_at") or last.get("updated_at") or utc_iso(self.clock())
|
||||
decision = last.get("decision") or "continue"
|
||||
st["decisions"].append({"checkpoint_id": st.get("checkpoint_id"), "by": "person", "decision": decision,
|
||||
"who": who, "at": when})
|
||||
st.update(state="started", approval_ref=f"{who} decided {decision} on {eid} at {when}")
|
||||
st.pop("waiting_since", None)
|
||||
self._save()
|
||||
self.log(f"[{stage.key}] {who} decided {decision} on {eid}")
|
||||
|
||||
# ---- finish --------------------------------------------------------------------------------------------------
|
||||
def _finish(self, stage: Stage, wf: Workflow, st: dict) -> None:
|
||||
eid = st["execution_id"]
|
||||
status = self.kab.status(eid)
|
||||
steps = self.kab.steps(eid)
|
||||
self.ledger.write("status.json", status, stage.key)
|
||||
self.ledger.write("steps.json", steps, stage.key)
|
||||
run_status = str(status.get("status", "")).lower()
|
||||
if run_status not in DONE_OK:
|
||||
st["state"] = "failed"
|
||||
self._save()
|
||||
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} ended {run_status or 'with no status'}")
|
||||
artifact = self._package_artifact(eid, status, steps, st.get("workflow_id"))
|
||||
if artifact is None:
|
||||
st["state"] = "failed"
|
||||
self._save()
|
||||
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {eid} completed with no package (no output has a manifest "
|
||||
"with a package_version)")
|
||||
st["package_artifact_ids"] = [artifact]
|
||||
if not st.get("approval_ref"):
|
||||
st["approval_ref"] = f"{eid} completed with no review point decided by the runner"
|
||||
if stage.case:
|
||||
case = yaml.safe_load(resolve_path(stage.case).read_text())
|
||||
rows, failed = self.evaluator(wf.feature, case, steps)
|
||||
evaluation = {"case": stage.case, "failed": failed,
|
||||
"checks": [dict(zip(("status", "severity", "where", "id", "detail"), r)) for r in rows]}
|
||||
self.ledger.write("evaluation.json", evaluation, stage.key)
|
||||
st["evaluation"] = {"failed": failed, "checks": len(rows)}
|
||||
if failed.get("critical"):
|
||||
st["state"] = "checks_failed"
|
||||
self._save()
|
||||
raise Stop(EXIT_STOPPED, f"stage {stage.key}: {failed['critical']} critical check(s) failed on {eid} "
|
||||
f"(see {stage.key}/evaluation.json)")
|
||||
st.update(state="completed", completed_at=utc_iso(self.clock()))
|
||||
self._save()
|
||||
self.log(f"[{stage.key}] completed {eid}; package {artifact}")
|
||||
|
||||
def _package_artifact(self, eid: str, status: dict, steps: Any, workflow_id: Optional[str]) -> Optional[str]:
|
||||
"""`<execution id>_step_<n>`, n the package step's 1-based place in the executed template."""
|
||||
outputs = gate.step_outputs(steps)
|
||||
holder = next((k for k, v in outputs.items() if isinstance(v, dict) and isinstance(v.get("manifest"), dict)
|
||||
and "package_version" in v["manifest"]), None)
|
||||
if holder is None:
|
||||
return None
|
||||
order = gate.template_order(status)
|
||||
if holder not in order and workflow_id:
|
||||
workflow = self.kab.workflow(workflow_id)
|
||||
order = [str(s.get("id")) for s in ((workflow.get("template") or workflow).get("steps") or [])]
|
||||
if holder not in order:
|
||||
return None
|
||||
return f"{eid}_step_{order.index(holder) + 1}"
|
||||
116
shakedown_lib/voyage.py
Normal file
116
shakedown_lib/voyage.py
Normal file
@@ -0,0 +1,116 @@
|
||||
"""Voyages (one engagement's chain of use cases) and the workflow registry, as loaded from this repository.
|
||||
|
||||
A path is relative to this repository, or to the cp1-agents clone when it starts with `cp1-agents:` (the committed
|
||||
F01 to F03 cases stay there and are read in place, plan §3).
|
||||
|
||||
A stage input is a literal, or a reference to an earlier stage's result: `{from: <stage key>, value: <name>}`, where
|
||||
the name is `execution_id`, `approval_ref` or `package_artifact_ids` (`["<execution id>_step_<n>"]` of its package
|
||||
step). References are resolved when the stage starts; the files themselves are never edited.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import yaml
|
||||
|
||||
from . import cp1_agents
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
CONTEXT_ADAPTERS = {"upload", "document", "none"}
|
||||
STAGE_VALUES = {"execution_id", "approval_ref", "package_artifact_ids"}
|
||||
|
||||
|
||||
class VoyageError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def resolve_path(text: str) -> Path:
|
||||
if text.startswith("cp1-agents:"):
|
||||
return cp1_agents.clone_dir() / text.split(":", 1)[1]
|
||||
path = Path(text)
|
||||
return path if path.is_absolute() else REPO / path
|
||||
|
||||
|
||||
@dataclass
|
||||
class Workflow:
|
||||
key: str
|
||||
feature: str
|
||||
names: List[str]
|
||||
context: str
|
||||
document_input: Optional[str] = None
|
||||
workflow_id: Optional[str] = None
|
||||
|
||||
|
||||
def load_workflow(key: str) -> Workflow:
|
||||
path = REPO / "workflows" / f"{key}.yaml"
|
||||
if not path.is_file():
|
||||
raise VoyageError(f"workflow {key!r} is not registered: no {path.relative_to(REPO)}")
|
||||
doc = yaml.safe_load(path.read_text()) or {}
|
||||
wf = Workflow(key=doc.get("key", key), feature=doc["feature"], names=list(doc["names"]), context=doc["context"],
|
||||
document_input=doc.get("document_input"), workflow_id=doc.get("workflow_id"))
|
||||
if wf.context not in CONTEXT_ADAPTERS:
|
||||
raise VoyageError(f"{path.name}: context must be one of {sorted(CONTEXT_ADAPTERS)}")
|
||||
if wf.context == "document" and not wf.document_input:
|
||||
raise VoyageError(f"{path.name}: a document context names its run input in document_input")
|
||||
return wf
|
||||
|
||||
|
||||
@dataclass
|
||||
class Stage:
|
||||
key: str
|
||||
workflow: str
|
||||
workflow_id: Optional[str] = None
|
||||
context: Optional[str] = None
|
||||
case: Optional[str] = None
|
||||
inputs: Dict[str, Any] = field(default_factory=dict)
|
||||
documents: Dict[str, str] = field(default_factory=dict)
|
||||
use_execution: Optional[str] = None
|
||||
approval_ref: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Voyage:
|
||||
name: str
|
||||
engagement_id: str
|
||||
project: str
|
||||
stages: List[Stage]
|
||||
capability: Optional[str] = None
|
||||
source: Optional[Path] = None
|
||||
|
||||
|
||||
def load_voyage(path: Path) -> Voyage:
|
||||
doc = yaml.safe_load(Path(path).read_text()) or {}
|
||||
for required in ("voyage", "engagement_id", "project", "stages"):
|
||||
if not doc.get(required):
|
||||
raise VoyageError(f"{path}: `{required}` is required")
|
||||
stages, seen = [], set()
|
||||
for raw in doc["stages"]:
|
||||
stage = Stage(key=raw["key"], workflow=raw["workflow"], workflow_id=raw.get("workflow_id"),
|
||||
context=raw.get("context"), case=raw.get("case"), inputs=dict(raw.get("inputs") or {}),
|
||||
documents=dict(raw.get("documents") or {}), use_execution=raw.get("use_execution"),
|
||||
approval_ref=raw.get("approval_ref"))
|
||||
if stage.key in seen:
|
||||
raise VoyageError(f"{path}: stage key {stage.key!r} appears twice")
|
||||
if stage.use_execution and not stage.approval_ref:
|
||||
raise VoyageError(f"{path}: stage {stage.key} uses execution {stage.use_execution}: give its approval_ref "
|
||||
"(who continued it at its review point, and when)")
|
||||
for name, value in stage.inputs.items():
|
||||
if isinstance(value, dict):
|
||||
if value.get("from") not in seen or value.get("value") not in STAGE_VALUES:
|
||||
raise VoyageError(f"{path}: stage {stage.key} input {name}: `from` must name an earlier stage and "
|
||||
f"`value` one of {sorted(STAGE_VALUES)}")
|
||||
seen.add(stage.key)
|
||||
stages.append(stage)
|
||||
return Voyage(name=doc["voyage"], engagement_id=doc["engagement_id"], project=doc["project"], stages=stages,
|
||||
capability=doc.get("capability"), source=Path(path))
|
||||
|
||||
|
||||
def load_config(path: Path | None = None) -> dict:
|
||||
"""Runner settings, from a committed file so a change to them is reviewed. The environment can't widen them."""
|
||||
path = path or REPO / "config" / "runner.yaml"
|
||||
doc = yaml.safe_load(path.read_text()) or {}
|
||||
doc["gate_projects"] = {str(p) for p in (doc.get("gate_projects") or [])}
|
||||
return doc
|
||||
0
tests/__init__.py
Normal file
0
tests/__init__.py
Normal file
320
tests/test_runner.py
Normal file
320
tests/test_runner.py
Normal file
@@ -0,0 +1,320 @@
|
||||
"""The runner against a fake KAB: python3 -m unittest discover -s tests (needs the cp1-agents clone beside this repo,
|
||||
or CP1_AGENTS_DIR)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
import textwrap
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import yaml
|
||||
|
||||
from shakedown_lib import gate
|
||||
from shakedown_lib.auth import AuthError, ClientCredentials
|
||||
from shakedown_lib.kab import Kab
|
||||
from shakedown_lib.ledger import Ledger
|
||||
from shakedown_lib.runner import EXIT_CONFIG, EXIT_OK, EXIT_STOPPED, Runner, Settings
|
||||
from shakedown_lib.voyage import load_voyage
|
||||
|
||||
PROJECT = "proj-1"
|
||||
ENG = "ENG-TEST-001"
|
||||
NAMES = {"wf-f01": "CP1 - Agentic Landing Zone - Discovery", "wf-f02": "CP1 - Agentic Landing Zone - Assessment",
|
||||
"wf-f03": "CP1 - Agentic Landing Zone - Strategy Synthesis"}
|
||||
|
||||
|
||||
def template(name: str, upload: bool) -> dict:
|
||||
first = "u1" if upload else "a1"
|
||||
return {"name": name, "version": "1.0", "steps": [{"id": first}] + [{"id": f"s{i}"} for i in range(1, 7)] + [{"id": "f1"}]}
|
||||
|
||||
|
||||
class FakeKab:
|
||||
"""Runs move: running -> paused at s5 (one pending review) -> after `continue`, completed with a package at s6."""
|
||||
|
||||
def __init__(self, verdict: str = "pass", polls_before_pause: int = 1, person_after: int | None = None,
|
||||
names: dict | None = None) -> None:
|
||||
self.verdict = verdict
|
||||
self.polls_before_pause = polls_before_pause
|
||||
self.person_after = person_after
|
||||
self.names = names or NAMES
|
||||
self.runs: dict = {"exec_f01_done": {"workflow": "wf-f01", "inputs": {"engagement_id": ENG}, "phase": "done",
|
||||
"polls": 0, "history": []}}
|
||||
self.executed: list = []
|
||||
self.decisions: list = []
|
||||
|
||||
def transport(self, method, url, headers, data):
|
||||
assert headers["Authorization"].startswith("Bearer "), "no token sent"
|
||||
path = urlparse(url).path
|
||||
body = json.loads(data) if data else None
|
||||
reply = self.route(method, path, body)
|
||||
return 200, json.dumps(reply).encode()
|
||||
|
||||
def _status(self, eid):
|
||||
run = self.runs[eid]
|
||||
state = {"running": "running", "paused": "attention", "done": "completed"}[run["phase"]]
|
||||
return {"execution_id": eid, "status": state, "inputs": run["inputs"],
|
||||
"template": template(self.names[run["workflow"]], run["workflow"] == "wf-f01")}
|
||||
|
||||
def _steps(self, eid):
|
||||
run = self.runs[eid]
|
||||
steps = {"s4": {"response": json.dumps({"phase": "critique", "status": "completed",
|
||||
"verdict": {"verdict": self.verdict}})}}
|
||||
if run["phase"] == "done":
|
||||
steps["s6"] = {"output": {"engagement_id": ENG, "manifest": {"package_version": "1.0"}}}
|
||||
return {"execution_id": eid, "steps": steps}
|
||||
|
||||
def route(self, method, path, body):
|
||||
if path.startswith("/api/v1/workflow-catalog/workflows/"):
|
||||
wid = path.rsplit("/", 1)[1]
|
||||
return {"workflow": {"template": template(self.names[wid], wid == "wf-f01")}}
|
||||
if path.endswith("/api-sample"):
|
||||
return {"required": []}
|
||||
if path.startswith("/api/v1/projects/"):
|
||||
return {"project": {"name": "Test project"}}
|
||||
if path == "/api/workflows/execute":
|
||||
wid = body["template"]["metadata"]["workflow_id"]
|
||||
eid = f"exec_{len(self.executed) + 1}"
|
||||
self.executed.append({"workflow": wid, "inputs": body["inputs"], "template": body["template"]})
|
||||
self.runs[eid] = {"workflow": wid, "inputs": body["inputs"], "phase": "running", "polls": 0, "history": []}
|
||||
return {"execution_id": eid, "status": "running"}
|
||||
if path.startswith("/api/workflows/status/"):
|
||||
eid = path.rsplit("/", 1)[1]
|
||||
run = self.runs[eid]
|
||||
run["polls"] += 1
|
||||
if run["phase"] == "running" and run["polls"] > self.polls_before_pause:
|
||||
run["phase"] = "paused"
|
||||
run["paused_at"] = run["polls"]
|
||||
if run["phase"] == "paused" and self.person_after is not None and run["polls"] - run["paused_at"] >= self.person_after:
|
||||
run["phase"] = "done"
|
||||
run["history"].append({"checkpoint_id": f"cp-{eid}", "decision": "continue",
|
||||
"decided_by": "lead@example.com", "decided_at": "2026-11-16T10:00:00Z"})
|
||||
return self._status(eid)
|
||||
if path.endswith("/steps"):
|
||||
return self._steps(path.split("/")[-2])
|
||||
if path.endswith("/reviews") and method == "GET":
|
||||
eid = path.split("/")[-2]
|
||||
run = self.runs[eid]
|
||||
pending = [{"checkpoint_id": f"cp-{eid}", "step_id": "s5"}] if run["phase"] == "paused" else []
|
||||
return {"pending": pending, "history": run["history"]}
|
||||
if "/reviews/" in path and method == "POST":
|
||||
eid = path.split("/")[-3]
|
||||
self.decisions.append({"execution": eid, **body})
|
||||
if body["decision"] == "continue":
|
||||
self.runs[eid]["phase"] = "done"
|
||||
return {"ok": True}
|
||||
raise AssertionError(f"unexpected {method} {path}")
|
||||
|
||||
|
||||
class Tokens:
|
||||
def __init__(self):
|
||||
self.calls = 0
|
||||
|
||||
def token(self, force=False):
|
||||
self.calls += 1
|
||||
return "t"
|
||||
|
||||
|
||||
class Clock:
|
||||
def __init__(self):
|
||||
self.now = 1_800_000_000.0
|
||||
|
||||
def __call__(self):
|
||||
return self.now
|
||||
|
||||
def sleep(self, seconds):
|
||||
self.now += seconds
|
||||
|
||||
|
||||
def write_voyage(folder: Path, f01_adopted: bool = True) -> Path:
|
||||
(folder / "context.yaml").write_text(textwrap.dedent(f"""\
|
||||
workflow_inputs:
|
||||
engagement_id: {ENG}
|
||||
provider: sample
|
||||
include_cmdb: yes
|
||||
customer: {{id: CUS-TEST, name: Test}}
|
||||
"""))
|
||||
(folder / "run-context.yaml").write_text(textwrap.dedent(f"""\
|
||||
workflow_inputs:
|
||||
engagement_id: {ENG}
|
||||
assessment_execution_id: <assessment execution id>
|
||||
assessment_approval_ref: "<who continued it>"
|
||||
context:
|
||||
customer: {{id: CUS-TEST}}
|
||||
"""))
|
||||
(folder / "rate-card.json").write_text('{"rates": []}')
|
||||
(folder / "case.yaml").write_text("id: test\nchecks: []\n")
|
||||
f01 = {"key": "f01", "workflow": "alz-f01-discovery", "workflow_id": "wf-f01", "context": f"{folder}/context.yaml"}
|
||||
if f01_adopted:
|
||||
f01.update(use_execution="exec_f01_done", approval_ref="approved by a person")
|
||||
doc = {"voyage": ENG, "engagement_id": ENG, "project": PROJECT, "stages": [
|
||||
f01,
|
||||
{"key": "f02", "workflow": "alz-f02-assessment", "workflow_id": "wf-f02",
|
||||
"inputs": {"discovery_execution_id": {"from": "f01", "value": "execution_id"},
|
||||
"discovery_approval_ref": {"from": "f01", "value": "approval_ref"},
|
||||
"artifact_ids": {"from": "f01", "value": "package_artifact_ids"}},
|
||||
"documents": {"rate_card_document": f"{folder}/rate-card.json"}, "case": f"{folder}/case.yaml"},
|
||||
{"key": "f03", "workflow": "alz-f03-strategy", "workflow_id": "wf-f03", "context": f"{folder}/run-context.yaml",
|
||||
"inputs": {"assessment_execution_id": {"from": "f02", "value": "execution_id"},
|
||||
"assessment_approval_ref": {"from": "f02", "value": "approval_ref"},
|
||||
"artifact_ids": {"from": "f02", "value": "package_artifact_ids"}}}]}
|
||||
voyage = folder / "voyage.yaml"
|
||||
voyage.write_text(yaml.safe_dump(doc, sort_keys=False))
|
||||
return voyage
|
||||
|
||||
|
||||
class RunnerTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.dir = Path(self.tmp.name)
|
||||
self.clock = Clock()
|
||||
self.evaluated = []
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def runner(self, fake: FakeKab, gate_projects=(PROJECT,), ledger: Ledger | None = None, adopted=True, **settings):
|
||||
voyage = load_voyage(write_voyage(self.dir, adopted))
|
||||
kab = Kab(Tokens(), api="https://kab.test", tenant="t", project=PROJECT, transport=fake.transport,
|
||||
sleep=lambda s: None)
|
||||
|
||||
def evaluator(feature, case, steps):
|
||||
self.evaluated.append(feature)
|
||||
return [("pass", "critical", "manifest", "c1", "")], {"critical": 0, "major": 0}
|
||||
|
||||
return Runner(kab=kab, ledger=ledger or Ledger(self.dir / "ledger"), voyage=voyage,
|
||||
settings=Settings(gate_projects=set(gate_projects), poll_seconds=30, **settings),
|
||||
evaluator=evaluator, clock=self.clock, sleep=self.clock.sleep, log=lambda m: None)
|
||||
|
||||
def test_whole_voyage_decides_gates_in_its_own_project(self):
|
||||
fake = FakeKab()
|
||||
runner = self.runner(fake)
|
||||
self.assertEqual(runner.run(), EXIT_OK)
|
||||
self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"])
|
||||
f02, f03 = fake.executed
|
||||
self.assertEqual(f02["inputs"]["discovery_execution_id"], "exec_f01_done")
|
||||
self.assertEqual(f02["inputs"]["artifact_ids"], ["exec_f01_done_step_7"])
|
||||
self.assertEqual(f02["inputs"]["rate_card_document"], '{"rates": []}')
|
||||
self.assertEqual(f02["template"]["metadata"]["project_id"], PROJECT)
|
||||
self.assertEqual(f03["inputs"]["artifact_ids"], ["exec_1_step_7"])
|
||||
self.assertIn("Shakedown runner (system) continued exec_1", f03["inputs"]["assessment_approval_ref"])
|
||||
document = f03["inputs"]["run_context_document"]
|
||||
self.assertIn("assessment_execution_id: exec_1", document)
|
||||
self.assertNotIn("<assessment execution id>", document)
|
||||
self.assertEqual([d["decision"] for d in fake.decisions], ["continue", "continue"])
|
||||
self.assertTrue(all("not a person" in d["note"] for d in fake.decisions))
|
||||
state = json.loads((self.dir / "ledger" / "state.json").read_text())
|
||||
self.assertEqual({k: v["state"] for k, v in state["stages"].items()},
|
||||
{"f01": "completed", "f02": "completed", "f03": "completed"})
|
||||
self.assertEqual(self.evaluated, ["f02"])
|
||||
self.assertTrue((self.dir / "ledger" / "f03" / "context.yaml").is_file())
|
||||
|
||||
def test_critic_fail_is_never_continued(self):
|
||||
fake = FakeKab(verdict="fail")
|
||||
self.assertEqual(self.runner(fake).run(), EXIT_STOPPED)
|
||||
self.assertEqual(fake.decisions, [])
|
||||
state = json.loads((self.dir / "ledger" / "state.json").read_text())
|
||||
self.assertEqual(state["stages"]["f02"]["state"], "needs_fix")
|
||||
self.assertIn("critic verdict is 'fail'", state["stopped"]["reason"])
|
||||
|
||||
def test_other_projects_wait_for_a_person(self):
|
||||
fake = FakeKab(person_after=3)
|
||||
self.assertEqual(self.runner(fake, gate_projects=()).run(), EXIT_OK)
|
||||
self.assertEqual(fake.decisions, [])
|
||||
f03 = fake.executed[1]
|
||||
self.assertEqual(f03["inputs"]["assessment_approval_ref"],
|
||||
"lead@example.com decided continue on exec_1 at 2026-11-16T10:00:00Z")
|
||||
|
||||
def test_nobody_deciding_times_out(self):
|
||||
fake = FakeKab()
|
||||
code = self.runner(fake, gate_projects=(), person_wait_s=120).run()
|
||||
self.assertEqual(code, 4)
|
||||
self.assertEqual(fake.decisions, [])
|
||||
|
||||
def test_a_stale_workflow_id_is_refused(self):
|
||||
fake = FakeKab(names={**NAMES, "wf-f02": "CP1 - Agentic Landing Zone - Assessment (individual agents)"})
|
||||
self.assertEqual(self.runner(fake).run(), EXIT_CONFIG)
|
||||
self.assertEqual(fake.executed, [])
|
||||
|
||||
def test_resume_never_starts_a_run_twice(self):
|
||||
fake = FakeKab(verdict="fail")
|
||||
ledger = Ledger(self.dir / "ledger")
|
||||
self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_STOPPED)
|
||||
fake.verdict = "pass" # say a fix was deployed and the stage rerun in KAB
|
||||
self.assertEqual(self.runner(fake, ledger=ledger).run(), EXIT_OK)
|
||||
self.assertEqual([e["workflow"] for e in fake.executed], ["wf-f02", "wf-f03"])
|
||||
|
||||
def test_dry_run_starts_nothing(self):
|
||||
fake = FakeKab()
|
||||
self.assertEqual(self.runner(fake, adopted=False).run(dry_run=True), EXIT_OK)
|
||||
self.assertEqual(fake.executed, [])
|
||||
shown = json.loads((self.dir / "ledger" / "f02" / "inputs.json").read_text())
|
||||
self.assertEqual(shown["discovery_execution_id"], "(at run time: f01.execution_id)")
|
||||
f01 = json.loads((self.dir / "ledger" / "f01" / "inputs.json").read_text())
|
||||
self.assertEqual(f01["file"], ["context.yaml"])
|
||||
self.assertEqual(f01["include_cmdb"], "yes")
|
||||
|
||||
|
||||
class GateTest(unittest.TestCase):
|
||||
def status(self):
|
||||
return {"status": "attention", "template": {"steps": [{"id": i} for i in ("a1", "s1", "s2", "s3", "s4", "s5", "s6")]}}
|
||||
|
||||
def test_pass(self):
|
||||
steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": "pass"}}}}
|
||||
self.assertTrue(gate.decide(self.status(), steps, "s5").proceed)
|
||||
|
||||
def test_a_later_fail_overrides_an_earlier_pass(self):
|
||||
steps = {"steps": {"s3": {"phase": "x", "verdict": "pass"}, "s4": {"phase": "critique", "verdict": {"verdict": "fail"}}}}
|
||||
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
|
||||
|
||||
def test_no_verdict(self):
|
||||
self.assertFalse(gate.decide(self.status(), {"steps": {"s1": {"phase": "plan"}}}, "s5").proceed)
|
||||
|
||||
def test_a_failed_step(self):
|
||||
steps = {"steps": {"s2": {"phase": "collect", "status": "failed"}, "s4": {"phase": "c", "verdict": "pass"}}}
|
||||
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
|
||||
|
||||
def test_unparseable_verdict(self):
|
||||
steps = {"steps": {"s4": {"phase": "critique", "verdict": {"verdict": None}}}}
|
||||
self.assertFalse(gate.decide(self.status(), steps, "s5").proceed)
|
||||
|
||||
|
||||
class AuthTest(unittest.TestCase):
|
||||
def test_renews_before_expiry_and_hides_the_secret(self):
|
||||
clock = Clock()
|
||||
issued = []
|
||||
|
||||
def post(url, form):
|
||||
issued.append(form["grant_type"])
|
||||
return {"access_token": f"tok{len(issued)}", "expires_in": 300}
|
||||
|
||||
tokens = ClientCredentials("kab-automation", "s3cret", post=post, clock=clock)
|
||||
self.assertEqual(tokens.token(), "tok1")
|
||||
clock.now += 200
|
||||
self.assertEqual(tokens.token(), "tok1")
|
||||
clock.now += 50 # inside the last minute
|
||||
self.assertEqual(tokens.token(), "tok2")
|
||||
self.assertEqual(issued, ["client_credentials", "client_credentials"])
|
||||
self.assertNotIn("s3cret", repr(tokens))
|
||||
|
||||
def test_no_secret(self):
|
||||
with self.assertRaises(AuthError):
|
||||
ClientCredentials("kab-automation", "")
|
||||
|
||||
def test_a_401_renews_the_token_once(self):
|
||||
calls = []
|
||||
|
||||
class T:
|
||||
def token(self, force=False):
|
||||
calls.append(force)
|
||||
return "x"
|
||||
|
||||
replies = iter([(401, b""), (200, b'{"ok": 1}')])
|
||||
kab = Kab(T(), transport=lambda *a: next(replies), sleep=lambda s: None)
|
||||
self.assertEqual(kab.call("GET", "/x"), {"ok": 1})
|
||||
self.assertEqual(calls, [False, True])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
8
workflows/alz-f01-discovery.yaml
Normal file
8
workflows/alz-f01-discovery.yaml
Normal file
@@ -0,0 +1,8 @@
|
||||
# ALZ-F01 Discovery (ADO Feature 3214441). Registry entry: plan §3c.
|
||||
key: alz-f01-discovery
|
||||
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
|
||||
feature: f01 # the feature name cp1-agents scripts/evaluate_case.py takes
|
||||
# Workflow names KAB may show for it. A run's workflow id is checked against these before it starts, because ids
|
||||
# change on every re-import (same names as the docs portal's registry/workflows.yaml).
|
||||
names: ["CP1 - Agentic Landing Zone - Discovery"]
|
||||
context: upload # the run context is uploaded to u1 (file); its workflow_inputs become run inputs
|
||||
8
workflows/alz-f02-assessment.yaml
Normal file
8
workflows/alz-f02-assessment.yaml
Normal file
@@ -0,0 +1,8 @@
|
||||
# ALZ-F02 Assessment & Disposition (ADO Feature 3214442). Registry entry: plan §3c.
|
||||
key: alz-f02-assessment
|
||||
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
|
||||
feature: f02
|
||||
# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change
|
||||
# on every re-import).
|
||||
names: ["CP1 - Agentic Landing Zone - Assessment"]
|
||||
context: none # no context file: inputs come from the voyage (an alignment finding, plan §3b)
|
||||
9
workflows/alz-f03-strategy.yaml
Normal file
9
workflows/alz-f03-strategy.yaml
Normal file
@@ -0,0 +1,9 @@
|
||||
# ALZ-F03 Strategy Synthesis (ADO Feature 3214443). Registry entry: plan §3c.
|
||||
key: alz-f03-strategy
|
||||
pattern: sub-agents # the 2026-11-15 pivot: the individual-pattern agents were removed from `agents`
|
||||
feature: f03
|
||||
# Workflow names KAB may show for it. A workflow id is checked against these before a stage starts (ids change
|
||||
# on every re-import).
|
||||
names: ["CP1 - Agentic Landing Zone - Strategy Synthesis"]
|
||||
context: document # the run context is sent as text in one run input (F03 ADR 0002)
|
||||
document_input: run_context_document
|
||||
Reference in New Issue
Block a user