Plan: use cases as scenarios, voyages, variations and the existing use cases
Use cases are scenario folders (run context, uploads, upstream execution), chained per engagement as voyages, with variations that state how outputs must respond to a changed business input. Inventory of the F01, F02 and F03 use cases in cp1-agents, their expectations and gaps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -6,7 +6,7 @@ enters service. This app does the same for every workflow before its outputs rea
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §8) |
|
||||
| Date | 2026-11-09 (revised the same day with the product owner's decisions, §9) |
|
||||
| Status | Draft. Nothing is built yet |
|
||||
| Repository | [`agents/cp1-shakedown`](https://gitea.kyndemo.live/agents/cp1-shakedown) on gitea (created 2026-11-09). It holds use cases and experiments, kept apart from the agents and workflows in `cp1-agents` |
|
||||
| Runs on | The `crucible-kab-dev-aks` cluster |
|
||||
@@ -40,14 +40,14 @@ without anyone pointing it at them.
|
||||
|
||||
## 2. What Shakedown does
|
||||
|
||||
Four capabilities, each usable through the UI and by the deep agent (§4):
|
||||
Four capabilities, each usable through the UI and by the deep agent (§5):
|
||||
|
||||
1. **Rate.** Score any run on three layers. A run's score is never a single number; each layer is shown on its own.
|
||||
- *Checks:* the existing case format (`case.yaml` `checks:` with `contains`, `item.match`, `absent`, `equals`,
|
||||
severities), run by a generic case runner that works for any workflow, not only F01.
|
||||
- *Invariants in code:* the run's figures trace to its facts, the package digests recompute, ids resolve across
|
||||
steps, every `evidence_ref` names a real execution, and token and time `limits:` are met.
|
||||
- *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§5).
|
||||
- *Business value:* a rubric scored by Claude Sonnet with quoted evidence (§6).
|
||||
2. **Experiment.** Change one variable and measure the effect against a baseline:
|
||||
- *Stability:* the same input N times (default 5), with the spread for each output field.
|
||||
- *Step replay:* re-send one step's recorded input straight to an agent over A2A (`message/send` on
|
||||
@@ -55,9 +55,12 @@ Four capabilities, each usable through the UI and by the deep agent (§4):
|
||||
tokens are spent.
|
||||
- *Variants:* prompt, model, `LITELLM_MAX_TOKENS` or a code hook changed in a shadow deployment of the agent, with
|
||||
the same replay inputs sent to the baseline and to the variant.
|
||||
3. **Build use cases.** Cases are drafted from ADO stories (acceptance criteria and worked examples), from earlier
|
||||
failures, and from adversarial patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate).
|
||||
Synthetic estates go in `alz-discovery-sample`, which is the one server that may be extended for eval data.
|
||||
3. **Build use cases and their variations.** A use case is a business scenario fed to a workflow (§3a): its run
|
||||
context, uploads and upstream execution, plus the expectations its outputs are judged against. Bosun drafts new
|
||||
ones from ADO stories (acceptance criteria and worked examples), from earlier failures, and from adversarial
|
||||
patterns (prompt injection, missing sources, a contradictory CMDB, an empty estate). From each base use case it
|
||||
derives **variations**: one business input changed, with the direction the output should move. Synthetic estates go
|
||||
in `alz-discovery-sample`, which is the one server that may be extended for eval data.
|
||||
4. **Align.** Check every workflow hand-off, F01 package → F02 intake → F03 intake:
|
||||
- *Statically:* the producer's `output.json` against the consumer's `input.json`, and the fields each consumer's
|
||||
prompt reads.
|
||||
@@ -79,11 +82,14 @@ cp1-shakedown/
|
||||
│ ├── subagents/ # one module per sub-agent: prompt + tool list
|
||||
│ └── tools/ # KAB, ledger, replay, cluster-read, repo-read, ADO-read, gitea-PR
|
||||
├── shakedown_lib/ # case runner, invariants, rubric scoring, KAB client, ledger
|
||||
├── use-cases/<workstream>/<workflow>/<case-id>/
|
||||
│ ├── case.yaml # checks, value bars, limits; frozen (hashed) before the first run
|
||||
│ ├── context.yaml # run context (workflow_inputs, customer, plan, context): no run_id, no credentials
|
||||
│ └── uploads/, README.md
|
||||
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§5)
|
||||
├── use-cases/<workflow>/<use-case-id>/ # e.g. use-cases/alz-f03-strategy/azure-crucible-kab-dev-strategy/
|
||||
│ ├── run-context.yaml # the scenario, in the shape its workflow takes; no run_id, no credentials
|
||||
│ ├── uploads/ # CMDB exports, rate cards …
|
||||
│ ├── expectations.yaml # checks, value bars, limits; frozen (hashed) before the first run
|
||||
│ ├── variations.yaml # derived scenarios, each with its expected direction of change (§3a)
|
||||
│ └── README.md # the business situation in a paragraph, and its upstream use case
|
||||
├── voyages/<engagement-id>.yaml # one engagement's chain: F01 → F02 → F03 use cases and how ids flow between them
|
||||
├── rubrics/<workflow>.yaml # business-value rubric for each workflow (§6)
|
||||
├── experiments/EXP-NNN-<slug>/
|
||||
│ ├── experiment.yaml # hypothesis, one variable, baseline, variants, N, metric, stop rule
|
||||
│ └── results.md # conclusion, the scores table, links to ledger runs; written by Bosun, reviewed by a person
|
||||
@@ -91,9 +97,11 @@ cp1-shakedown/
|
||||
└── deploy/ # Shakedown's own Kubernetes manifests, with <placeholder_for_*> as in cp1-agents
|
||||
```
|
||||
|
||||
- **Existing cases stay where they are.** F01's 7 workflow cases, F02's 2, and each agent's `eval/cases.yaml` remain
|
||||
in `cp1-agents`. Shakedown reads them by path from a read-only clone pinned to a commit, and lists them beside its
|
||||
own cases with their source shown. Only **new** use cases and **all** experiments go into the Shakedown repository.
|
||||
- **New use cases start here, from F03.** F03's `azure-crucible-kab-dev-strategy` (written 2026-11-09, not yet
|
||||
committed to cp1-agents) is the first use case in this repository, and every later one is created here. The
|
||||
committed F01 cases (7) and F02 cases (2), and each agent's `eval/cases.yaml`, stay in `cp1-agents`. Shakedown reads
|
||||
them by path from a read-only clone pinned to a commit, and lists them beside its own with their source shown.
|
||||
**All** experiments go into this repository.
|
||||
- **No change to cp1-agents.** Shakedown imports the scripts it reuses (`kab_run_workflow.py`'s `Kab`,
|
||||
`load_context` and `new_ids`; `evaluate_workflow_run.py`; `collect_run_usage.py`; the package agents'
|
||||
`scorecard.py`) from that pinned clone by path, instead of extracting a shared package from cp1-agents.
|
||||
@@ -101,6 +109,111 @@ cp1-shakedown/
|
||||
- **Changes reach the repositories as pull requests.** Bosun opens a gitea pull request for a new case, an experiment
|
||||
record or a proposed agent patch (to cp1-agents). It never pushes to a main branch, and a person merges.
|
||||
|
||||
## 3a. Use cases, voyages and variations
|
||||
|
||||
**A use case is a business scenario, not a test script.** Today's folders show its parts, spread unevenly across
|
||||
Features:
|
||||
|
||||
| Feature | Scenario | How it reaches the run | Expectations |
|
||||
|---|---|---|---|
|
||||
| F01 | `context.yaml` (customer, plan, scope, sources) and CMDB uploads for live cases; `brief.txt` for the sample cases | Uploaded through `kab_run_workflow.py --context`, or `--file` | `case.yaml` checks in 5 of 7; `azure-live` and `gcp-live` have none |
|
||||
| F02 | `rate-card.json`; everything else is typed as `--input` flags (execution id, approval reference, artifact id) | Run inputs, including `rate_card_document`; no context file, unlike F01 and F03 | Written as prose in the README only |
|
||||
| F03 | `run-context.yaml` (objectives, constraints, priorities, planning assumptions) | The run input `run_context_document`, as text (F03 ADR 0002) | None |
|
||||
|
||||
Shakedown standardises the folder, not the workflows:
|
||||
- **`run-context.yaml` stays in its workflow's own format.** A small adapter for each workflow knows how to deliver
|
||||
it (upload or run input). Shakedown never asks cp1-agents to change its inputs. That the three Features take their
|
||||
context three ways is reported as an alignment finding (§2, Align) for you to decide on.
|
||||
- **`expectations.yaml`** gathers what each Feature now keeps in different places (checks, value bars, `limits:`),
|
||||
and every use case must have one before it runs. Expectations that are only prose today (F02's README) are turned
|
||||
into checks by Chartmaker and reviewed by you.
|
||||
|
||||
**A voyage chains one engagement's use cases across workflows.** F01's `azure-crucible-kab-dev-cmdb`, F02's
|
||||
`azure-crucible-kab-dev-assessment` and F03's `azure-crucible-kab-dev-strategy` all describe `ENG-AZURE-KAB-DEV-002`,
|
||||
but the links are execution ids copied by hand (F02's README asks for `discovery_execution_id` and the package
|
||||
artifact id; F03 has `assessment_execution_id: exec_374d5d577077`). `voyages/<engagement>.yaml` names the use case for each stage and which upstream output fills each
|
||||
downstream input, for example `assessment_execution_id: {from: alz-f02/azure-crucible-kab-dev-assessment, run:
|
||||
latest_signed_off}`. When a run starts, Shakedown resolves the reference into the uploaded copy under `generated:`, as
|
||||
`kab_run_workflow.py` already does for `run_id`. The use case file is never edited. A voyage is also the live
|
||||
alignment test: the downstream intake has to accept what the upstream run really produced.
|
||||
|
||||
**A variation changes one business input and states how the output must respond.** This is the most direct test of
|
||||
business value: an output that doesn't move when the business situation changes is not reasoning about the business.
|
||||
Each entry in `variations.yaml` has a `change` (a path in `run-context.yaml` and its new value) and one or more
|
||||
`expect` relations:
|
||||
|
||||
| Relation | Meaning | Judged by |
|
||||
|---|---|---|
|
||||
| `direction` | A named output metric goes up or down against the base run | Code, from the metric's path |
|
||||
| `bound` | A limit from the input is respected (no plan above `budget_ceiling`) | Code |
|
||||
| `unchanged` | Something the change shouldn't touch stays the same (the disposition set when only the timeline moves) | Code |
|
||||
| `rationale` | The output's reasoning cites the changed input | Claude Sonnet, with a quote |
|
||||
|
||||
Seed variations for F03's `azure-crucible-kab-dev-strategy`, drafted from its run context:
|
||||
|
||||
| Change | Expected |
|
||||
|---|---|
|
||||
| `priorities.cost` 0.3 → 0.7 (risk 0.4 → 0.1) | Total cost of the recommended strategy goes down; the rationale cites cost priority |
|
||||
| `constraints.budget_ceiling.amount` 40000 → 20000 | No recommended plan exceeds 20,000 USD, or the output says none can, and why |
|
||||
| `constraints.risk_appetite` medium → low | Refactor share goes down; risk mitigations go up |
|
||||
| `planning_assumptions.team_capacity` 2 → 1 | The timeline gets longer; the disposition set is unchanged |
|
||||
| `constraints.exit_date` 2027-06-30 → 2027-03-31 | Scope is cut or the output says the date is out of reach; the timeline never claims to fit when it doesn't |
|
||||
| `constraints.data_residency` drop `eastus2` | No target in `eastus2` |
|
||||
|
||||
**Variations stay cheap.** Only the stage whose input changed is run again. The upstream ids stay pinned to the base
|
||||
run, so an F03 variation costs one F03 run, not a voyage. Each variation runs N times (default 3), and a relation
|
||||
passes only if it holds beyond the base use case's own run-to-run spread (§2, Stability). Purser keeps a token budget
|
||||
for each batch, and `start_replay_batch` asks for your approval.
|
||||
|
||||
## 3b. The existing use cases
|
||||
|
||||
Shakedown starts from the use cases already written. The committed ones stay in cp1-agents and are read in place
|
||||
(§3), and F03's moves to this repository. Paths are relative to `cp1-agents/individual-agents/`.
|
||||
|
||||
**ALZ-F01 Discovery:** `ALZ-F01: Discovery Workflow Agent/workflows/cases/`
|
||||
|
||||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||||
|---|---|---|---|---|
|
||||
| `azure-crucible-kab-dev-cmdb` | Live Azure, resource group `crucible-kab-dev`, with an uploaded ServiceNow CMDB export and relationships that plant drift, ghosts, a dangling relationship and a prompt injection | `ENG-AZURE-KAB-DEV-002` | 48 checks, `value:` bar, `limits:` | First stage of the `ENG-AZURE-KAB-DEV-002` voyage; variations on the CMDB (none uploaded, a stale export, a conflicting owner) |
|
||||
| `azure-crucible-kab-dev` | The same group, no CMDB or ITSM: every gap declared, a complete package | `ENG-AZURE-KAB-DEV-001` | 15 checks, `value:` bar, `limits:` | The base its `-cmdb` sibling is a variation of: `unchanged` relations for the estate facts, `direction` for drift findings |
|
||||
| `meridian-baseline` | Meridian Retail Group sample estate (`provider: sample`): six scope entries, one unscannable HSM, a planted prompt injection | `ENG-MERIDIAN-001` | 45 checks | First stage of the Meridian voyage; rubric scores (no `value:` bar yet) |
|
||||
| `meridian-without-itsm` | Meridian with ITSM turned off on the run form | `ENG-MERIDIAN-001` | 6 checks | Recast as a variation of `meridian-baseline` (`include_itsm` yes → no) with `unchanged` and `direction` relations |
|
||||
| `snapshot-missing-sources` | A real estate from a snapshot, scanning only: no CMDB, ITSM, flow or schema source | any | 8 checks | Adversarial base for "declare the gap, don't invent it" |
|
||||
| `azure-live` | Whole subscription `sub-kyn-RITM0250380-653-dev`, about 644 resources | `ENG-AZURE-LIVE-001` | None | Expectations drafted by Chartmaker (scale, cost and coverage `limits:` first) |
|
||||
| `gcp-live` | Whole live GCP project | `ENG-GCP-LIVE-001` | None | Expectations drafted by Chartmaker; the same expectations as `azure-live` where the providers should agree (hyperscaler servers stay separate, the workflow is the same) |
|
||||
|
||||
**ALZ-F02 Assessment & Disposition:** `ALZ-F02: Assessment & Disposition Workflow Agent/workflows/cases/`
|
||||
|
||||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||||
|---|---|---|---|---|
|
||||
| `azure-crucible-kab-dev-assessment` | The Discovery package of an `azure-crucible-kab-dev-cmdb` run, and an illustrative Azure rate card | `ENG-AZURE-KAB-DEV-002` | Eight expectations as prose in the README (intake, applications, preconditions, in-cloud estate, risk, TCO, sign-off, evidence), written before the first run | Prose turned into `expectations.yaml` (reviewed by you); a `run-context.yaml` gathering its `--input` flags; second stage of the voyage; seed experiments EXP-001 to EXP-005 (§8) run on it; rate-card variations (a compute rate present or absent) |
|
||||
| `meridian-assessment` | A Meridian Discovery run (`RUN-008`, critic `pass`) and the sample rate card | `ENG-MERIDIAN-001` | Six expectations as prose in `cases/README.md` (no cloud, risk, partial score, dependents, sign-off, evidence); checks still `todo` in F02 PLAN step 10 | Expectations written and frozen before its first scored run; second stage of the Meridian voyage |
|
||||
|
||||
**ALZ-F03 Strategy Synthesis:** `ALZ-F03: Strategy Synthesis Workflow Agent/workflows/cases/`
|
||||
|
||||
| Use case | Scenario | Engagement | Expectations today | What Shakedown adds |
|
||||
|---|---|---|---|---|
|
||||
| `azure-crucible-kab-dev-strategy` | Strategy for the KAB Platform on the signed-off Assessment `exec_374d5d577077`: security and cost drivers, 40,000 USD ceiling, start 2027-01-11 and exit 2027-06-30, residency in `eastus` and `eastus2`, priorities and planning assumptions | `ENG-AZURE-KAB-DEV-002` | None | Moves to `cp1-shakedown/use-cases/alz-f03-strategy/`; expectations; third stage of the voyage; the six seed variations (§3a) |
|
||||
|
||||
**Each agent's `eval/cases.yaml`** (F01: 12 cases over 11 agents; F02: 7 over 9; F03: 6 over 6) tests one agent
|
||||
alone. It stays with its agent. Shakedown uses these cases as replay inputs for experiments on a single step, and
|
||||
reports agents with no eval case.
|
||||
|
||||
**Voyages to define first:**
|
||||
- `ENG-AZURE-KAB-DEV-002`: F01 `azure-crucible-kab-dev-cmdb` → F02 `azure-crucible-kab-dev-assessment` → F03
|
||||
`azure-crucible-kab-dev-strategy`. This is the only engagement with all three stages today.
|
||||
- `ENG-MERIDIAN-001`: F01 `meridian-baseline` → F02 `meridian-assessment`. There is no F03 stage yet; Chartmaker
|
||||
drafts one.
|
||||
|
||||
**Gaps across the set** (each one a first job for Chartmaker, reviewed by you):
|
||||
- Expectations are missing for `azure-live`, `gcp-live` and F03, and are only prose for both F02 use cases.
|
||||
- F02 has no context file. Its scenario is spread across `--input` flags in a README, which goes against the
|
||||
convention that every run starts from a YAML context file. This is an alignment finding for cp1-agents to decide on.
|
||||
Until then, Shakedown's F02 adapter turns `run-context.yaml` into those inputs.
|
||||
- No use case varies the business situation yet (objectives, budget, risk appetite, regulation). The F01 and F02
|
||||
variants vary sources and data.
|
||||
- Only `ENG-AZURE-KAB-DEV-002` has a CMDB upload, and no use case has ITSM data from a file.
|
||||
|
||||
## 4. Architecture on the cluster
|
||||
|
||||
```
|
||||
@@ -168,7 +281,7 @@ them as follows:
|
||||
| **Lookout** (run operator) | Starts runs from context files and replays, watches them, collects outputs and usage. Stops at every HITL pause and tells you | KAB, replay, cluster read | Ledger entries |
|
||||
| **Surveyor** (judge) | Scores the checks and invariants (in code) and the business-value rubric. Quotes evidence for every score. Never sees the agent's own critic verdict before scoring | case runner, rubric, ledger read | `scores.json` |
|
||||
| **Shipwright** (fixer) | Turns a confirmed finding into a proposed diff to a prompt, schema or hook, plus a draft ADR when a requirement changes | repo read, gitea PR | Pull request to cp1-agents, never merged by Bosun |
|
||||
| **Chartmaker** (case author) | Drafts cases and checks from ADO acceptance criteria and worked examples, and adversarial variants of them | ADO read, repo read, gitea PR | Pull request to Shakedown |
|
||||
| **Chartmaker** (use-case author) | Drafts use cases, expectations, voyages and variations from ADO acceptance criteria and worked examples, plus adversarial versions | ADO read, repo read, gitea PR | Pull request to Shakedown |
|
||||
| **Purser** (cost) | Tokens, time and cost per step; flags steps over `limits:` and the cheapest variant that keeps the score | ledger read | Cost section of the scores |
|
||||
|
||||
Every sub-agent runs on Claude Sonnet 5.5. Surveyor's prompt and rubric are versioned on their own, so a change to
|
||||
@@ -221,12 +334,12 @@ repository exists.
|
||||
|
||||
| # | Phase | Delivers | Verified by |
|
||||
|---|---|---|---|
|
||||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||||
| 0 | **Foundations** | Repository scaffold in `cp1-shakedown`; Keycloak client and ingress hostname registered; ingress with sign-in; the Azure Claude call path proven from LangChain; `shakedown` namespace, ServiceAccount and read-only RBAC; pinned cp1-agents clone; ledger schema; F03's `azure-crucible-kab-dev-strategy` brought in as the first use case (agreed with whoever is running F03); first deploy | Sign in once and follow a live run for 30 minutes with no manual token refresh; a Sonnet call from the pod succeeds; the ServiceAccount can read agent logs and cannot read Secrets |
|
||||
| 1 | **Ledger and rating** | Backfill of F02 runs 004 to 015 and F01's live runs; generic case runner over the existing cp1-agents cases; invariants in code (figures, digests, ids, limits); runs list and run detail in the UI | The `azure-crucible-kab-dev-assessment` checks score runs 014 and 015; a report figure changed in a copy of a run is caught |
|
||||
| 2 | **Bosun v1, read-only** | Deep agent with Lookout (reads only), Surveyor and Purser; "explain this run" and "compare these runs" | **The acceptance test in §1** passes on runs 010 to 015 |
|
||||
| 3 | **Experiments** | Stability runs; step replay over A2A; variant overlays rendered for you to apply; side-by-side diffs of fields and scores; `experiments/` records by pull request | 5 replays of F02 `s5` on run 015's input show the P2 spread; a variant with P1 to P3 in code (improvement 3) shows zero spread |
|
||||
| 4 | **Alignment and value** | Static contract checks F01 → F02 → F03; live hand-off test; acceptance-criteria traceability from ADO; F02 rubric and judge calibration | Every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs |
|
||||
| 5 | **Use-case factory** | Chartmaker and Shipwright; drafting of adversarial cases; a patch pull request and ADR draft for each confirmed finding | F03's cases and checks are merged in Shakedown **before** F03's first live run; F03 is onboarded with no new Shakedown code |
|
||||
| 4 | **Alignment, value and variations** | Static contract checks F01 → F02 → F03; voyages with ids resolved from upstream runs; acceptance-criteria traceability from ADO; rubrics and judge calibration; variations with `direction`, `bound`, `unchanged` and `rationale` relations | The `ENG-AZURE-KAB-DEV-002` voyage runs F02 → F03 with no id typed by hand; every F02 acceptance criterion maps to a check or is listed as uncovered; Surveyor agrees with you on 8 of 10 outputs; the six F03 seed variations (§3a) each pass, or Shakedown reports which relation failed, with the evidence |
|
||||
| 5 | **Use-case factory** | Chartmaker and Shipwright; use cases, expectations and variations drafted from ADO stories; adversarial use cases; a patch pull request and ADR draft for each confirmed finding | F04's use case, expectations and variations are merged in Shakedown **before** F04's first live run; F04 is onboarded with no new Shakedown code |
|
||||
| 6 | **Scale out** | F04 and F05, the sub-agent pattern as a separate suite, then CP4 workstream workflows | A new workflow is onboarded by adding config only: its export path, cases and rubric |
|
||||
|
||||
Phases 0 to 2 are the minimum useful product: they remove the token toil and replace the hand checks. Phase 3 is where
|
||||
@@ -245,7 +358,9 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
4. **EXP-004 Security posture cost** (improvement 5): `LITELLM_MAX_TOKENS` tiers, and `gpt-4.1` against
|
||||
`gpt-5.6-sol`; the metrics are tokens and the value score. A model change remains your call.
|
||||
5. **EXP-005 Report figures** (improvement 4): turns the hand check into a permanent invariant.
|
||||
6. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the
|
||||
6. **EXP-006 F03 business sensitivity:** the six seed variations of `azure-crucible-kab-dev-strategy` (§3a); the
|
||||
metric is the share of relations that hold beyond run-to-run spread. This is the first variation experiment.
|
||||
7. **Temperature, as context for all of them:** every agent runs at `LITELLM_TEMPERATURE: "1"`, probably because the
|
||||
gpt-5.x models accept only that value. Treat run-to-run variation as a given, and measure it, rather than assuming
|
||||
it can be tuned away.
|
||||
|
||||
@@ -259,7 +374,8 @@ These come straight from F02's "Improvements to make" list. Shakedown measures e
|
||||
| Bosun's model and the judge | Claude Sonnet 5.5 for both |
|
||||
| Model route | The Kyndryl Azure resource, called directly; not the cluster's LiteLLM, not the Anthropic API |
|
||||
| Repository | `agents/cp1-shakedown` on gitea, for use cases and experiments, apart from the agents and workflows |
|
||||
| Existing cases | Stay in cp1-agents and are read by path; only new use cases and all experiments go to Shakedown |
|
||||
| Use cases | Scenario folders like F03's `azure-crucible-kab-dev-strategy`. New ones, starting with that one, live in cp1-shakedown; committed F01 and F02 cases stay in cp1-agents and are read by path |
|
||||
| Variations | A core feature: derived scenarios with expected relations, judged in code first and by Sonnet for rationale |
|
||||
| Hosting | The crucible cluster |
|
||||
| Shadow agents | The `shakedown` namespace |
|
||||
| UI access | Ingress with Keycloak sign-in from Phase 0 (no port-forward stage) |
|
||||
|
||||
Reference in New Issue
Block a user