ALZ-F02 Assessment: individual agents, intake, rate card input, critic fixes
- Six declarative agents generated from the v1.1 stories (scaffold_from_stories.py, ADR 0001),
and a 9-step workflow export (build_workflow.py).
- Code-only intake (ADR 0008) verifies the one Discovery package in code before any agent.
- The rate card is the run input rate_card_document; no upload step (ADR 0009).
- Critic 0.1.6 after live runs on crucible-kab-dev:
- K-12 model independence checked at build time against each agent's deployed model
(ADR 0010); the critic reports it from its two inputs;
- receives in_cloud_estate for K-10; an unverified K-10 never fails the run alone;
- K-08 allows the scoring note to list findings as advisory, as ALZ-ASM-04 must;
- carries ALZ-ASM-03's and ALZ-ASM-04's rule sections, copied by the scaffold, so it can
recalculate risk, confidence and TCO.
- The scaffold refuses to regenerate unless every story file is present.
- Case azure-crucible-kab-dev-assessment with an illustrative rate card.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,23 @@
|
||||
# 0001. Six declarative agents; scoring as prompt rules, recalculated by the critic
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Product owner (2026-11-02: individual pattern first; no new MCP servers) |
|
||||
| Requirements changed | ASM-02 to 06: the source tool operations `score_disposition`, `model_tco`, `analyze_licensing`, `write_disposition`, `risk_model`, `profile`, `assess`, `evaluate_controls`, `rank_findings` |
|
||||
|
||||
## Context
|
||||
|
||||
The source definitions name scoring and writing operations as tools. No server provides them, and the product owner asked to build on the existing MCP servers rather than add new ones (Feature 3214442, decision of 2026-11-02).
|
||||
|
||||
## Decision
|
||||
|
||||
- **One declarative agent per ALZ-ASM story,** chained by the KAB workflow, with no orchestrator. This mirrors the ALZ-F01 individual pattern.
|
||||
- **The calculations are not tools.** Profiling, risk scoring, the 7R rule set, confidence, TCO and licensing are fixed rules in each agent's `prompt.md`, written in its output schema. ALZ-ASM-06 recalculates every number from the same inputs (K-04, K-05 and K-06).
|
||||
- **No write tools.** The workflow stores each step's reply.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The arithmetic rests on the model following the rules,** with the critic as the check. That critic is a different model (ADR 0005), and the human approval at its step is the final gate.
|
||||
- **The sub-agent pattern can do the same arithmetic in code.** Comparing the two patterns on the same cases is the way to measure the difference.
|
||||
@@ -0,0 +1,26 @@
|
||||
# 0002. Discovery's outputs reach the agents as parsed text; run id and approval are run inputs
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Built as a default; awaiting product owner confirmation |
|
||||
| Requirements changed | Feature AC4 and ASM-01 US-001 (the approval and run-status preconditions) |
|
||||
|
||||
## Context
|
||||
|
||||
A workflow can't run or chain another workflow (`docs/platform-issues/workflow-composition/`). The Artifacts step's only output is `artifacts`: per-file records with base64 content. Review decisions are written to the activity log, not onto artifacts.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Front steps.** `a1` (Artifacts) selects the Discovery run's artifacts by exact id. `p1` (Parse Assessment Files) turns `a1.artifacts` into `parsed_content` and `file_metadata`. Every agent that reads Discovery takes both, as `discovery_documents` and `discovery_files`.
|
||||
- **No model copies Discovery's records** between steps.
|
||||
- **Run inputs:** `artifact_ids` (unless the export fixes them), `discovery_execution_id`, and `discovery_approval_ref`, the decision record approving the Discovery run.
|
||||
- **`discovery_completed`** is read as "the critic and report documents are both present". A failed run registers only the outputs it reached.
|
||||
- **References** to Discovery records are written `discovery:{execution_id}/{agent_id}/{record_key}`, because Discovery's records carry no record-level reference of their own.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The approval can't be verified by the agents;** they trust the reference they're given. The fix is platform-side: record the run outcome on its artifacts (workflow-composition question 3).
|
||||
- **An output too large to inline** (over 8 MiB live) arrives with no content and is skipped by the parser. The agents report it as a missing document.
|
||||
- **The person starting the run has to find the right artifact ids.**
|
||||
@@ -0,0 +1,22 @@
|
||||
# 0003. The rate card is an uploaded JSON file
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Superseded by [0009](0009.md) |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Built as a default; the source (rate card uploaded with the run) was decided by the product owner on 2026-11-02 |
|
||||
| Requirements changed | ASM-01 CAP-01 and ASM-04 CAP-03 / CAP-04 (form of the rate card) |
|
||||
|
||||
## Context
|
||||
|
||||
The Feature decided that target rates and licensing terms come from a rate card uploaded with the run, but not its format.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Upload:** the rate card comes through the run form's `file` field on `u1`, and the planner, the disposition agent and the critic take `u1.parsed_content` as `rate_card_document`.
|
||||
- **Format:** JSON with `rate_card_id`, `currency` and `items`. Each item has `rate_id`, `category`, `description`, `annual_rate`, `equivalent_for` (services, products or asset classes) and `licence_terms`. A sample is in `workflows/cases/meridian-assessment/rate-card.json`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A missing rate stops nothing,** but it's named in `missing_inputs` and the TCO stays partial.
|
||||
- **Another format (a spreadsheet, say) arrives as parsed text** and is harder for the agents to read reliably. Keep to JSON until a template exists.
|
||||
@@ -0,0 +1,22 @@
|
||||
# 0004. `disposition_signoff` is the critic step's approval
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Built as a default; awaiting product owner confirmation |
|
||||
| Requirements changed | Feature human gate `disposition_signoff`; ASM-06 CAP-08 (per-item routing) |
|
||||
|
||||
## Context
|
||||
|
||||
KAB's human review is per workflow step, not per item.
|
||||
|
||||
## Decision
|
||||
|
||||
- **The critic's step (`s6`) stops for the Engagement Lead's approval.** The reviewer reads the verdict and every disposition in `signoff_queue`: confidence below 0.80, and always repurchase and relocate.
|
||||
- **Approving the step signs off the queue,** and a `fail` verdict is never approved.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **There is no separate decision record per disposition.** One step approval covers the queue.
|
||||
- **Rerunning the step re-validates; it doesn't rescore.** A per-item sign-off needs a platform feature or a later step.
|
||||
@@ -0,0 +1,22 @@
|
||||
# 0005. Models: executors on `gpt-5.6-sol`, the critic on `gpt-5.4`
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Built as a default; awaiting product owner confirmation |
|
||||
| Requirements changed | None. This meets PR-07 (the critic on a different model) |
|
||||
|
||||
## Context
|
||||
|
||||
F01's individual critic had to share a model with its executors (F01 ADR 0008). The cluster's LiteLLM serves `gpt-5.6-sol`, `gpt-5.4` and `gpt-4o`.
|
||||
|
||||
## Decision
|
||||
|
||||
- **The five executors use `gpt-5.6-sol`. The critic uses `gpt-5.4`.**
|
||||
- **The workflow passes both names to the critic,** as static inputs, so K-12 can check them. The build checks them
|
||||
against the deployed models and settles K-12 ([ADR 0010](0010.md)).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **`gpt-5.4` is untested on these checks.** Watch its recalculation accuracy in the eval and workflow cases, and change it if it misses seeded defects.
|
||||
@@ -0,0 +1,22 @@
|
||||
# 0006. The critic can't check tool calls in this pattern (K-10)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Built as a default; awaiting product owner confirmation |
|
||||
| Requirements changed | ASM-06 CAP-06 and K-10 (every MCP call on the caller's allowlist) |
|
||||
|
||||
## Context
|
||||
|
||||
No MCP call log reaches a declarative agent through the workflow, and the existing servers have no call-log tool.
|
||||
|
||||
## Decision
|
||||
|
||||
- **K-10 is reported `unverified`, and doesn't fail the verdict on its own.**
|
||||
- **What K-10 still checks:** that ALZ-ASM-05 reported `not_applicable`, with no findings, when the plan says the estate is on-premises.
|
||||
- **Allowlists are enforced by the platform** (PR-10).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A tool call outside the plan's scope isn't caught by the critic** in this pattern. Review the MCP gateway's logs in the workflow case.
|
||||
@@ -0,0 +1,22 @@
|
||||
# 0007. ALZ-ASM-02 and ALZ-ASM-05 need all three discovery servers registered
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | 2026-11-02 |
|
||||
| Decided by | Consequence of the Feature's MCP decision; recorded for deployment |
|
||||
| Requirements changed | None (deployment prerequisite) |
|
||||
|
||||
## Context
|
||||
|
||||
A declared MCP server that isn't offered stops an agent's tools loading. ALZ-ASM-02 declares the three discovery servers. ALZ-ASM-05 declares the same three, including their `<provider>_search_well_architected` tools. (Until 2026-11-04 it also declared `cloud-knowledge`, which is now retired.)
|
||||
|
||||
## Decision
|
||||
|
||||
- **Both agents declare every server for every provider,** so one workflow works for a customer on any cloud. The cloud servers stay standalone and are never merged.
|
||||
- **The three servers must be deployed and registered in the CP1 tenant** before either agent can start, even for an on-premises estate like Meridian.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **This blocks the first workflow run.** The discovery servers aren't deployed, and customer credentials need the Vault Service.
|
||||
- **For an on-premises sample run,** the servers only need to be registered and healthy, without customer credentials. Confirm that on the platform before relying on it.
|
||||
@@ -0,0 +1,45 @@
|
||||
# 0008. Assessment reads one Discovery package, checked in code by an intake step before any agent
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "lets move forward with each of these", on the plan step "F02 verifies the package's manifest in code (F02's own ADR), and selects one artifact" (F01 `docs/discovery-value-plan.md`, phase 1; F01 ADR 0020) |
|
||||
| Requirements changed | ALZ-ASM-01 US-001 (preconditions): "Discovery completed" and "critic verdict pass" are checked in code from the package's manifest before the planner runs. ADR 0002: one artifact, the package, replaces one artifact per Discovery step. ADR 0001: one code-only step, outside the six generated agents |
|
||||
|
||||
## Context
|
||||
|
||||
- Discovery now writes one package per reviewed run (F01 ADR 0020): every step's output as a named part, with a
|
||||
manifest holding a `sha256` per part, `complete`, the critic's verdict and the Discovery execution id.
|
||||
- Until now, the person starting Assessment typed in one artifact id per Discovery step, and the planner, a model,
|
||||
judged from the text whether Discovery had completed and passed (ADR 0002). Nothing checked that the documents were
|
||||
unchanged, complete, or from the run named.
|
||||
- The six Assessment agents are declarative and generated from the story files (ADR 0001); a hook added to one would
|
||||
be lost when they're regenerated.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **A code-only intake step, `s0`, `cp1-assessment-intake-agent`,** runs after the parse step and before the
|
||||
planner. It isn't generated from the stories, and it has no model and no tools.
|
||||
2. **It checks, in code** (`app/intake.py`): exactly one package among the selected artifacts, wherever KAB's wrapping
|
||||
puts it; a package version it reads (`1.0`); every part the manifest lists is present and matches its `sha256`;
|
||||
`complete`; the critic's verdict `pass`; the manifest's execution id equals the run's `discovery_execution_id`, and
|
||||
its engagement equals `engagement_id`.
|
||||
3. **It fails closed.** Any failed check raises, so the step fails with the reasons and no Assessment agent runs.
|
||||
4. **The planner waits for it** (`s1` depends on `s0`). On success it returns `verified`, the parts it checked, the
|
||||
source gaps, the agent versions, Discovery's applications (F01 ADR 0021) and the scorecard.
|
||||
5. **The person starting a run selects one artifact:** the Discovery run's package step output.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **"Discovery completed" and "critic passed" are facts checked in code,** not the planner's reading of the text.
|
||||
The planner still sees the documents and still writes its preconditions; it can't start on a package the intake
|
||||
refused.
|
||||
- **The approval is still implied, not explicit.** The package exists only for a run continued past Discovery's
|
||||
review point, but the decision record itself isn't visible to Assessment (ADR 0002's open platform question).
|
||||
- **A run selecting the old per-step artifacts is refused,** because they hold no package. Assessment hasn't been
|
||||
deployed, so nothing in use breaks.
|
||||
- **The six agents' inputs are unchanged.** They still read `discovery_documents`, now one package rather than ten
|
||||
documents. Giving them typed fields from the intake (and shrinking each prompt) is a later step.
|
||||
- **Tested** on the live package of Discovery run `exec_14956d524a8b`, and on changed, removed, failed, foreign,
|
||||
duplicate, missing and newer-version packages.
|
||||
@@ -0,0 +1,36 @@
|
||||
# 0009. The rate card is a run input, its JSON as text; Assessment has no upload step
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "Run input text" (the rate card is a run input, its JSON pasted into the run form or passed by the script) |
|
||||
| Requirements changed | Supersedes [ADR 0003](0003.md) (the rate card as an uploaded JSON file). ALZ-ASM-01, 04 and 06 read the rate card from the run input `rate_card_document` |
|
||||
|
||||
## Context
|
||||
|
||||
- The first Assessment run, `exec_20d55cfe6fd7`, stopped at the intake ([ADR 0008](0008.md)): "0 Discovery packages
|
||||
among the selected artifacts". The Discovery package was selected correctly.
|
||||
- **The cause is how KAB passes run inputs.** Every run input reaches every step. The rate card was uploaded as the
|
||||
run's `file`, so it also reached `p1`, the parse step wired to the Artifacts step's `files`. The parse activity
|
||||
reads a `file` input before `files`, so `p1` parsed the rate card again and never saw the package. Its output was
|
||||
the rate card's text.
|
||||
- So while the rate card is uploaded as `file`, no Assessment agent can read Discovery. Discovery is unaffected: it
|
||||
has one parse step, and its uploads are meant for it.
|
||||
|
||||
## Decision
|
||||
|
||||
- **No upload step.** The workflow starts with `a1` (Artifacts) and `p1` (parse), so `file` is never a run input.
|
||||
- **The rate card is the run input `rate_card_document`:** its JSON as text, in the same format as before (Feature
|
||||
README). The planner, the disposition agent and the critic take it under the same name, so no agent's behaviour
|
||||
changes; only the descriptions say where it comes from (planner, disposition and critic 0.1.1).
|
||||
- `scripts/kab_run_workflow.py` reads an input from a file with `--input rate_card_document=@<path>`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The parse step only ever sees the Discovery package,** and the intake's check means something.
|
||||
- **In the KAB run form the rate card is pasted,** not uploaded. It's small (a few kilobytes), so a text field is
|
||||
enough. A large rate card would be better as a project artifact selected beside the package, which the intake would
|
||||
then have to tell apart.
|
||||
- **Any future step that parses an upload in a workflow that also parses artifacts hits the same rule.** Keep one
|
||||
file-parsing source per workflow, or none.
|
||||
@@ -0,0 +1,36 @@
|
||||
# 0010. Model independence (K-12) is checked when the workflow is built, not judged by the critic
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "go with the recommendation" (check K-12 at build time; the critic reports it) |
|
||||
| Requirements changed | ALZ-ASM-06 K-12: the check runs in `scripts/build_workflow.py`; the critic reports its result. Refines [ADR 0005](0005.md) |
|
||||
|
||||
## Context
|
||||
|
||||
- K-12 asks whether the critic runs on a different model from the agents it checks (PR-07). The workflow passes the
|
||||
critic both names as static inputs (ADR 0005), so the answer is fixed before any run starts.
|
||||
- The critic judged it anyway, and failed it on Discovery's models in two runs out of three:
|
||||
- `exec_4be193061c41`: a Discovery critic note that Discovery's network agent runs on `gpt-5.4`.
|
||||
- `exec_0b0afb70b8b9`: a Discovery critic note that Discovery's critic was validated on `gpt-5.4`. Every other
|
||||
check passed; the prompt (critic 0.1.2) already said that models named in Discovery's documents don't count.
|
||||
- Discovery is a separate run with its own critic. Its models have no bearing on this run's independence.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **`build_workflow.py` checks K-12** before writing the export. It reads each agent's deployed model
|
||||
(`LITELLM_MODEL` in its `kubernetes/configmap.yaml`) and refuses to build unless:
|
||||
- the critic is deployed on the static `critic_model`;
|
||||
- every other step's agent is deployed on a model in `executor_models`;
|
||||
- `critic_model` is not in `executor_models`.
|
||||
2. **The critic reports K-12** as `pass` when `critic_model` is not in `executor_models`, and reads nothing else for
|
||||
it. It has no code: the agent stays declarative (ADR 0001).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A workflow with a critic on an executor's model can't be built,** so it never reaches KAB.
|
||||
- **The names the critic is given are checked against the deployments,** not just typed in. A model changed in a
|
||||
configmap without a rebuild of the workflow isn't caught: rebuild and update the workflow after any model change.
|
||||
- **The critic can still misreport K-12,** being a model. If it does, the report contradicts its own inputs, which
|
||||
is visible in the check's details.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Architecture decision records: individual agents, ALZ-F02 Assessment & Disposition
|
||||
|
||||
Each record documents a change to, or an interpretation of, the original requirements for ALZ-F02: ADO Feature
|
||||
3214442, its stories 3214459–3214464, and the v1.1 story files (`drafts/cp1/agentic-landing-zone/stories/F02-assessment/`
|
||||
in the engineering hub workspace). These records apply to the **individual-agents pattern** in this Feature folder.
|
||||
The sub-agent pattern keeps its own records. Which pattern is adopted has not been decided.
|
||||
|
||||
Each agent's `docs/adr/README.md` lists the records that apply to its own story.
|
||||
|
||||
**Status values:**
|
||||
- **Accepted:** decided by the product owner.
|
||||
- **Proposed:** built as a default and awaiting confirmation.
|
||||
|
||||
| ADR | Decision | Status | Stories |
|
||||
|---|---|---|---|
|
||||
| [0001](0001.md) | Six declarative agents; scoring as prompt rules, recalculated by the critic | Accepted | ASM-02 to 06 |
|
||||
| [0002](0002.md) | Discovery's outputs reach the agents as parsed text; run id and approval are run inputs | Proposed | All |
|
||||
| [0003](0003.md) | The rate card is an uploaded JSON file | Superseded by 0009 | ASM-01, 04, 06 |
|
||||
| [0004](0004.md) | `disposition_signoff` is the critic step's approval | Proposed | ASM-06 |
|
||||
| [0005](0005.md) | Models: executors on `gpt-5.6-sol`, the critic on `gpt-5.4` | Proposed | ASM-06 |
|
||||
| [0006](0006.md) | The critic can't check tool calls in this pattern (K-10) | Proposed | ASM-06 |
|
||||
| [0007](0007.md) | ALZ-ASM-02 and ALZ-ASM-05 need all four MCP servers registered | Proposed | ASM-02, 05 |
|
||||
| [0008](0008.md) | Assessment reads one Discovery package, checked in code by an intake step before any agent | Accepted | ASM-01, intake (new) |
|
||||
| [0009](0009.md) | The rate card is a run input, its JSON as text; Assessment has no upload step | Accepted | ASM-01, 04, 06 |
|
||||
| [0010](0010.md) | Model independence (K-12) is checked when the workflow is built, not judged by the critic | Accepted | ASM-06 |
|
||||
|
||||
New records copy [template.md](template.md) and take the next number.
|
||||
@@ -0,0 +1,14 @@
|
||||
# NNNN. Title
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Proposed |
|
||||
| Date | YYYY-MM-DD |
|
||||
| Decided by | |
|
||||
| Requirements changed | Story / ADO item and the section that no longer holds |
|
||||
|
||||
## Context
|
||||
|
||||
## Decision
|
||||
|
||||
## Consequences
|
||||
Reference in New Issue
Block a user