ALZ-F02 Assessment: individual agents, intake, rate card input, critic fixes

- Six declarative agents generated from the v1.1 stories (scaffold_from_stories.py, ADR 0001),
  and a 9-step workflow export (build_workflow.py).
- Code-only intake (ADR 0008) verifies the one Discovery package in code before any agent.
- The rate card is the run input rate_card_document; no upload step (ADR 0009).
- Critic 0.1.6 after live runs on crucible-kab-dev:
  - K-12 model independence checked at build time against each agent's deployed model
    (ADR 0010); the critic reports it from its two inputs;
  - receives in_cloud_estate for K-10; an unverified K-10 never fails the run alone;
  - K-08 allows the scoring note to list findings as advisory, as ALZ-ASM-04 must;
  - carries ALZ-ASM-03's and ALZ-ASM-04's rule sections, copied by the scaffold, so it can
    recalculate risk, confidence and TCO.
- The scaffold refuses to regenerate unless every story file is present.
- Case azure-crucible-kab-dev-assessment with an illustrative rate card.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-05 22:13:25 +00:00
parent 9e503885cc
commit e22d037ab0
153 changed files with 13004 additions and 0 deletions

View File

@@ -0,0 +1,35 @@
# Context: individual agents, ALZ-F02 Assessment & Disposition
Facts for this Feature's individual-agents pattern, for coding agents. Repository-wide facts are in `../../.agents/CONTEXT.md`. Per-agent context is in `<agent>/.agents/CONTEXT.md`.
## Pattern
- **Six declarative agents, one per ALZ-ASM story,** plus the code-only intake (`s0`, ADR 0008). Only ALZ-ASM-02 and ALZ-ASM-05 declare MCP tools; the other four and the intake have none.
- **The workflow (`workflows/cp1-alz-assessment-individual.json`) is the orchestration.** Step order comes from `parallelGroup` and `executionLevel`, and data from `from_steps`. The step map is `STEPS` in `scripts/build_workflow.py`, and the scaffold reads it for each agent's context file.
- **Order:** `a1` the Discovery package artifact, `p1` parse, then `s0` intake (checks the Discovery package in code; a refusal fails the run), then `s1` planner, then `s2` profiler, `s3` risk and `s4` security posture in parallel, then `s5` disposition, then `s6` critic. `s6` has a human approval, which is `disposition_signoff` (ADR 0004).
- **Nothing persists between steps** except their replies, as in F01 (its ADR 0010).
## Sources of truth
| What | Where |
|---|---|
| Requirements | ADO Feature 3214442 and stories 3214459–3214464. The v1.1 story files are in the engineering hub workspace, at `drafts/cp1/agentic-landing-zone/stories/F02-assessment/` |
| Decisions on the board | Feature 3214442 "Decisions" table (2026-10-30 and 2026-11-02) |
| Changes to requirements | `docs/adr/` (this pattern only) |
| MCP servers | `../../mcp-servers/` (existing, shared, unchanged) |
| Discovery's output shapes | `../ALZ-F01: Discovery Workflow Agent/cp1-*/schemas/output.json` |
## Platform facts (verified in `modernization-platform` @ `c89aedcb9`)
- **`select_artifacts` has exactly one output, `artifacts`:** per-file records with base64 `content` while the selection fits the inline allowance (8 MiB live). Beyond that, a record carries only a `file_path` (migration 024; `artifacts_activities.py`).
- **`parse_assessment_files` accepts those records** under any input key (`files` here). It returns `parsed_content`, the texts joined by `\n\n---\n\n` with no names, and `file_metadata`, which has the names in the same order (`data_file_processing.py` `parse_files`).
- **A record with no inline content is skipped silently by the parser.** An oversized Discovery output therefore arrives as a missing document, and the agents report it.
- **Static inputs are applied after run inputs.** So the `a1` step leaves `artifact_ids` to the run (`from_workflow`) unless `--artifact-id` fixes them in the export.
- **Step outputs are registered as project artifacts only at the end of a run, and only when it has a `project_id`.** The artifact name is `{execution_id}_{step_id}`.
- **Review decisions go to the activity log** (`record_review_decision`), not onto artifacts. That's why the approval reference is a run input (ADR 0002).
- **Models served by the cluster's LiteLLM:** `gpt-5.6-sol`, `gpt-5.4`, `gpt-4o` (checked 2026-11-01).
- **No upload step** (ADR 0009). KAB hands every run input to every step, and `parse_assessment_files` reads a `file` input before the `files` wired to it, so a rate card uploaded as `file` was parsed by `p1` in place of the Discovery package (run `exec_20d55cfe6fd7`). The rate card is the run input `rate_card_document`, its JSON as text.
- **Run `exec_4be193061c41` (2026-11-05):** intake verified the package; every step ran. Two faults: (1) every Azure tool call from ALZ-ASM-02 and ALZ-ASM-05 got HTTP 401 from Azure, because the project's Azure Discovery connection holds an `az account get-access-token` token (about an hour); refresh it, or store a service principal, before a run. (2) The critic failed K-12 on a Discovery agent's model read from the package; critic 0.1.2 compares only `critic_model` with `executor_models` and gets `in_cloud_estate` for K-10.
- **Never pass the scaffold an option:** `scaffold_from_stories.py --help` once took `--help` as the stories folder and deleted the six agent folders (restored from the deployed images and the scaffold). It now refuses unless every story file is there.
- **K-12 at build time (ADR 0010):** `build_workflow.py` refuses a critic deployed on an executor's model, reading each agent's `LITELLM_MODEL`; critic 0.1.4 reports K-12 from its two inputs. Run `exec_0b0afb70b8b9` (first with live Azure evidence: posture 0.69, 10 findings) failed only on K-12.
- **The critic's recalculation rules (critic 0.1.6):** the scaffold copies ALZ-ASM-03's and ALZ-ASM-04's rule sections into the critic's prompt (`RECALC_RULES`), so K-04 to K-06 recalculate with the producers' own rules. Before 0.1.6 the critic had none (run `exec_1779863642d6`, K-04).

View File

@@ -0,0 +1,20 @@
# Implementation plan: individual agents, ALZ-F02 Assessment & Disposition
For coding agents. Update the status in the same change as the work.
Status values: `done`, `blocked (<on whom>)`, `todo`.
| # | Step | Verified by | Status |
|---|---|---|---|
| 1 | Generate the six agent folders from the v1.1 story files (`scripts/scaffold_from_stories.py`) | `check_agents.py` (cp4's rules): 6 folders, 0 problems; 25 declared MCP tools all exist on the existing servers | done (2026-11-02) |
| 2 | Eval cases per agent: Meridian sample Discovery outputs, `timeout: 330` | `python -m kyndryl_agent_runtime.eval --validate` | todo: locally, every schema is valid JSON Schema and every case input fits its input schema (2026-11-02); not yet run through the runtime's validator, or live |
| 3 | Workflow export: `u1`, `a1`, `p1` and six agent steps, `from_steps` hand-offs, critic approval | `build_workflow.py` asserts every field on both sides: 9 steps, 22 connections | done (2026-11-02) |
| 4 | Docs: README, ADRs 0001–0007, per-agent user guides and context | Review | done (2026-11-02) |
| 5 | Deploy profile entries for the six agents, then render | `render_k8s.py` + `kustomize build`: 3 objects each; critic on `gpt-5.4` | done (2026-11-02) |
| 6 | Deploy and register the three discovery servers in the CP1 tenant. ALZ-ASM-02 and ALZ-ASM-05 can't start without all three (ADR 0007). `cloud-knowledge` was retired on 2026-11-04; guidance is on the discovery servers | Gateway lists them healthy | partly done (2026-11-02): all four servers run in `mcp-servers` at their current versions (azure-discovery 0.1.3, aws-discovery 0.1.1, gcp-discovery 0.1.2, cloud-knowledge 0.1.1). Registration, and whether the gateway accepts the changed tool schemas, still to confirm with a KAB token. Customer credentials need the Vault Service |
| 7 | Build and push the six images; store each `MCP_AGENT_API_KEY`; `kubectl diff -k`, then `apply -k` | Pods ready; agent cards served | todo (needs the user's approval for each cluster change) |
| 8 | Validate and import the workflow (`scripts/kab_import_workflow.py`) | `validate-json` accepts it | todo |
| 9 | Run on Meridian: Discovery `RUN-008` (approved, verdict `pass`), its artifact ids, the approval reference, and `workflows/cases/meridian-assessment/rate-card.json` | Every step completes; the critic step waits for approval | todo |
| 10 | Workflow case `meridian-assessment` with checks written before the run, scored like F01's `evaluate_workflow_run.py` | Scored run | todo |
| 11 | Confirm the parsed Discovery text in a live run: the `---` separator, `file_metadata` names and order, and what happens when an output is too large to inline | `p1` output inspected | todo |
| 12 | Intake step `s0` (ADR 0008): `cp1-assessment-intake-agent` 0.1.0 checks the Discovery package in code (one package, sha256 per part, complete, critic pass, named run and engagement); the planner waits for it; one artifact selected | `tests/test_intake.py` 9 passed on the live package of `exec_14956d524a8b`; workflow export 10 steps | in progress (2026-11-05): all seven built, deployed and registered in the agent catalog; MCP keys for ASM-02 and ASM-05 stored (expire 2026-12-05); workflow imported as `wf_7b6cae08-327d-4ae4-875b-5ef7d044834f`; first run `exec_20d55cfe6fd7` stopped at the intake: the uploaded rate card reached `p1` (ADR 0009). Rate card moved to a run input; planner, disposition and critic 0.1.1 |