Files
cp1-agents/individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/.agents/CONTEXT.md
Jonathan Boniface 66c32745aa
Some checks failed
check agents / check (push) Failing after 14s
F02: Assessment report and package (ADR 0014, 0015); register agents in KAB
- Report (cp1-assessment-report-agent 0.2.0, step s7): a model writes the
  Assessment report from facts worked out in code (register, gaps, scorecard,
  rule traces, checks, and what each application is from the Discovery
  package); code sets the verdict and renders the pages. The
  disposition_signoff review point moves from the critic to the report.
- Package (cp1-assessment-package-agent 0.1.1, step s8): after sign-off, every
  step's output in one package, with a register, a gaps register, the
  sign-off record, lineage and a sha256 per part, in code. Page step s9.
- scripts/kab_register_agents.py: registers deployed agents in KAB's agent
  catalog (an in-place workflow update doesn't).
- Runs exec_1b1257380df7 (014) and exec_374d5d577077 (015) completed on
  crucible-kab-dev: critic pass, report figures all in its facts, package
  complete with every digest recomputing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:19:10 +00:00

9.1 KiB
Raw Blame History

Context: individual agents, ALZ-F02 Assessment & Disposition

Facts for this Feature's individual-agents pattern, for coding agents. Repository-wide facts are in ../../.agents/CONTEXT.md. Per-agent context is in <agent>/.agents/CONTEXT.md.

Pattern

  • Six declarative agents, one per ALZ-ASM story, plus the code-only intake (s0, ADR 0008). Only ALZ-ASM-02 and ALZ-ASM-05 declare MCP tools; the other four and the intake have none.
  • The workflow (workflows/cp1-alz-assessment-individual.json) is the orchestration. Step order comes from parallelGroup and executionLevel, and data from from_steps. The step map is STEPS in scripts/build_workflow.py, and the scaffold reads it for each agent's context file.
  • Order: a1 the Discovery package artifact, p1 parse, then s0 intake (checks the Discovery package in code; a refusal fails the run), then s1 planner, then s2 profiler, s3 risk and s4 security posture in parallel, then s5 disposition, then s6 critic. s6 has a human approval, which is disposition_signoff (ADR 0004).
  • Nothing persists between steps except their replies, as in F01 (its ADR 0010).

Sources of truth

What Where
Requirements ADO Feature 3214442 and stories 3214459–3214464. The v1.1 story files are in the engineering hub workspace, at drafts/cp1/agentic-landing-zone/stories/F02-assessment/
Decisions on the board Feature 3214442 "Decisions" table (2026-10-30 and 2026-11-02)
Changes to requirements docs/adr/ (this pattern only)
MCP servers ../../mcp-servers/ (existing, shared, unchanged)
Discovery's output shapes ../ALZ-F01: Discovery Workflow Agent/cp1-*/schemas/output.json

Platform facts (verified in modernization-platform @ c89aedcb9)

  • select_artifacts has exactly one output, artifacts: per-file records with base64 content while the selection fits the inline allowance (8 MiB live). Beyond that, a record carries only a file_path (migration 024; artifacts_activities.py).
  • parse_assessment_files accepts those records under any input key (files here). It returns parsed_content, the texts joined by \n\n---\n\n with no names, and file_metadata, which has the names in the same order (data_file_processing.py parse_files).
  • A record with no inline content is skipped silently by the parser. An oversized Discovery output therefore arrives as a missing document, and the agents report it.
  • Static inputs are applied after run inputs. So the a1 step leaves artifact_ids to the run (from_workflow) unless --artifact-id fixes them in the export.
  • Step outputs are registered as project artifacts only at the end of a run, and only when it has a project_id. The artifact name is {execution_id}_{step_id}.
  • Review decisions go to the activity log (record_review_decision), not onto artifacts. That's why the approval reference is a run input (ADR 0002).
  • Models served by the cluster's LiteLLM: gpt-5.6-sol, gpt-5.4, gpt-4.1, gpt-4o, and the embedding model text-embedding-3-small (checked 2026-11-08 through KAB's GET /api/v1/config/models, which asks LiteLLM; gpt-4.1 is new since 2026-11-01).
  • No upload step (ADR 0009). KAB hands every run input to every step, and parse_assessment_files reads a file input before the files wired to it, so a rate card uploaded as file was parsed by p1 in place of the Discovery package (run exec_20d55cfe6fd7). The rate card is the run input rate_card_document, its JSON as text.
  • Run exec_4be193061c41 (2026-11-05): intake verified the package; every step ran. Two faults: (1) every Azure tool call from ALZ-ASM-02 and ALZ-ASM-05 got HTTP 401 from Azure, because the project's Azure Discovery connection holds an az account get-access-token token (about an hour); refresh it, or store a service principal, before a run. (2) The critic failed K-12 on a Discovery agent's model read from the package; critic 0.1.2 compares only critic_model with executor_models and gets in_cloud_estate for K-10.
  • Never pass the scaffold an option: scaffold_from_stories.py --help once took --help as the stories folder and deleted the six agent folders (restored from the deployed images and the scaffold). It now refuses unless every story file is there.
  • K-12 at build time (ADR 0010): build_workflow.py refuses a critic deployed on an executor's model, reading each agent's LITELLM_MODEL; critic 0.1.4 reports K-12 from its two inputs. Run exec_0b0afb70b8b9 (first with live Azure evidence: posture 0.69, 10 findings) failed only on K-12.
  • The critic's recalculation rules (critic 0.1.6): the scaffold copies ALZ-ASM-03's and ALZ-ASM-04's rule sections into the critic's prompt (RECALC_RULES), so K-04 to K-06 recalculate with the producers' own rules. Before 0.1.6 the critic had none (run exec_1779863642d6, K-04).
  • Critic 0.2.0 (ADR 0011): K-04 to K-07, K-11 and the verdict are set in code by the critic's hooks; the scaffold keeps its app/ and tests/ when regenerating. Run exec_4d88027a4d26 failed on the model's own sums (0.70 for 1.00; 0.60 for 0.50).
  • Run exec_2cd7a953335b (critic 0.2.0): code checks ran; K-04 to K-06 passed. Two faults fixed: K-07 compared the wrong queue (critic 0.2.1), and the dependents rule counted represents edges (ADR 0012; risk 0.1.1, disposition 0.1.2).
  • Critic 0.3.0 (ADR 0013): K-02's citation lookup is in code (app/evidence.py, against the verified package); the model judges only support. Run exec_1f518b04d6e4 failed K-02 on lookups that resolve.
  • Run exec_d170329cba65 (critic 0.3.0, 2026-11-06): every citation resolved in code; K-04 to K-12 passed except K-10 (unverified). Failed K-03: the profiler had its own dependents rule without ADR 0012's represents exclusion and listed CI0010001 to CI0010008 as dependents (profiler 0.1.1). Failed K-02 on the model's support judgement, partly for missing rate-card evidence, which the prompt says isn't Discovery evidence.
  • Run exec_659b13e5f947 (profiler 0.1.1): K-03 passed (no dependents). Failed only K-02: the model judged the one evidence_ref alone. Critic 0.3.1 judges a disposition's citations together (ADR 0013 addendum).
  • The scaffold builds every prompt before it deletes anything (2026-11-06): a rewrite that no longer matched its story stopped it after the critic's folder was deleted (restored from a backup).
  • Run exec_49e355141382 (critic 0.3.1, 2026-11-06): verdict pass, the first. K-01 to K-12 pass; K-10 unverified (no call log). APP-001: no dependents, risk 0.00 (partial: eol, data, change missing), posture 0.69 with 5 advisory findings, rehost at confidence 0.50, target cost USD 11,622 a year, routed to sign-off. Confidence was 0.60 in run exec_659b13e5f947 on the same package: the disposition agent applies the adjustments differently between runs.
  • The Assessment package (ADR 0014, 2026-11-08): step s8, cp1-assessment-package-agent, code only, after disposition_signoff. It is hand-written like the intake; the scaffold doesn't touch it. INPUTS in app/package.py and PACKAGE_INPUTS in scripts/build_workflow.py must agree (a test checks). Tested on run exec_49e355141382's step outputs.
  • The Assessment report (ADR 0015, 2026-11-08): step s7, cp1-assessment-report-agent, carries the review point. The model writes from facts made in code; app/package.py there is a copy of the package agent's (a test fails if they differ). scripts/package_schemas.py writes both agents' input schemas from INPUTS.
  • A new agent must be in KAB's agent catalog before a workflow that uses it validates ("Agent '...' not found in this environment"); --update doesn't register agents. The API is POST /api/v1/agent-catalog/discover-and-register with {"url": ...} (Editor), which reads the agent card. A code-only agent needs no MCP key Secret.
  • Run exec_1b1257380df7 (RUN-ASM-AZURE-014, 2026-11-08), first on the 12-step workflow: critic pass; the report (model, about 9k tokens) quoted only figures in its facts (19 of 19 checked); the Engagement Lead continued at the report; the package (step 11 in KAB's numbering) was complete, every part's sha256 recomputed, APP-001 signed_off. Run exec_52f93ebb2dee (013, old workflow) was stopped.
  • Claude models in Foundry (2026-11-08): claude-sonnet-5-5 and claude-haiku-5-5 are deployed on crucible-kab-eng-dev-ai (the resource behind the gateway's models) but not mapped in LiteLLM, so agents can't call them. Mapping one is a LiteLLM admin step (KAB's AKS/LLM_configuration_Foundry_LiteLLM.md); the product owner chose not to do it for now.
  • Report 0.2.0 (ADR 0015 addendum): the report also reads p1.parsed_content and has a ## The application section (assets, owner, how Discovery identified it, data stores, dependencies), extracted in code by app/discovery.py.
  • Run exec_374d5d577077 (RUN-ASM-AZURE-015, report 0.2.0, 2026-11-08): critic pass; the report's new ## The application section lists all ten assets with their types, CMDB class and support group, the grouping basis, and the seven dependencies; 17 of 17 figures and every id in the report are in its facts. Report agent: about 13k tokens.