Files
Jonathan Boniface 9e503885cc ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part
  hashed (sha256 over canonical JSON), with a manifest and a value scorecard.
- Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021):
  CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them.
- Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum
  narrower than its producer's.
- Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic
  agents; the report and Well-Architected review agents; ADRs 0012-0021.
- Workflow cases with value bars and run limits; run evaluation and usage collection scripts;
  the run context template and the field-by-field review for ADR 0016.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:13:25 +00:00

3.8 KiB

0015. A live run's inventory is built in batches, with at most three tries per batch

Status Accepted
Date 2026-11-04
Decided by Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step"
Requirements changed ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged

Context

The inventory agent built a live run's whole inventory in one model reply. Azure run exec_cf87d8d3a563 (2026-11-04) scanned 97 resources:

  • The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token output limit. Run exec_beb8879632b8's 81 records had fitted.
  • The output contract's retry returned {"records": [], "sme_review_queue": []} in 15 seconds. That met the schema, so every later step ran on an empty estate and nothing failed.

A single reply caps the estate a run can hold, and a shortfall is not detected.

Decision

  • Live runs (provider azure or gcp) answer from code in pre_process (app/business_logic.py, app/inventory.py), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and prompt.md unchanged.
  • One model call per batch of findings, through the normalise_batch sub-agent (as_tool: false, prompt prompts/normalise_batch.md), one batch at a time. The batch size is INVENTORY_BATCH_SIZE (20).
  • The model judges; code copies and counts.
    • The model judges, per finding: hostname, fqdn, asset_class, environment, os, services, facts, fact_source and conflicts.
    • Code sets: canonical_id (CI-001… in the findings' order), scope_entry_id, sources, match_basis, provider_attributes, facts.site (the region, with a GCP zone reduced to its region) and confidence (0.85 for one source; 0.79 with a conflict).
    • Code also builds sme_review_queue (every record below 0.8).
  • Every reply is checked before it is kept. It must be valid JSON (a markdown fence is tolerated). It must hold exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's types and enums. Each fact value must be found in the finding or the run context file.
  • At most three tries per batch (INVENTORY_MAX_TRIES, 3 in all). A rejected reply or a failed model call (for example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the last reply was rejected.
  • A batch that fails all three tries fails the step. The error names the batch and its findings. Later batches are not asked, and no partial inventory is returned.
  • The whole inventory is checked before the reply: one record per (de-duplicated) finding, and the output schema in code, because the output contract cannot repair a pre_process reply.

Consequences

  • An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input tokens each, well inside the model's per-minute token limit.
  • A shortfall is a failed step, never an empty inventory passed as complete.
  • Code-set fields can no longer drift from the finding, which also fixes environment being left unknown when the context file gives one (the batch prompt reads context.assets.<name>.environment).
  • Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's 19-minute budget (AGENT_MAX_DURATION_SECONDS).
  • Not changed: the sample estate still builds its inventory in one reply. Batching it means merging one asset's scan and CMDB records across applications, which needs its own design.