# 0015. A live run's inventory is built in batches, with at most three tries per batch | | | |---|---| | Status | Accepted | | Date | 2026-11-04 | | Decided by | Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step" | | Requirements changed | ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged | ## Context The inventory agent built a live run's whole inventory in one model reply. Azure run `exec_cf87d8d3a563` (2026-11-04) scanned 97 resources: - The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token output limit. Run `exec_beb8879632b8`'s 81 records had fitted. - The output contract's retry returned `{"records": [], "sme_review_queue": []}` in 15 seconds. That met the schema, so every later step ran on an empty estate and nothing failed. A single reply caps the estate a run can hold, and a shortfall is not detected. ## Decision - **Live runs (`provider` `azure` or `gcp`) answer from code in `pre_process`** (`app/business_logic.py`, `app/inventory.py`), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and `prompt.md` unchanged. - **One model call per batch** of findings, through the `normalise_batch` sub-agent (`as_tool: false`, prompt `prompts/normalise_batch.md`), one batch at a time. The batch size is `INVENTORY_BATCH_SIZE` (20). - **The model judges; code copies and counts.** - The model judges, per finding: `hostname`, `fqdn`, `asset_class`, `environment`, `os`, `services`, `facts`, `fact_source` and `conflicts`. - Code sets: `canonical_id` (`CI-001`… in the findings' order), `scope_entry_id`, `sources`, `match_basis`, `provider_attributes`, `facts.site` (the region, with a GCP zone reduced to its region) and `confidence` (0.85 for one source; 0.79 with a conflict). - Code also builds `sme_review_queue` (every record below 0.8). - **Every reply is checked before it is kept.** It must be valid JSON (a markdown fence is tolerated). It must hold exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's types and enums. Each fact value must be found in the finding or the run context file. - **At most three tries per batch** (`INVENTORY_MAX_TRIES`, 3 in all). A rejected reply or a failed model call (for example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the last reply was rejected. - **A batch that fails all three tries fails the step.** The error names the batch and its findings. Later batches are not asked, and no partial inventory is returned. - **The whole inventory is checked** before the reply: one record per (de-duplicated) finding, and the output schema in code, because the output contract cannot repair a `pre_process` reply. ## Consequences - An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input tokens each, well inside the model's per-minute token limit. - A shortfall is a failed step, never an empty inventory passed as complete. - Code-set fields can no longer drift from the finding, which also fixes `environment` being left `unknown` when the context file gives one (the batch prompt reads `context.assets..environment`). - Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's 19-minute budget (`AGENT_MAX_DURATION_SECONDS`). - **Not changed:** the sample estate still builds its inventory in one reply. Batching it means merging one asset's scan and CMDB records across applications, which needs its own design.