ALZ-F01 Discovery: package step, value scorecard, application grouping, cases
- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,57 @@
|
||||
# 0015. A live run's inventory is built in batches, with at most three tries per batch
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-04 |
|
||||
| Decided by | Product owner, 2026-11-04: "the inventory should be built iteratively not in a single one shot request. If it fails it should retry up to a maximum of three tries"; on a batch that fails all three: "it should fail the whole step" |
|
||||
| Requirements changed | ALZ-DSC-03 (ADO 3214453): how the canonical inventory of a live run is produced. What it holds is unchanged |
|
||||
|
||||
## Context
|
||||
|
||||
The inventory agent built a live run's whole inventory in one model reply. Azure run `exec_cf87d8d3a563`
|
||||
(2026-11-04) scanned 97 resources:
|
||||
|
||||
- The first reply ran 171 seconds and 69,636 tokens and was not valid JSON. It had hit the agent's 32,000-token
|
||||
output limit. Run `exec_beb8879632b8`'s 81 records had fitted.
|
||||
- The output contract's retry returned `{"records": [], "sme_review_queue": []}` in 15 seconds. That met the schema,
|
||||
so every later step ran on an empty estate and nothing failed.
|
||||
|
||||
A single reply caps the estate a run can hold, and a shortfall is not detected.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Live runs (`provider` `azure` or `gcp`) answer from code in `pre_process`** (`app/business_logic.py`,
|
||||
`app/inventory.py`), like the Well-Architected review (ADR 0014). The sample estate keeps the agent loop and
|
||||
`prompt.md` unchanged.
|
||||
- **One model call per batch** of findings, through the `normalise_batch` sub-agent (`as_tool: false`, prompt
|
||||
`prompts/normalise_batch.md`), one batch at a time. The batch size is `INVENTORY_BATCH_SIZE` (20).
|
||||
- **The model judges; code copies and counts.**
|
||||
- The model judges, per finding: `hostname`, `fqdn`, `asset_class`, `environment`, `os`, `services`, `facts`,
|
||||
`fact_source` and `conflicts`.
|
||||
- Code sets: `canonical_id` (`CI-001`… in the findings' order), `scope_entry_id`, `sources`, `match_basis`,
|
||||
`provider_attributes`, `facts.site` (the region, with a GCP zone reduced to its region) and `confidence` (0.85
|
||||
for one source; 0.79 with a conflict).
|
||||
- Code also builds `sme_review_queue` (every record below 0.8).
|
||||
- **Every reply is checked before it is kept.** It must be valid JSON (a markdown fence is tolerated). It must hold
|
||||
exactly one record per finding in the batch: none missing, extra or repeated. Each record must meet the schema's
|
||||
types and enums. Each fact value must be found in the finding or the run context file.
|
||||
- **At most three tries per batch** (`INVENTORY_MAX_TRIES`, 3 in all). A rejected reply or a failed model call (for
|
||||
example a 429 LiteLLM has given up on) spends a try. The next try waits 10 s, then 30 s, and its prompt says why the
|
||||
last reply was rejected.
|
||||
- **A batch that fails all three tries fails the step.** The error names the batch and its findings. Later batches
|
||||
are not asked, and no partial inventory is returned.
|
||||
- **The whole inventory is checked** before the reply: one record per (de-duplicated) finding, and the output schema
|
||||
in code, because the output contract cannot repair a `pre_process` reply.
|
||||
|
||||
## Consequences
|
||||
|
||||
- An estate's size no longer depends on one reply's output limit. 97 findings take 5 calls of about 6,000 input
|
||||
tokens each, well inside the model's per-minute token limit.
|
||||
- A shortfall is a failed step, never an empty inventory passed as complete.
|
||||
- Code-set fields can no longer drift from the finding, which also fixes `environment` being left `unknown` when the
|
||||
context file gives one (the batch prompt reads `context.assets.<name>.environment`).
|
||||
- Worst case is 3 tries per batch, at about 30 s per call. 100 findings could take about 8 minutes, inside the step's
|
||||
19-minute budget (`AGENT_MAX_DURATION_SECONDS`).
|
||||
- **Not changed:** the sample estate still builds its inventory in one reply. Batching it means merging one asset's
|
||||
scan and CMDB records across applications, which needs its own design.
|
||||
Reference in New Issue
Block a user