# 0018. Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code | | | |---|---| | Status | Accepted | | Date | 2026-11-05 | | Decided by | Product owner, 2026-11-05: "the model should ingest the data and normalise it to the data schema that we own and judge". Plan and decisions D1–D4: `engineering_hub_responsiblities/reports/cp1/cmdb-upload-ingestion-plan-2026-11-05.md` | | Requirements changed | ALZ-DSC-04 (ADO 3214454): a live run reads CMDB exports uploaded with the run. ALZ-DSC-01 (ADO 3214451): an uploaded CMDB is a source, not a gap | ## Context - **A live run never read a CMDB.** `cmdb` was always a source gap, and the reconciliation step returned nothing. - **A second uploaded file broke the run.** The upload step joins files with `\n\n---\n\n`. The planner, inventory and report parsed the whole text as one YAML document, failed, and carried on as if no run context had been given. - **Exports differ in layout.** CMDB exports differ by customer and tool, so no column can be fixed in code. ## Decision - **Uploads are listed in the run context.** The context's `uploads:` list names the files uploaded after it, in order, each with a `source` (e.g. `cmdb`). `kab_run_workflow.py` uploads them. The shared `app/run_context.py` (identical in the planner, inventory, reconciliation and report agents) splits the text into the context and each listed file. It cuts a CSV, or each XLSX sheet, into a header line and one line per row. It knows no column. - **The model normalises; code judges.** The `normalise_cmdb_batch` sub-agent maps each batch of rows (25 by default, each batch with its header) to `schemas/cmdb_ci.json`. Every string is copied verbatim; `environment` and `lifecycle` are the only judgments; any other column can be kept in `other_attributes`. Code rejects a reply unless: - every row has exactly one result (or a reasoned `skip`); - every value is found in its row; - every key is a header column; - every judgment is an allowed value. A batch gets at most three tries (ADR 0015's rule). One that fails all three, or an upload the platform couldn't parse, fails the step. - **Injections:** a row is flagged when the model or a code pattern sees an instruction. The quoted text goes to the critic (`cmdb_injections`) and to the report. - **Reconciliation is code** (`app/reconcile.py`), against the step's own scope listing (`app/scope.py`, the planner's rules) and the inventory: - **match:** by cloud resource id, otherwise by name when exactly one listed resource has it; - **stale attributes:** - environment, compared at enum level (D1); - location, compared with the record's site; - a retired CI whose resource exists (`investigate`); - owner and business service, when the record carries them; - **ghost:** an unmatched, not-retired CI whose id lies in a scope container the listing shows (D2). It's `investigate` outside `context.scope.regions`, in `exclude`, or with an unknown lifecycle, and otherwise `retire_ci`. A CI with no cloud id is `unplaced` (D3), and one in another container is `out_of_scope`. Neither is drift. - **orphan:** an inventory record no CI matched (`add_ci`). Folded components are never orphans. - **The planner** adds one `ingest_cmdb` task per uploaded CMDB file, and drops the `cmdb` gap when `include_cmdb` is `yes` and a file is uploaded. ## Consequences - **New layouts need no code change.** A new export layout, whether a ServiceNow CSV or an asset register XLSX with other column names, is handled by the model's mapping. Code checks that mapping on every row. - **The model never decides drift.** A manipulated normaliser can only return values that are in the row. - **The step makes model calls** (one per batch) on a live run with a CMDB. Without one, it makes none. - **Subscription scope is cautious.** At subscription scope, a CI in a resource group with no listed resources is `out_of_scope`, not a ghost. Returning the connection's scope root from the existing discovery servers would make this exact; that is left until it's needed. - **Planner and step listings can differ.** The reconciliation step lists the scope again, so a resource created between the planner's listing and this one is matched, not a ghost. ## Changes after the first live run (2026-11-05) Run `exec_a0a3c486ec9b` (case `azure-crucible-kab-dev-cmdb`) passed 13 of 19 checks. Every planted CMDB result was found except the environment difference. - **`env` tags were redacted** by the discovery servers' shared `_scrub` (`env` is a secret key, for a container's environment variables). The inventory then took `env: [redacted]` over `Environment: dev`, and 8 of 10 records had environment `unknown`, so the planted environment difference couldn't be found. A single value under `env` is now kept; a list or map under it is still redacted (azure-discovery 0.1.6, gcp-discovery 0.1.4, aws-discovery 0.1.2). - **Orphans name their resource** (`field` `resource`, `discovered_value` the resource id). Before, they named only the inventory's canonical id (reconciliation 0.5.1). - **The critic** no longer raises "drift with no CMDB source" when `source_gaps` has no `cmdb` (critic 0.5.2). - **The report's drift table** shows both the asset and the CMDB record (report 0.7.2).