- New cp1-discovery-package-agent (ADR 0020): one package per reviewed run, every part hashed (sha256 over canonical JSON), with a manifest and a value scorecard. - Applications grouped CMDB -> tag -> resource group with a judged model fallback (ADR 0021): CMDB 0.7.0 carries ownership, dependency mapping 0.6.0 groups, report 0.9.0 shows them. - Critic 0.6.1 accepts the producers' enums; build_workflow.py refuses a consumer enum narrower than its producer's. - Code hooks, sub-agents and tests for the inventory, CMDB, dependency, planner and critic agents; the report and Well-Architected review agents; ADRs 0012-0021. - Workflow cases with value bars and run limits; run evaluation and usage collection scripts; the run context template and the field-by-field review for ADR 0016. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.3 KiB
0018. Uploaded CMDB exports are normalised by the model to our CI schema, and judged and reconciled in code
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "the model should ingest the data and normalise it to the data schema that we own and judge". Plan and decisions D1–D4: engineering_hub_responsiblities/reports/cp1/cmdb-upload-ingestion-plan-2026-11-05.md |
| Requirements changed | ALZ-DSC-04 (ADO 3214454): a live run reads CMDB exports uploaded with the run. ALZ-DSC-01 (ADO 3214451): an uploaded CMDB is a source, not a gap |
Context
- A live run never read a CMDB.
cmdbwas always a source gap, and the reconciliation step returned nothing. - A second uploaded file broke the run. The upload step joins files with
\n\n---\n\n. The planner, inventory and report parsed the whole text as one YAML document, failed, and carried on as if no run context had been given. - Exports differ in layout. CMDB exports differ by customer and tool, so no column can be fixed in code.
Decision
-
Uploads are listed in the run context. The context's
uploads:list names the files uploaded after it, in order, each with asource(e.g.cmdb).kab_run_workflow.pyuploads them. The sharedapp/run_context.py(identical in the planner, inventory, reconciliation and report agents) splits the text into the context and each listed file. It cuts a CSV, or each XLSX sheet, into a header line and one line per row. It knows no column. -
The model normalises; code judges. The
normalise_cmdb_batchsub-agent maps each batch of rows (25 by default, each batch with its header) toschemas/cmdb_ci.json. Every string is copied verbatim;environmentandlifecycleare the only judgments; any other column can be kept inother_attributes. Code rejects a reply unless:- every row has exactly one result (or a reasoned
skip); - every value is found in its row;
- every key is a header column;
- every judgment is an allowed value.
A batch gets at most three tries (ADR 0015's rule). One that fails all three, or an upload the platform couldn't parse, fails the step.
- every row has exactly one result (or a reasoned
-
Injections: a row is flagged when the model or a code pattern sees an instruction. The quoted text goes to the critic (
cmdb_injections) and to the report. -
Reconciliation is code (
app/reconcile.py), against the step's own scope listing (app/scope.py, the planner's rules) and the inventory:- match: by cloud resource id, otherwise by name when exactly one listed resource has it;
- stale attributes:
- environment, compared at enum level (D1);
- location, compared with the record's site;
- a retired CI whose resource exists (
investigate); - owner and business service, when the record carries them;
- ghost: an unmatched, not-retired CI whose id lies in a scope container the listing shows (D2). It's
investigateoutsidecontext.scope.regions, inexclude, or with an unknown lifecycle, and otherwiseretire_ci. A CI with no cloud id isunplaced(D3), and one in another container isout_of_scope. Neither is drift. - orphan: an inventory record no CI matched (
add_ci). Folded components are never orphans.
-
The planner adds one
ingest_cmdbtask per uploaded CMDB file, and drops thecmdbgap wheninclude_cmdbisyesand a file is uploaded.
Consequences
- New layouts need no code change. A new export layout, whether a ServiceNow CSV or an asset register XLSX with other column names, is handled by the model's mapping. Code checks that mapping on every row.
- The model never decides drift. A manipulated normaliser can only return values that are in the row.
- The step makes model calls (one per batch) on a live run with a CMDB. Without one, it makes none.
- Subscription scope is cautious. At subscription scope, a CI in a resource group with no listed resources is
out_of_scope, not a ghost. Returning the connection's scope root from the existing discovery servers would make this exact; that is left until it's needed. - Planner and step listings can differ. The reconciliation step lists the scope again, so a resource created between the planner's listing and this one is matched, not a ghost.
Changes after the first live run (2026-11-05)
Run exec_a0a3c486ec9b (case azure-crucible-kab-dev-cmdb) passed 13 of 19 checks. Every planted CMDB result
was found except the environment difference.
envtags were redacted by the discovery servers' shared_scrub(envis a secret key, for a container's environment variables). The inventory then tookenv: [redacted]overEnvironment: dev, and 8 of 10 records had environmentunknown, so the planted environment difference couldn't be found. A single value underenvis now kept; a list or map under it is still redacted (azure-discovery 0.1.6, gcp-discovery 0.1.4, aws-discovery 0.1.2).- Orphans name their resource (
fieldresource,discovered_valuethe resource id). Before, they named only the inventory's canonical id (reconciliation 0.5.1). - The critic no longer raises "drift with no CMDB source" when
source_gapshas nocmdb(critic 0.5.2). - The report's drift table shows both the asset and the CMDB record (report 0.7.2).