Some checks failed
check agents / check (push) Failing after 14s
- Critic 0.3.0: the hooks look up every disposition citation in the verified Discovery package (app/evidence.py). An unresolved citation fails K-02 and rejects the disposition; whether the records support the basis stays the model's judgement. - Critic 0.3.1: K-02 judges a disposition's citations together (evidence_ref is the primary record), ADR 0013 addendum. - Profiler 0.1.1: represents edges are not dependencies (ADR 0012, now covering ALZ-ASM-02). - Scaffold: builds every prompt before deleting anything. - Tests on Discovery run exec_58ba6a71d00b's package and runs 007 and 009 (15). - Run exec_49e355141382 (RUN-ASM-AZURE-012) passed on crucible-kab-dev. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
56 lines
3.6 KiB
Markdown
56 lines
3.6 KiB
Markdown
# 0013. K-02's lookup runs in code; whether the evidence supports the basis stays the model's
|
|
|
|
| | |
|
|
|---|---|
|
|
| Status | Accepted |
|
|
| Date | 2026-11-06 |
|
|
| Decided by | Product owner, 2026-11-06: "go ahead with the split" |
|
|
| Requirements changed | ALZ-ASM-06 K-02: the citation lookup is in code, the support judgement in the model. Extends [ADR 0011](0011.md) |
|
|
|
|
## Context
|
|
|
|
- K-02 passes when every `evidence_ref` "is `discovery:{discovery_execution_id}/<agent_id>/<record_key>` and resolves to
|
|
a supplied record that supports the rule-trace basis". The first half is a lookup; the second is a judgement.
|
|
- In run `exec_1f518b04d6e4`, every other check passed, and the critic model failed K-02 on two grounds:
|
|
- `ALZ-DSC-05/CI-002` and `ALZ-DSC-05/CI-010` "do not resolve". They do: the package's classification records carry
|
|
those `canonical_id`s, and the prompt says classification records are cited by `canonical_id`;
|
|
- the rate-card mappings have no Discovery citation. The rate card isn't Discovery evidence: its items are cited by
|
|
`rate_id` in the TCO line items, which K-06 checks in code.
|
|
|
|
## Decision
|
|
|
|
1. **The critic's hooks look up every citation** of every disposition (`evidence_ref` and `evidence_refs`) in the
|
|
Discovery package the intake verified (`app/evidence.py`). A citation resolves when:
|
|
- its execution id is the run's `discovery_execution_id`;
|
|
- its agent is one Assessment cites: ALZ-DSC-03 (inventory), 04 (CMDB and ITSM), 05 (classification),
|
|
06 (dependencies) or 07 (network);
|
|
- that agent's part of the package has a record with that key in an id field (`canonical_id`, `drift_id`, `ci_ref`,
|
|
`edge_id`, a node `id`, or a finding, rule, range or flow id), or the key is the part's own field name (a list
|
|
cited as a whole, for example to show it's empty).
|
|
2. **A citation that doesn't resolve fails K-02** and rejects the disposition, in code.
|
|
3. **When every citation resolves,** the model's K-02 stands as it wrote it: it judges whether the cited records
|
|
support the rule-trace basis. The prompt says a resolved citation is valid, and that the rate card is cited by
|
|
`rate_id`, never as Discovery evidence.
|
|
|
|
## Consequences
|
|
|
|
- **A wrong lookup by the model no longer fails a run;** a citation that really is wrong always does.
|
|
- **The model can still fail K-02 on support,** which is the judgement K-02 exists for.
|
|
- **The package is read twice,** by the intake (to verify it) and the critic (to look up citations). The finding
|
|
logic is the intake's, copied, not shared, since the two are separate images.
|
|
|
|
## Addendum, 2026-11-06: the citations are judged together
|
|
|
|
Decided by the product owner, 2026-11-06 ("Lets go with A recomenadation"), after run `exec_659b13e5f947`.
|
|
|
|
- **What happened:** every citation resolved, and the critic model said the broader `evidence_refs` helped, but it
|
|
failed K-02 because the one required `evidence_ref` (`ALZ-DSC-03/CI-001`) didn't support the basis on its own. K-02's
|
|
row says "Every `evidence_ref` … supports the rule-trace basis", and the model read it as the single field.
|
|
- **Decision:** K-02 judges support on a disposition's citations taken together. `evidence_ref` is the primary record
|
|
and `evidence_refs` the rest. A disposition covers a whole application, so it is never failed because its primary
|
|
record alone covers only one asset. K-02's row in ALZ-ASM-06 is rewritten to say so (critic 0.3.1).
|
|
- **Not chosen:** keeping the single-record test and making the disposition's `evidence_ref` the package's application
|
|
record. That would need the package's `applications` part in the lookup and a change to ALZ-ASM-04.
|
|
- **Requirements changed:** ALZ-ASM-06's K-02 row.
|
|
|