Files
cp1-agents/individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/docs/adr/0011.md
Jonathan Boniface 041e199248
Some checks failed
check agents / check (push) Failing after 12s
ALZ-F02 Assessment: critic code checks (ADR 0011), represents edges (ADR 0012)
- Critic 0.2.0 / 0.2.1: K-04 (risk), K-05 (confidence), K-06 (TCO), K-07 (routing) and K-11
  (schema and 7R enum) run in code in the critic's hooks (app/arithmetic.py,
  app/business_logic.py); post_process sets those checks, the rejected dispositions, the
  sign-off queue and the verdict. The model keeps the judgement checks; a different reading of
  a code check is kept in unverified_claims. K-07 checks the disposition agent's own queue.
  Tests on the records of run exec_4d88027a4d26, whose model sums were wrong.
- A represents edge is not a dependency (ADR 0012): risk scorer 0.1.1 and disposition 0.1.2
  leave it out of dependents and dependencies; the disposition and critic prompts carry the
  risk scorer's rule. Run exec_1f518b04d6e4: APP-001's risk 1.00 -> 0.00, out of the tail.
- The scaffold declares the critic's hooks, copies app/ in its Dockerfile, and keeps its
  app/ and tests/ when regenerating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:20:40 +00:00

50 lines
3.5 KiB
Markdown

# 0011. The critic's arithmetic, routing and schema checks run in code; the model judges the rest
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-05 |
| Decided by | Product owner, 2026-11-05: "yes" (do the arithmetic in code; leave the judgement to the model) |
| Requirements changed | ALZ-ASM-06: K-04, K-05, K-06, K-07, K-11 and the verdict are set in code. Refines [ADR 0001](0001.md): the critic is generated from its story but carries hooks, as the intake does ([ADR 0008](0008.md)) |
## Context
- ADR 0001 keeps the scoring rules as prompt rules, applied by the producing agents and recalculated by the critic.
- Run `exec_1779863642d6` showed that the critic had none of the rules it recalculates with; critic 0.1.6 copies them
into its prompt. Run `exec_4d88027a4d26` then showed the critic applying the right rules and getting the sums wrong:
- K-04: it expected a risk score of 0.70; with one factor available (fan-in 1.0, weight 0.25), the renormalised
score is 1.00, as the risk scorer recorded;
- K-05: it listed 0.90 - 0.10 - 0.20 - 0.10 and called it 0.60; it is 0.50, as the disposition agent recorded.
- The cluster's other model, `gpt-4o`, is weaker, and `gpt-5.6-sol` runs the executors (K-12, ADR 0010).
- These checks are sums, lookups and set equality over the producers' own records. They need no judgement.
## Decision
1. **Code checks, in the critic's hooks** (`app/arithmetic.py`, `app/business_logic.py`):
- **K-04:** each risk score from its factors and the fixed weights, renormalised over the available ones; each
contribution; `partial`; `missing_factors`; the 0.70 tail rule; `high_risk` and the high-risk tail.
- **K-05:** each confidence from its workings: the 0.90 base, each adjustment's prescribed delta (P2 at -0.10 a
fact, -0.20 at most), the C1 cap for repurchase and relocate, the result, `signoff_required`.
- **K-06:** each line item's rate id in the rate card at its annual rate; current, target and delta sums; a missing
cost named; `indicative`.
- **K-07:** the sign-off queue is the dispositions below 0.80 confidence, less the rejected ones.
- **K-11:** the input contract validated every record on arrival; the 7R enum is checked again.
2. **`pre_process`** works these out before the model runs and shows them to it as `checked_in_code`.
**`post_process`** sets those checks, the rejected dispositions (K-05, K-06, K-11), the sign-off queue and the
verdict: `fail` if any check fails or any check other than K-10 is unverified ([ADR 0006](0006.md)).
3. **The model keeps the judgement:** K-01 to K-03, K-08 to K-10 and K-12, and, inside the code checks, whether a
factor's value or an adjustment was due. Where its reading of a code check differs, the reading is kept in
`unverified_claims`, so the Engagement Lead sees it.
4. **When the planner refused to start** (`preconditions.failed`), the hooks change nothing: the model returns `fail`
on K-01 alone.
5. **The scaffold keeps the critic's `app/` and `tests/`** when it regenerates, and declares the hooks.
## Consequences
- **The verdict follows from the checks in code,** so a sum the model gets wrong no longer fails a run, and one the
producer gets wrong always does.
- **Whether a rule applied is still a model judgement.** The code checks that P2 is -0.10 a fact, not that the facts
were unknown, and that a rate id exists at its rate, not that it is the right equivalent.
- **The critic is no longer purely declarative.** Its code is small, standard library only, and tested on the
records of a live run (`tests/test_arithmetic.py`).