Files
cp1-agents/individual-agents/ALZ-F02: Assessment & Disposition Workflow Agent/docs/adr/0011.md
Jonathan Boniface 041e199248
Some checks failed
check agents / check (push) Failing after 12s
ALZ-F02 Assessment: critic code checks (ADR 0011), represents edges (ADR 0012)
- Critic 0.2.0 / 0.2.1: K-04 (risk), K-05 (confidence), K-06 (TCO), K-07 (routing) and K-11
  (schema and 7R enum) run in code in the critic's hooks (app/arithmetic.py,
  app/business_logic.py); post_process sets those checks, the rejected dispositions, the
  sign-off queue and the verdict. The model keeps the judgement checks; a different reading of
  a code check is kept in unverified_claims. K-07 checks the disposition agent's own queue.
  Tests on the records of run exec_4d88027a4d26, whose model sums were wrong.
- A represents edge is not a dependency (ADR 0012): risk scorer 0.1.1 and disposition 0.1.2
  leave it out of dependents and dependencies; the disposition and critic prompts carry the
  risk scorer's rule. Run exec_1f518b04d6e4: APP-001's risk 1.00 -> 0.00, out of the tail.
- The scaffold declares the critic's hooks, copies app/ in its Dockerfile, and keeps its
  app/ and tests/ when regenerating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 13:20:40 +00:00

3.5 KiB

0011. The critic's arithmetic, routing and schema checks run in code; the model judges the rest

Status Accepted
Date 2026-11-05
Decided by Product owner, 2026-11-05: "yes" (do the arithmetic in code; leave the judgement to the model)
Requirements changed ALZ-ASM-06: K-04, K-05, K-06, K-07, K-11 and the verdict are set in code. Refines ADR 0001: the critic is generated from its story but carries hooks, as the intake does (ADR 0008)

Context

  • ADR 0001 keeps the scoring rules as prompt rules, applied by the producing agents and recalculated by the critic.
  • Run exec_1779863642d6 showed that the critic had none of the rules it recalculates with; critic 0.1.6 copies them into its prompt. Run exec_4d88027a4d26 then showed the critic applying the right rules and getting the sums wrong:
    • K-04: it expected a risk score of 0.70; with one factor available (fan-in 1.0, weight 0.25), the renormalised score is 1.00, as the risk scorer recorded;
    • K-05: it listed 0.90 - 0.10 - 0.20 - 0.10 and called it 0.60; it is 0.50, as the disposition agent recorded.
  • The cluster's other model, gpt-4o, is weaker, and gpt-5.6-sol runs the executors (K-12, ADR 0010).
  • These checks are sums, lookups and set equality over the producers' own records. They need no judgement.

Decision

  1. Code checks, in the critic's hooks (app/arithmetic.py, app/business_logic.py):
    • K-04: each risk score from its factors and the fixed weights, renormalised over the available ones; each contribution; partial; missing_factors; the 0.70 tail rule; high_risk and the high-risk tail.
    • K-05: each confidence from its workings: the 0.90 base, each adjustment's prescribed delta (P2 at -0.10 a fact, -0.20 at most), the C1 cap for repurchase and relocate, the result, signoff_required.
    • K-06: each line item's rate id in the rate card at its annual rate; current, target and delta sums; a missing cost named; indicative.
    • K-07: the sign-off queue is the dispositions below 0.80 confidence, less the rejected ones.
    • K-11: the input contract validated every record on arrival; the 7R enum is checked again.
  2. pre_process works these out before the model runs and shows them to it as checked_in_code. post_process sets those checks, the rejected dispositions (K-05, K-06, K-11), the sign-off queue and the verdict: fail if any check fails or any check other than K-10 is unverified (ADR 0006).
  3. The model keeps the judgement: K-01 to K-03, K-08 to K-10 and K-12, and, inside the code checks, whether a factor's value or an adjustment was due. Where its reading of a code check differs, the reading is kept in unverified_claims, so the Engagement Lead sees it.
  4. When the planner refused to start (preconditions.failed), the hooks change nothing: the model returns fail on K-01 alone.
  5. The scaffold keeps the critic's app/ and tests/ when it regenerates, and declares the hooks.

Consequences

  • The verdict follows from the checks in code, so a sum the model gets wrong no longer fails a run, and one the producer gets wrong always does.
  • Whether a rule applied is still a model judgement. The code checks that P2 is -0.10 a fact, not that the facts were unknown, and that a rate id exists at its rate, not that it is the right equivalent.
  • The critic is no longer purely declarative. Its code is small, standard library only, and tested on the records of a live run (tests/test_arithmetic.py).