Some checks failed
check agents / check (push) Failing after 12s
- Critic 0.2.0 / 0.2.1: K-04 (risk), K-05 (confidence), K-06 (TCO), K-07 (routing) and K-11 (schema and 7R enum) run in code in the critic's hooks (app/arithmetic.py, app/business_logic.py); post_process sets those checks, the rejected dispositions, the sign-off queue and the verdict. The model keeps the judgement checks; a different reading of a code check is kept in unverified_claims. K-07 checks the disposition agent's own queue. Tests on the records of run exec_4d88027a4d26, whose model sums were wrong. - A represents edge is not a dependency (ADR 0012): risk scorer 0.1.1 and disposition 0.1.2 leave it out of dependents and dependencies; the disposition and critic prompts carry the risk scorer's rule. Run exec_1f518b04d6e4: APP-001's risk 1.00 -> 0.00, out of the tail. - The scaffold declares the critic's hooks, copies app/ in its Dockerfile, and keeps its app/ and tests/ when regenerating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
50 lines
3.5 KiB
Markdown
50 lines
3.5 KiB
Markdown
# 0011. The critic's arithmetic, routing and schema checks run in code; the model judges the rest
|
|
|
|
| | |
|
|
|---|---|
|
|
| Status | Accepted |
|
|
| Date | 2026-11-05 |
|
|
| Decided by | Product owner, 2026-11-05: "yes" (do the arithmetic in code; leave the judgement to the model) |
|
|
| Requirements changed | ALZ-ASM-06: K-04, K-05, K-06, K-07, K-11 and the verdict are set in code. Refines [ADR 0001](0001.md): the critic is generated from its story but carries hooks, as the intake does ([ADR 0008](0008.md)) |
|
|
|
|
## Context
|
|
|
|
- ADR 0001 keeps the scoring rules as prompt rules, applied by the producing agents and recalculated by the critic.
|
|
- Run `exec_1779863642d6` showed that the critic had none of the rules it recalculates with; critic 0.1.6 copies them
|
|
into its prompt. Run `exec_4d88027a4d26` then showed the critic applying the right rules and getting the sums wrong:
|
|
- K-04: it expected a risk score of 0.70; with one factor available (fan-in 1.0, weight 0.25), the renormalised
|
|
score is 1.00, as the risk scorer recorded;
|
|
- K-05: it listed 0.90 - 0.10 - 0.20 - 0.10 and called it 0.60; it is 0.50, as the disposition agent recorded.
|
|
- The cluster's other model, `gpt-4o`, is weaker, and `gpt-5.6-sol` runs the executors (K-12, ADR 0010).
|
|
- These checks are sums, lookups and set equality over the producers' own records. They need no judgement.
|
|
|
|
## Decision
|
|
|
|
1. **Code checks, in the critic's hooks** (`app/arithmetic.py`, `app/business_logic.py`):
|
|
- **K-04:** each risk score from its factors and the fixed weights, renormalised over the available ones; each
|
|
contribution; `partial`; `missing_factors`; the 0.70 tail rule; `high_risk` and the high-risk tail.
|
|
- **K-05:** each confidence from its workings: the 0.90 base, each adjustment's prescribed delta (P2 at -0.10 a
|
|
fact, -0.20 at most), the C1 cap for repurchase and relocate, the result, `signoff_required`.
|
|
- **K-06:** each line item's rate id in the rate card at its annual rate; current, target and delta sums; a missing
|
|
cost named; `indicative`.
|
|
- **K-07:** the sign-off queue is the dispositions below 0.80 confidence, less the rejected ones.
|
|
- **K-11:** the input contract validated every record on arrival; the 7R enum is checked again.
|
|
2. **`pre_process`** works these out before the model runs and shows them to it as `checked_in_code`.
|
|
**`post_process`** sets those checks, the rejected dispositions (K-05, K-06, K-11), the sign-off queue and the
|
|
verdict: `fail` if any check fails or any check other than K-10 is unverified ([ADR 0006](0006.md)).
|
|
3. **The model keeps the judgement:** K-01 to K-03, K-08 to K-10 and K-12, and, inside the code checks, whether a
|
|
factor's value or an adjustment was due. Where its reading of a code check differs, the reading is kept in
|
|
`unverified_claims`, so the Engagement Lead sees it.
|
|
4. **When the planner refused to start** (`preconditions.failed`), the hooks change nothing: the model returns `fail`
|
|
on K-01 alone.
|
|
5. **The scaffold keeps the critic's `app/` and `tests/`** when it regenerates, and declares the hooks.
|
|
|
|
## Consequences
|
|
|
|
- **The verdict follows from the checks in code,** so a sum the model gets wrong no longer fails a run, and one the
|
|
producer gets wrong always does.
|
|
- **Whether a rule applied is still a model judgement.** The code checks that P2 is -0.10 a fact, not that the facts
|
|
were unknown, and that a rate id exists at its rate, not that it is the right equivalent.
|
|
- **The critic is no longer purely declarative.** Its code is small, standard library only, and tested on the
|
|
records of a live run (`tests/test_arithmetic.py`).
|