ALZ-F02 Assessment: critic code checks (ADR 0011), represents edges (ADR 0012)
Some checks failed
check agents / check (push) Failing after 12s
Some checks failed
check agents / check (push) Failing after 12s
- Critic 0.2.0 / 0.2.1: K-04 (risk), K-05 (confidence), K-06 (TCO), K-07 (routing) and K-11 (schema and 7R enum) run in code in the critic's hooks (app/arithmetic.py, app/business_logic.py); post_process sets those checks, the rejected dispositions, the sign-off queue and the verdict. The model keeps the judgement checks; a different reading of a code check is kept in unverified_claims. K-07 checks the disposition agent's own queue. Tests on the records of run exec_4d88027a4d26, whose model sums were wrong. - A represents edge is not a dependency (ADR 0012): risk scorer 0.1.1 and disposition 0.1.2 leave it out of dependents and dependencies; the disposition and critic prompts carry the risk scorer's rule. Run exec_1f518b04d6e4: APP-001's risk 1.00 -> 0.00, out of the tail. - The scaffold declares the critic's hooks, copies app/ in its Dockerfile, and keeps its app/ and tests/ when regenerating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,267 @@
|
||||
"""The critic's mechanical checks, in code (F02 ADR 0011): sums, lookups and set equality, never judgement.
|
||||
|
||||
The model judges which rule applies (a factor's value, whether an adjustment is due, which rate maps to an asset).
|
||||
This module checks only what follows from the producers' own records by arithmetic:
|
||||
|
||||
- K-04: every risk score from its factors' values and the fixed weights, renormalised over the available ones; the
|
||||
partial flag, the missing factors, the 0.70 tail rule and the high-risk tail.
|
||||
- K-05: every confidence from its workings: the 0.90 base, each adjustment's prescribed delta, the C1 cap, the result.
|
||||
- K-06: every TCO model: each rate id in the rate card at its annual rate, the line items' sums, the delta.
|
||||
- K-07: the disposition agent's sign-off queue is exactly its dispositions with confidence below 0.80. The critic's
|
||||
own queue is those, less the dispositions it rejects.
|
||||
- K-11: the records passed the input contract on arrival (schema and 7R enum), so they validate.
|
||||
|
||||
The rules are ALZ-ASM-03's and ALZ-ASM-04's (their prompts' Scoring and Tail, Confidence, and TCO rules). Standard
|
||||
library only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
W = {"eol": 0.35, "fan_in": 0.25, "data": 0.25, "change": 0.15}
|
||||
TAIL_CUTOFF = 0.70
|
||||
BASE = 0.90
|
||||
SIGNOFF_BELOW = 0.80
|
||||
INTERIM_CAP = 0.79
|
||||
INTERIM = ("repurchase", "relocate")
|
||||
SEVEN_R = ("retire", "repurchase", "replatform", "refactor", "retain", "relocate", "rehost")
|
||||
# Each adjustment's delta (ALZ-ASM-04 Confidence Rules). P2 is -0.10 per unknown fact, -0.20 at most in all.
|
||||
DELTA = {"P1": -0.10, "P3": -0.10, "P4": -0.10, "P5": -0.10, "P6": -0.15}
|
||||
P2_STEP, P2_FLOOR = -0.10, -0.20
|
||||
TOL = 0.005
|
||||
CODE_CHECKS = ("K-04", "K-05", "K-06", "K-07", "K-11")
|
||||
REJECTS = ("K-05", "K-06", "K-11") # a failure rejects the disposition; K-04 and K-07 fail the run
|
||||
|
||||
|
||||
def _near(a: Any, b: Any) -> bool:
|
||||
if a is None or b is None:
|
||||
return a is None and b is None
|
||||
return abs(float(a) - float(b)) <= TOL
|
||||
|
||||
|
||||
def _detail(record: str, expected: Any, recorded: Any, reason: str) -> Dict[str, str]:
|
||||
return {"record": record, "expected": str(expected), "recorded": str(recorded), "reason": reason}
|
||||
|
||||
|
||||
def risk(scores: List[dict], high_risk_tail: List[str]) -> List[dict]:
|
||||
"""K-04 details: empty when every score recalculates."""
|
||||
out = []
|
||||
for s in scores or []:
|
||||
app = s.get("application_id", "?")
|
||||
factors = {f.get("factor"): f for f in s.get("factors") or []}
|
||||
for name, f in factors.items():
|
||||
if name in W and not _near(f.get("weight"), W[name]):
|
||||
out.append(_detail(f"{app} {name}", f"weight {W[name]}", f"weight {f.get('weight')}", "weights are fixed"))
|
||||
value = f.get("value")
|
||||
expected = None if value is None else round(float(value) * W.get(name, 0), 4)
|
||||
if name in W and not _near(f.get("contribution"), expected):
|
||||
out.append(_detail(f"{app} {name}", f"contribution {expected}", f"contribution {f.get('contribution')}",
|
||||
"contribution = value x weight, null when the value is missing"))
|
||||
available = {k: f["value"] for k, f in factors.items() if k in W and f.get("value") is not None}
|
||||
missing = sorted(k for k in W if k not in available)
|
||||
if not available:
|
||||
out.append(_detail(app, "at least one factor with a value", "none", "no factor is available to score"))
|
||||
continue
|
||||
score = round(sum(W[k] * float(v) for k, v in available.items()) / sum(W[k] for k in available), 2)
|
||||
if not _near(s.get("score"), score):
|
||||
out.append(_detail(app, f"score {score}", f"score {s.get('score')}",
|
||||
f"SUM(W x value) / SUM(W) over the available factors {sorted(available)}"))
|
||||
if bool(s.get("partial")) != (len(available) < 4):
|
||||
out.append(_detail(app, f"partial {len(available) < 4}", f"partial {s.get('partial')}", "partial = fewer than 4 factors"))
|
||||
if sorted(s.get("missing_factors") or []) != missing:
|
||||
out.append(_detail(app, f"missing_factors {missing}", f"missing_factors {sorted(s.get('missing_factors') or [])}",
|
||||
"the factors with no value"))
|
||||
rules = s.get("tail_rules") or []
|
||||
if (score >= TAIL_CUTOFF) != ("score>=0.70" in rules):
|
||||
out.append(_detail(app, f"'score>=0.70' {'in' if score >= TAIL_CUTOFF else 'not in'} tail_rules", rules,
|
||||
"the 0.70 tail rule"))
|
||||
if bool(s.get("high_risk")) != bool(rules):
|
||||
out.append(_detail(app, f"high_risk {bool(rules)}", f"high_risk {s.get('high_risk')}", "high_risk = any tail rule"))
|
||||
if bool(s.get("high_risk")) != (app in (high_risk_tail or [])):
|
||||
out.append(_detail(app, f"{'in' if s.get('high_risk') else 'not in'} high_risk_tail", high_risk_tail,
|
||||
"the tail lists exactly the high-risk applications"))
|
||||
return out
|
||||
|
||||
|
||||
def _rule(label: Any) -> Optional[str]:
|
||||
m = re.match(r"\s*(P[1-6]|C1)\b", str(label or ""))
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def confidence(dispositions: List[dict]) -> Dict[str, List[dict]]:
|
||||
"""K-05 details, per application."""
|
||||
out: Dict[str, List[dict]] = {}
|
||||
for d in dispositions or []:
|
||||
app, problems = d.get("application_id", "?"), []
|
||||
w = d.get("confidence_workings") or {}
|
||||
if not _near(w.get("base"), BASE):
|
||||
problems.append(_detail(app, f"base {BASE}", f"base {w.get('base')}", "confidence starts at 0.90"))
|
||||
total, p2, seen = 0.0, 0.0, set()
|
||||
for a in w.get("adjustments") or []:
|
||||
rule, delta = _rule(a.get("rule")), float(a.get("delta") or 0)
|
||||
if rule is None:
|
||||
problems.append(_detail(app, "an adjustment P1-P6", a.get("rule"), "not one of the confidence rules"))
|
||||
continue
|
||||
if rule == "P2":
|
||||
if not _near(round(delta / P2_STEP) * P2_STEP, delta) or delta > 0:
|
||||
problems.append(_detail(app, "P2 in steps of -0.10", delta, "P2 is -0.10 per unknown fact"))
|
||||
p2 += delta
|
||||
elif rule in DELTA:
|
||||
if rule in seen:
|
||||
problems.append(_detail(app, f"{rule} once", f"{rule} again", "each rule applies once"))
|
||||
if not _near(delta, DELTA[rule]):
|
||||
problems.append(_detail(app, f"{rule} {DELTA[rule]}", f"{rule} {delta}", "the rule's fixed delta"))
|
||||
seen.add(rule)
|
||||
total += delta
|
||||
if p2 < P2_FLOOR - TOL:
|
||||
problems.append(_detail(app, f"P2 at least {P2_FLOOR}", f"P2 {round(p2, 2)}", "P2 is capped at -0.20 in all"))
|
||||
expected = round(max(0.0, BASE + total), 2)
|
||||
if d.get("disposition") in INTERIM:
|
||||
expected = min(expected, INTERIM_CAP)
|
||||
if not _near(w.get("result"), expected):
|
||||
problems.append(_detail(app, f"result {expected}", f"result {w.get('result')}",
|
||||
"0.90 plus the adjustments, at least 0, with C1 for repurchase and relocate"))
|
||||
if not _near(d.get("confidence"), w.get("result")):
|
||||
problems.append(_detail(app, f"confidence {w.get('result')}", f"confidence {d.get('confidence')}",
|
||||
"confidence is the workings' result"))
|
||||
if bool(d.get("signoff_required")) != (float(d.get("confidence") or 0) < SIGNOFF_BELOW):
|
||||
problems.append(_detail(app, f"signoff_required {float(d.get('confidence') or 0) < SIGNOFF_BELOW}",
|
||||
f"signoff_required {d.get('signoff_required')}", "sign-off below 0.80"))
|
||||
if problems:
|
||||
out[app] = problems
|
||||
return out
|
||||
|
||||
|
||||
def _rate_card(document: Any) -> Optional[dict]:
|
||||
if isinstance(document, dict):
|
||||
return document
|
||||
try:
|
||||
card = json.loads(document) if document else None
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return card if isinstance(card, dict) else None
|
||||
|
||||
|
||||
def tco(dispositions: List[dict], rate_card_document: Any) -> Dict[str, List[dict]]:
|
||||
"""K-06 details, per application."""
|
||||
card = _rate_card(rate_card_document)
|
||||
rates = {i.get("rate_id"): i for i in (card or {}).get("items") or [] if isinstance(i, dict)}
|
||||
out: Dict[str, List[dict]] = {}
|
||||
for d in dispositions or []:
|
||||
app, problems, m = d.get("application_id", "?"), [], d.get("tco_model") or {}
|
||||
if card is None:
|
||||
out[app] = [_detail(app, "a rate card", "none readable", "rate_card_document is not a JSON rate card")]
|
||||
continue
|
||||
lines = m.get("line_items") or []
|
||||
for li in lines:
|
||||
rid = li.get("rate_id")
|
||||
if rid is None:
|
||||
continue
|
||||
if rid not in rates:
|
||||
problems.append(_detail(f"{app} {li.get('canonical_id')}", "a rate id in the rate card", rid, "unknown rate id"))
|
||||
elif not _near(li.get("target"), rates[rid].get("annual_rate")):
|
||||
problems.append(_detail(f"{app} {li.get('canonical_id')}", f"target {rates[rid].get('annual_rate')}",
|
||||
f"target {li.get('target')}", f"the annual rate of {rid}"))
|
||||
if m.get("currency") and card.get("currency") and m["currency"] != card["currency"]:
|
||||
problems.append(_detail(app, f"currency {card['currency']}", f"currency {m['currency']}", "the rate card's currency"))
|
||||
disposition = d.get("disposition")
|
||||
currents = [li.get("current") for li in lines]
|
||||
current = None if (not lines or any(c is None for c in currents)) else round(sum(float(c) for c in currents), 2)
|
||||
if disposition == "retire":
|
||||
target = 0.0
|
||||
elif disposition == "retain":
|
||||
target = current
|
||||
else:
|
||||
targets = [li.get("target") for li in lines if li.get("rate_id") is not None]
|
||||
target = None if not targets or any(t is None for t in targets) else round(sum(float(t) for t in targets), 2)
|
||||
if not _near(m.get("current_annual"), current):
|
||||
problems.append(_detail(app, f"current_annual {current}", f"current_annual {m.get('current_annual')}",
|
||||
"the sum of the line items' current costs; null if any is missing"))
|
||||
if current is None and lines and not m.get("missing_inputs"):
|
||||
problems.append(_detail(app, "the missing costs named", "missing_inputs empty", "a missing cost is named"))
|
||||
if not _near(m.get("target_annual"), target):
|
||||
problems.append(_detail(app, f"target_annual {target}", f"target_annual {m.get('target_annual')}",
|
||||
"0 for retire, current for retain, else the sum of the mapped rates"))
|
||||
delta = None if target is None or current is None else round(target - current, 2)
|
||||
if not _near(m.get("annual_delta"), delta):
|
||||
problems.append(_detail(app, f"annual_delta {delta}", f"annual_delta {m.get('annual_delta')}",
|
||||
"target minus current; null if either is null"))
|
||||
if m.get("indicative") is not True:
|
||||
problems.append(_detail(app, "indicative true", f"indicative {m.get('indicative')}", "the figure is indicative"))
|
||||
if problems:
|
||||
out[app] = problems
|
||||
return out
|
||||
|
||||
|
||||
def schema(dispositions: List[dict]) -> Dict[str, List[dict]]:
|
||||
"""K-11 details: the input contract validated every record on arrival; the 7R enum is checked again here."""
|
||||
out: Dict[str, List[dict]] = {}
|
||||
for d in dispositions or []:
|
||||
bad = [c for c in [d.get("disposition")] + [t.get("category") for t in d.get("rule_trace") or []] if c not in SEVEN_R]
|
||||
if bad:
|
||||
out[d.get("application_id", "?")] = [_detail(d.get("application_id", "?"), "a 7R category", bad, "the 7R enum")]
|
||||
return out
|
||||
|
||||
|
||||
def check(payload: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Every code check's details, from the critic's inputs."""
|
||||
dispositions = payload.get("dispositions") or []
|
||||
return {"K-04": risk(payload.get("scores") or [], payload.get("high_risk_tail") or []),
|
||||
"K-05": confidence(dispositions), "K-06": tco(dispositions, payload.get("rate_card_document")),
|
||||
"K-11": schema(dispositions), "K-07": routing(dispositions, payload.get("signoff_queue") or []),
|
||||
"confidence": {d.get("application_id"): d.get("confidence") for d in dispositions}}
|
||||
|
||||
|
||||
def routing(dispositions: List[dict], producer_queue: List[str]) -> List[dict]:
|
||||
"""K-07 details: the disposition agent routed exactly the dispositions below 0.80 (ALZ-ASM-04)."""
|
||||
below = sorted(d.get("application_id") for d in dispositions or []
|
||||
if d.get("confidence") is not None and float(d["confidence"]) < SIGNOFF_BELOW)
|
||||
if sorted(producer_queue or []) == below:
|
||||
return []
|
||||
return [_detail("signoff_queue (ALZ-ASM-04)", below, sorted(producer_queue or []),
|
||||
"the disposition agent's queue is its dispositions with confidence below 0.80")]
|
||||
|
||||
|
||||
def apply(reply: Dict[str, Any], found: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""The critic's reply with the code checks, the sign-off queue and the verdict set from `found`."""
|
||||
checks = {c.get("check_id"): c for c in reply.get("checks") or []}
|
||||
notes = list(reply.get("unverified_claims") or [])
|
||||
by_app_reject: Dict[str, List[str]] = {}
|
||||
for cid in ("K-05", "K-06", "K-11"):
|
||||
for app in found[cid]:
|
||||
by_app_reject.setdefault(app, []).append(cid)
|
||||
rejected = [r for r in reply.get("rejected_dispositions") or [] if r.get("check_id") not in CODE_CHECKS]
|
||||
for app, cids in by_app_reject.items():
|
||||
for cid in cids:
|
||||
rejected.append({"application_id": app, "check_id": cid, "named_dependent": None,
|
||||
"reason": "; ".join(f"{x['record']}: expected {x['expected']}, recorded {x['recorded']}"
|
||||
for x in found[cid][app])})
|
||||
rejected_apps = {r.get("application_id") for r in rejected}
|
||||
queue = sorted(a for a, c in found["confidence"].items() if c is not None and float(c) < SIGNOFF_BELOW
|
||||
and a not in rejected_apps)
|
||||
details = {"K-04": found["K-04"], "K-05": [x for v in found["K-05"].values() for x in v],
|
||||
"K-06": [x for v in found["K-06"].values() for x in v], "K-07": found["K-07"],
|
||||
"K-11": [x for v in found["K-11"].values() for x in v]}
|
||||
for cid in CODE_CHECKS:
|
||||
model = checks.get(cid) or {}
|
||||
result = "fail" if details[cid] else "pass"
|
||||
if model.get("result") not in (None, result):
|
||||
reading = "; ".join(x.get("reason", "") for x in model.get("details") or []) or "no detail"
|
||||
notes.append(f"{cid}: the critic model's reading was {model.get('result')} ({reading[:400]}); the result is "
|
||||
f"{result}, set in code from the producers' records (ADR 0011)")
|
||||
checks[cid] = {"check_id": cid, "result": result, "details": details[cid]}
|
||||
ordered = [checks[f"K-{i:02d}"] for i in range(1, 13) if f"K-{i:02d}" in checks]
|
||||
failed = [c["check_id"] for c in ordered if c["result"] == "fail"]
|
||||
unverified = [c["check_id"] for c in ordered if c["result"] == "unverified" and c["check_id"] != "K-10"]
|
||||
verdict = "fail" if failed or unverified else "pass"
|
||||
changed = verdict != reply.get("verdict")
|
||||
summary = reply.get("summary") or ""
|
||||
if changed or failed:
|
||||
summary = (f"Verdict {verdict}, set in code from the checks (ADR 0011): "
|
||||
+ (f"{', '.join(failed)} failed" if failed else "no check failed")
|
||||
+ (f"; {', '.join(unverified)} unverified" if unverified else "")
|
||||
+ ". K-04 to K-07 and K-11 are checked in code; the other checks are the critic model's.")
|
||||
return {**reply, "verdict": verdict, "checks": ordered, "rejected_dispositions": rejected,
|
||||
"signoff_queue": queue, "unverified_claims": notes, "summary": summary}
|
||||
@@ -0,0 +1,49 @@
|
||||
"""The critic's mechanical checks run in code (F02 ADR 0011); the model judges the rest.
|
||||
|
||||
`pre_process` works out K-04 to K-07 and K-11 from the producers' records, before the model runs, and shows the
|
||||
result to the model as `checked_in_code`. `post_process` then sets those checks, the sign-off queue and the verdict
|
||||
in the reply. What the model judged (K-01 to K-03, K-08 to K-10, K-12) is kept as it wrote it; where its reading of a
|
||||
code check differs, it's kept in `unverified_claims` for the Engagement Lead.
|
||||
|
||||
The two hooks run in the same request, so the result passes between them in a ContextVar: one per request, never
|
||||
shared between requests that run at once.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import contextvars
|
||||
import logging
|
||||
from typing import Any, Optional
|
||||
|
||||
from app import arithmetic
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
_found: contextvars.ContextVar[Optional[dict]] = contextvars.ContextVar("critic_checked_in_code", default=None)
|
||||
|
||||
|
||||
def pre_process(payload: dict) -> Optional[Any]:
|
||||
if (payload.get("preconditions") or {}).get("failed"):
|
||||
_found.set(None) # the planner refused: the critic returns fail on K-01 alone, and nothing is checked here
|
||||
return None
|
||||
found = arithmetic.check(payload)
|
||||
_found.set(found)
|
||||
payload["checked_in_code"] = {
|
||||
"note": "K-04, K-05, K-06, K-07 and K-11 are checked in code from the producers' records (ADR 0011), and "
|
||||
"their results are set in your reply after you answer. Report them as you find them; the other checks "
|
||||
"are yours alone.",
|
||||
"K-04": found["K-04"] or "every risk score recalculates",
|
||||
"K-05": found["K-05"] or "every confidence recalculates",
|
||||
"K-06": found["K-06"] or "every TCO model adds up against the rate card",
|
||||
"K-07": found["K-07"] or "the disposition agent routed exactly its dispositions below 0.80",
|
||||
"K-11": found["K-11"] or "every record passed the input contract; every category is in the 7R enum",
|
||||
}
|
||||
return payload
|
||||
|
||||
|
||||
def post_process(result: dict) -> Optional[dict]:
|
||||
found = _found.get()
|
||||
if found is None:
|
||||
if result.get("verdict") is not None:
|
||||
logger.info("critic_code_checks_skipped")
|
||||
return None
|
||||
_found.set(None)
|
||||
return arithmetic.apply(result, found)
|
||||
Reference in New Issue
Block a user