ALZ-F02 Assessment: critic code checks (ADR 0011), represents edges (ADR 0012)
Some checks failed
check agents / check (push) Failing after 12s
Some checks failed
check agents / check (push) Failing after 12s
- Critic 0.2.0 / 0.2.1: K-04 (risk), K-05 (confidence), K-06 (TCO), K-07 (routing) and K-11 (schema and 7R enum) run in code in the critic's hooks (app/arithmetic.py, app/business_logic.py); post_process sets those checks, the rejected dispositions, the sign-off queue and the verdict. The model keeps the judgement checks; a different reading of a code check is kept in unverified_claims. K-07 checks the disposition agent's own queue. Tests on the records of run exec_4d88027a4d26, whose model sums were wrong. - A represents edge is not a dependency (ADR 0012): risk scorer 0.1.1 and disposition 0.1.2 leave it out of dependents and dependencies; the disposition and critic prompts carry the risk scorer's rule. Run exec_1f518b04d6e4: APP-001's risk 1.00 -> 0.00, out of the tail. - The scaffold declares the critic's hooks, copies app/ in its Dockerfile, and keeps its app/ and tests/ when regenerating. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -17,6 +17,8 @@ The source definitions name scoring and writing operations as tools. No server p
|
||||
- **The calculations are not tools.** Profiling, risk scoring, the 7R rule set, confidence, TCO and licensing are fixed rules in each agent's `prompt.md`, written in its output schema. ALZ-ASM-06 recalculates every number from the same inputs (K-04, K-05 and K-06).
|
||||
- **No write tools.** The workflow stores each step's reply.
|
||||
|
||||
> Refined by [ADR 0011](0011.md): the critic's arithmetic, routing and schema checks run in code.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The arithmetic rests on the model following the rules,** with the critic as the check. That critic is a different model (ADR 0005), and the human approval at its step is the final gate.
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# 0011. The critic's arithmetic, routing and schema checks run in code; the model judges the rest
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "yes" (do the arithmetic in code; leave the judgement to the model) |
|
||||
| Requirements changed | ALZ-ASM-06: K-04, K-05, K-06, K-07, K-11 and the verdict are set in code. Refines [ADR 0001](0001.md): the critic is generated from its story but carries hooks, as the intake does ([ADR 0008](0008.md)) |
|
||||
|
||||
## Context
|
||||
|
||||
- ADR 0001 keeps the scoring rules as prompt rules, applied by the producing agents and recalculated by the critic.
|
||||
- Run `exec_1779863642d6` showed that the critic had none of the rules it recalculates with; critic 0.1.6 copies them
|
||||
into its prompt. Run `exec_4d88027a4d26` then showed the critic applying the right rules and getting the sums wrong:
|
||||
- K-04: it expected a risk score of 0.70; with one factor available (fan-in 1.0, weight 0.25), the renormalised
|
||||
score is 1.00, as the risk scorer recorded;
|
||||
- K-05: it listed 0.90 - 0.10 - 0.20 - 0.10 and called it 0.60; it is 0.50, as the disposition agent recorded.
|
||||
- The cluster's other model, `gpt-4o`, is weaker, and `gpt-5.6-sol` runs the executors (K-12, ADR 0010).
|
||||
- These checks are sums, lookups and set equality over the producers' own records. They need no judgement.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **Code checks, in the critic's hooks** (`app/arithmetic.py`, `app/business_logic.py`):
|
||||
- **K-04:** each risk score from its factors and the fixed weights, renormalised over the available ones; each
|
||||
contribution; `partial`; `missing_factors`; the 0.70 tail rule; `high_risk` and the high-risk tail.
|
||||
- **K-05:** each confidence from its workings: the 0.90 base, each adjustment's prescribed delta (P2 at -0.10 a
|
||||
fact, -0.20 at most), the C1 cap for repurchase and relocate, the result, `signoff_required`.
|
||||
- **K-06:** each line item's rate id in the rate card at its annual rate; current, target and delta sums; a missing
|
||||
cost named; `indicative`.
|
||||
- **K-07:** the sign-off queue is the dispositions below 0.80 confidence, less the rejected ones.
|
||||
- **K-11:** the input contract validated every record on arrival; the 7R enum is checked again.
|
||||
2. **`pre_process`** works these out before the model runs and shows them to it as `checked_in_code`.
|
||||
**`post_process`** sets those checks, the rejected dispositions (K-05, K-06, K-11), the sign-off queue and the
|
||||
verdict: `fail` if any check fails or any check other than K-10 is unverified ([ADR 0006](0006.md)).
|
||||
3. **The model keeps the judgement:** K-01 to K-03, K-08 to K-10 and K-12, and, inside the code checks, whether a
|
||||
factor's value or an adjustment was due. Where its reading of a code check differs, the reading is kept in
|
||||
`unverified_claims`, so the Engagement Lead sees it.
|
||||
4. **When the planner refused to start** (`preconditions.failed`), the hooks change nothing: the model returns `fail`
|
||||
on K-01 alone.
|
||||
5. **The scaffold keeps the critic's `app/` and `tests/`** when it regenerates, and declares the hooks.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The verdict follows from the checks in code,** so a sum the model gets wrong no longer fails a run, and one the
|
||||
producer gets wrong always does.
|
||||
- **Whether a rule applied is still a model judgement.** The code checks that P2 is -0.10 a fact, not that the facts
|
||||
were unknown, and that a rate id exists at its rate, not that it is the right equivalent.
|
||||
- **The critic is no longer purely declarative.** Its code is small, standard library only, and tested on the
|
||||
records of a live run (`tests/test_arithmetic.py`).
|
||||
@@ -0,0 +1,34 @@
|
||||
# 0012. A `represents` edge is not a dependency
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Status | Accepted |
|
||||
| Date | 2026-11-05 |
|
||||
| Decided by | Product owner, 2026-11-05: "lets solve these first" (the run's faults are fixed before it is repeated); the rule follows Discovery's F01 ADR 0019 |
|
||||
| Requirements changed | ALZ-ASM-03 and ALZ-ASM-04: the Application Dependents rule leaves out `represents` edges. ALZ-ASM-04 and ALZ-ASM-06 use ALZ-ASM-03's rule |
|
||||
|
||||
## Context
|
||||
|
||||
- The Application Dependents rule (ALZ-ASM-03) lifts every ALZ-DSC-06 edge to applications.
|
||||
- Since F01 ADR 0019, Discovery also writes a `represents` edge for each CMDB record it matched to an inventory asset
|
||||
(CMDB CI → resource). It records that the two are the same asset; it isn't a dependency.
|
||||
- In run `exec_2cd7a953335b`, the eight `represents` edges of APP-001 (E-008 to E-015) were counted as eight dependents:
|
||||
- the risk scorer set fan-in to 1.0, the only factor available, so the score was 1.00 and APP-001 was high-risk;
|
||||
- the disposition agent ruled out retire for eight dependents.
|
||||
The critic model rejected the disposition under K-02 for citing them. Its other edges (E-001, E-002, E-005, E-007)
|
||||
run from APP-001's own assets outwards: they are dependencies, not dependents.
|
||||
- ALZ-ASM-04's prompt had no rule of its own for lifting dependents.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **`dependency(e) = e.type != 'represents'`** in ALZ-ASM-03's Application Dependents rule: a `represents` edge is
|
||||
left out of both an application's dependents and its dependencies.
|
||||
2. **ALZ-ASM-04 and ALZ-ASM-06 carry ALZ-ASM-03's rule,** copied into their prompts by the scaffold, so all three lift
|
||||
dependents the same way.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **APP-001 has no dependents** in the current Discovery package: fan-in 0.0, so its risk score falls from 1.00 to
|
||||
0.00 on the one factor available, and it leaves the high-risk tail. Retire is still ruled out by its assets'
|
||||
`retire_constraint`.
|
||||
- **A future Discovery edge type** that isn't a dependency needs the same treatment here.
|
||||
@@ -23,5 +23,7 @@ Each agent's `docs/adr/README.md` lists the records that apply to its own story.
|
||||
| [0008](0008.md) | Assessment reads one Discovery package, checked in code by an intake step before any agent | Accepted | ASM-01, intake (new) |
|
||||
| [0009](0009.md) | The rate card is a run input, its JSON as text; Assessment has no upload step | Accepted | ASM-01, 04, 06 |
|
||||
| [0010](0010.md) | Model independence (K-12) is checked when the workflow is built, not judged by the critic | Accepted | ASM-06 |
|
||||
| [0011](0011.md) | The critic's arithmetic, routing and schema checks run in code; the model judges the rest | Accepted | ASM-06 |
|
||||
| [0012](0012.md) | A `represents` edge is not a dependency | Accepted | ASM-03, 04, 06 |
|
||||
|
||||
New records copy [template.md](template.md) and take the next number.
|
||||
|
||||
Reference in New Issue
Block a user