F02: Assessment report and package (ADR 0014, 0015); register agents in KAB
Some checks failed
check agents / check (push) Failing after 14s

- Report (cp1-assessment-report-agent 0.2.0, step s7): a model writes the
  Assessment report from facts worked out in code (register, gaps, scorecard,
  rule traces, checks, and what each application is from the Discovery
  package); code sets the verdict and renders the pages. The
  disposition_signoff review point moves from the critic to the report.
- Package (cp1-assessment-package-agent 0.1.1, step s8): after sign-off, every
  step's output in one package, with a register, a gaps register, the
  sign-off record, lineage and a sha256 per part, in code. Page step s9.
- scripts/kab_register_agents.py: registers deployed agents in KAB's agent
  catalog (an in-place workflow update doesn't).
- Runs exec_1b1257380df7 (014) and exec_374d5d577077 (015) completed on
  crucible-kab-dev: critic pass, report figures all in its facts, package
  complete with every digest recomputing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-08 20:19:10 +00:00
parent 3b3ca64de0
commit 66c32745aa
58 changed files with 7106 additions and 26 deletions

View File

@@ -0,0 +1,52 @@
# 0014. One Assessment package after sign-off, with a register, a gaps register and the sign-off record, built in code
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-08 |
| Decided by | Product owner, 2026-11-08: "yes lets go with the receommations", on [../assessment-outputs-proposal.md](../assessment-outputs-proposal.md) (decisions 1, 2, 4 and 6). Decision 5 had no recommendation: the new agent is recorded here, not on the board, until the product owner says otherwise |
| Requirements changed | ALZ-F02 Outputs ("Assessment deliverable"): now the Assessment package. ALZ-ASM-06's `disposition_signoff` is no longer the last step. New step `s8` (first built as `s7`; moved behind the report by ADR 0015), agent `cp1-assessment-package-agent` (no story) |
## Context
- The Feature promises an "Assessment deliverable", but the workflow ended at the critic's review point. It left seven
step artifacts, each in its own shape, and a review decision held only in KAB's activity log.
- Strategy (F03) needs "signed-off dispositions and risk scores"; the Business Case (F04) the TCO models; Target
Architecture (F05) and Wave Planning (F06) the signed-off dispositions. Each would have had to find and join seven
artifacts, the problem Discovery's package solved for Assessment (F01 ADR 0020).
- What Assessment had to do without was never stated in one place. In run `exec_49e355141382`, APP-001's risk score is
partial, and its confidence lost 0.40 to missing telemetry, unknown support status and the partial score.
## Decision
1. **A package step, `s8`, answered in code** (`cp1-assessment-package-agent`, `app/package.py`, no model, no tools):
- It depends on every agent step, so it runs only after the run is continued at `disposition_signoff` (on the
report step since ADR 0015, which also gives the package a `report` part).
- Every step's output fields are **copied unchanged** into parts named by meaning: `discovery` (the intake's
verification), `plan`, `profiles`, `risk`, `security`, `dispositions`, `critic`.
- Its `manifest` holds the lineage (the Discovery run, its package's artifact id and the intake's verification; the
rate card's id and sha256, not its content; the models; each agent's version), the sign-off record, a `sha256`
and item count per part, `complete`, `problems` and a scorecard.
- Each `sha256` is over canonical JSON, computed exactly as Discovery's package step does, so F03 can verify the
package the way the intake verifies Discovery's (ADR 0008).
- `complete` only when every copied part is present, the critic passed and the intake verified the Discovery package.
2. **Two parts are worked out in code, never by a model:**
- `register`: one row per planned application, joining disposition, confidence, matched rule, sign-off, risk,
dependents, readiness, cost, advisory findings and the primary evidence. It is what the later Features read.
- `gaps`: per application, each thing Assessment had to do without (a missing risk factor, an unknown readiness
signal, a confidence deduction, a missing TCO input, a coverage or profile gap), its effect, the producer's basis,
and the Discovery source or field that would fill it. Discovery's source gaps are carried through.
3. **Sign-off: continuing at `disposition_signoff` signs off the queue as it stands.** Each register row's `signoff` is
`signed_off` (in the critic's queue), `not_required`, `rejected`, `not_disposed`, or `not_signed_off` when the
verdict isn't `pass` (a fail verdict is never approved). The next workflow records the approval as its run input
`assessment_approval_ref`, as Assessment takes `discovery_approval_ref` (ADR 0002). Overriding single dispositions
is left for later.
## Consequences
- **The later Features select one artifact,** the package step's (`<exec>_step_<N>`, as KAB names it), and can
check every part before trusting it.
- **The Assessment report** (proposal decision 3) is ADR 0015.
- **The reviewer's identity and time are not in the package:** KAB keeps them in the activity log
(`kab_run_workflow.py status` shows `decided_by`). The next workflow's `assessment_approval_ref` carries them.
- **A run continued with a fail verdict** still gets a package, marked incomplete and signed off for nothing.

View File

@@ -0,0 +1,57 @@
# 0015. A model writes the Assessment report from facts made in code, and the Engagement Lead signs off from it
| | |
|---|---|
| Status | Accepted |
| Date | 2026-11-08 |
| Decided by | Product owner, 2026-11-08: "in terms of the report the model should genereate it based on the data collected and should get before sign off" (proposal decision 3, reopened after the correction that Discovery's report calls no model) |
| Requirements changed | ALZ-ASM-06: `disposition_signoff` moves from the critic's step to the report's. New step `s7`, agent `cp1-assessment-report-agent` (no story); the package becomes `s8` (ADR 0014); new page step `s9` |
## Context
- The Engagement Lead signed off from the critic's raw JSON: nothing readable showed the dispositions, why they were
chosen, what lowered their confidence, or what signing off commits to.
- The proposal's precedent for a model writing the report was wrong: Discovery's report (`cp1-discovery-report-agent`)
is rendered entirely in code. The product owner chose a model anyway, writing from the data collected, before
sign-off.
- A model writing figures is the failure ADR 0011 removed from the critic (it summed 0.90 − 0.10 − 0.20 − 0.10 to 0.60).
## Decision
1. **The report step, `s7`, after the critic, carries the `disposition_signoff` review point.** The critic's step no
longer pauses. The review shows the report as a page marked as a draft for sign-off.
2. **Code works out the facts; the model writes from them.** `pre_process` (`app/facts.py`) replaces the step outputs
with facts: the register, gaps and scorecard from the package's own module (copied; a test keeps the copies
identical), each disposition's rule trace and confidence workings, the critic's checks, the top ten security
findings and the lineage. The model writes the report in Markdown, in the sections `prompt.md` sets, and is told
to take every figure from the facts and to calculate none.
3. **Code sets what must not be the model's:** `post_process` sets the verdict, ids and sign-off queue from the facts,
and renders the HTML (`app/markdown.py`, which escapes the model's text before adding markup): `report_html`, the
draft shown at the review point, and `final_html`.
4. **After the review point:** the package (`s8`) holds the report's Markdown as its `report` part, and a
`format_for_ui` step (`s9`) makes `final_html` the run's page, as Discovery does (F01 ADR 0012).
## Consequences
- **The Engagement Lead reads a report, not JSON,** at the point where continuing signs off the queue.
- **The model can still misquote a figure.** The facts are correct, and the prompt forbids new figures, but nothing
checks the prose against them. If a run shows a misquote, the next step is a check in `post_process` that every
number in the report appears in the facts.
- **Each run costs one more model call,** about 13k characters in.
- **The report is not checked by the critic:** it is written after it, from the critic's own results.
## Addendum, 2026-11-08: the report says what each application is
Asked for by the product owner after run `exec_1b1257380df7`: "it doesnt breakd down what the application is within
the report. This is necessary to understand what it has identified".
- **Why it was missing:** the Assessment steps' outputs name an application's assets by id only, and the report was
written from them alone.
- **Decision:** the report step also receives `discovery_documents` (the parse step's text, `p1.parsed_content`).
Code (`app/discovery.py`) reads the Discovery package there and adds to the facts, per application: how Discovery
identified it and why it asked for a review, its business service and owner, every asset (name, resource type,
environment, location, criticality, services, size, state, CMDB class and support group, data class, support and
cost facts), its data stores, and its dependencies without `represents` edges (ADR 0012). The model writes a new
section, `## The application`, from them, and never receives the package itself.
- **Consequences:** the facts grow from about 13k to 22k characters for one application. The package (ADR 0014) is
unchanged: it already names the Discovery package it rests on, which holds these records.

View File

@@ -26,5 +26,7 @@ Each agent's `docs/adr/README.md` lists the records that apply to its own story.
| [0011](0011.md) | The critic's arithmetic, routing and schema checks run in code; the model judges the rest | Accepted | ASM-06 |
| [0012](0012.md) | A `represents` edge is not a dependency | Accepted | ASM-03, 04, 06 |
| [0013](0013.md) | K-02's lookup runs in code; whether the evidence supports the basis stays the model's | Accepted | ASM-06 |
| [0014](0014.md) | One Assessment package after sign-off, with a register, a gaps register and the sign-off record, in code | Accepted | ASM-06, package (new) |
| [0015](0015.md) | A model writes the Assessment report from facts made in code; the Engagement Lead signs off from it | Accepted | ASM-06, report (new) |
New records copy [template.md](template.md) and take the next number.

View File

@@ -0,0 +1,103 @@
# Assessment outputs: proposal
| | |
|---|---|
| Status | Decisions 1, 2, 4 and 6 accepted 2026-11-08 (ADR 0014). Decision 3 decided 2026-11-08: a model writes the report from the data, before sign-off (ADR 0015). Decision 5: recorded in the ADR, not on the board |
| Asked by | Product owner, 2026-11-06: "propose outputs for the assessment workflow as currently there are not any specific" |
| Becomes | An ADR (0014) once decided; Discovery's equivalent is F01 ADR 0020 (package) and ADR 0012 (report page) |
## Why
The Feature promises an **"Assessment deliverable"** besides the per-application records, but no step makes one:
- The workflow ends at the critic's review point (`s6`, `disposition_signoff`). What it leaves is seven separate step
artifacts (`<exec>_s0` … `_s6`), each in its own shape, plus a review decision that lives only in KAB's activity log.
- A person has nothing to read: the Engagement Lead signs off from the critic's raw JSON.
- The next Features have nothing to select. Strategy (F03) needs "signed-off dispositions and risk scores";
Business Case (F04) needs the TCO models, dispositions and Discovery's cost facts; Target Architecture (F05) and
Wave Planning (F06) need signed-off dispositions. Each would have to find and join seven artifacts by hand, which is
the problem Discovery's package solved for F02 (F01 ADR 0020).
- What Assessment *couldn't* do is never stated in one place. In run `exec_49e355141382` APP-001's risk is a partial
score (end of life, data and change factors missing) and its confidence 0.50. Nobody is told which Discovery data
would raise it.
## Proposal: three outputs, after the sign-off review point
All three run only after the Engagement Lead continues the run at `disposition_signoff`, so they exist only for
assessments a person let through (as in Discovery). The critic's review stops being the final step.
```text
s0 intake → s1 planner → s2 profiler ┐
s3 risk ├→ s5 disposition → s6 critic ⏸ disposition_signoff
s4 security ┘ ├→ s7 report (model + code) → s8 page
└→ s9 package (code only)
```
### Output 1: the Assessment package (machine hand-off, code only)
New agent `cp1-assessment-package-agent`, no model and no tools, like `cp1-discovery-package-agent`. Its output is
**one artifact** that F03 to F06 select, with every step's fields **copied unchanged** into named parts:
| Part | From | Holds |
|---|---|---|
| `manifest` | all | engagement, run, execution id; **lineage**: Discovery execution id, the Discovery package's sha256 (as the intake verified it), `rate_card_id` and its sha256; each agent's version; `critic_model` and `executor_models`; critic verdict; a sha256 and item count per part; `complete`; `problems` |
| `register` | s1–s6, joined in code | **one row per application**, the view the next Features read: disposition, confidence, `signoff_required`, sign-off status, risk score, `partial` and `missing_factors`, high-risk tail, current and target annual cost, posture summary (advisory), `evidence_ref`, and whether the critic rejected it |
| `plan` | s1 | preconditions, applications, application source, `in_cloud_estate`, coverage gaps |
| `profiles` | s2 | readiness profiles and gaps |
| `risk` | s3 | scores with factors, high-risk tail |
| `security` | s4 | status, posture score, controls, findings, pillars (always advisory) |
| `dispositions` | s5 | dispositions with rule trace, confidence workings, TCO model, licensing and scoring notes |
| `critic` | s6 | checks, rejected dispositions, sign-off queue, unverified claims, summary |
| `gaps` | s1–s5, in code | **a data-improvement register**: per application, each missing input (risk factor, TCO input, readiness signal, telemetry) with the Discovery source or field that would fill it, and what it costs today (for example "−0.10 confidence: P1"). Discovery's source gaps are carried through |
| `scorecard` | in code | dispositions per 7R category; share above and below 0.80; share of partial risk scores; rejected count; advisory findings by severity; TCO coverage (applications with both current and target cost) |
`complete` is true only when every required part is present, the critic's verdict is `pass` and the run was
continued at the review point. Each part's sha256 is computed the same way as Discovery's (canonical JSON), so F03 can
verify it the way F02's intake verifies Discovery's package (ADR 0008).
### Output 2: the Assessment report (for people)
> **Correction, 2026-11-08:** `cp1-discovery-report-agent` calls no model: its `app/report.py` renders the whole
> report in code. The precedent given below for a model writing prose was wrong, so decision 3 is open again.
New agent `cp1-assessment-report-agent` (model plus hooks, like `cp1-discovery-report-agent`), then a
`format_for_ui` page step. **Every figure is filled in by code** from the step outputs (ADR 0011); the model writes
only the prose around them. Sections:
1. **Summary**: applications assessed, the verdict, how many need sign-off and why.
2. **Dispositions**: one table row per application (7R, confidence, rule that matched, sign-off), and the 7R mix.
3. **Sign-off queue**: each low-confidence or interim-rule item with the deductions that lowered it.
4. **Risk**: scores, the high-risk tail, and which scores are partial.
5. **Cost**: current against target annual cost per application, with indicative and missing inputs named.
6. **Security posture (advisory)**: posture score and the top findings; says plainly they changed no disposition.
7. **What would raise confidence**: the `gaps` register in words, ordered by effect.
8. **Assurance**: the critic's checks, what is checked in code, what is unverified (K-10) and why.
9. **Lineage**: the Discovery run, the rate card, agent versions and models.
### Output 3: the sign-off record
KAB keeps a review decision in its activity log, not on an artifact (see `.agents/CONTEXT.md`), so "signed-off
dispositions" can't be read from the run today. Proposed: **continuing the run at `disposition_signoff` signs off the
whole queue as it stands.** The package records each application as `signed_off` (it was in the queue) or
`not_required`. The next workflow records the approval as a run input, `assessment_approval_ref`, the same way F02 takes
`discovery_approval_ref` (ADR 0002). Overriding single dispositions (accepting one, changing another) is left for later.
## Decisions for the product owner
1. **The package** (Output 1): build it as proposed? *Recommended*: it's what F03 to F06 need, and it repeats a
pattern that already works for Discovery.
2. **The `register` and `gaps` parts**: include them? *Recommended*. They're the only parts that aren't copies; both
are worked out in code, so the figures can't drift.
3. **The report** (Output 2): a model writing prose around code-filled figures, or **code only** (a fixed template, no
model, cheaper and identical every run, but less readable)? *Recommended*: model plus code, as Discovery does.
4. **Sign-off** (Output 3): whole-queue sign-off on continue now, per-application overrides later? *Recommended*.
The alternative is a sign-off input file (one decision per application) on a rerun, which needs a design of its own.
5. **The board**: the package and report agents are new agents. Add them as User Stories under ALZ-F02 (for example
ALZ-ASM-07 and ALZ-ASM-08), or record them in this repository only, in an ADR?
6. **Order**: package first (unblocks F03), then report, then sign-off? *Recommended*.
## Not in this proposal
- How F03 reads the package (F03 has no agents yet).
- Telemetry, end-of-life or cost sources that would fill the gaps (Discovery value plan, phases 3 to 7).
- Changes to the six existing agents' outputs: everything above is assembled from what they already produce.