Some checks failed
check agents / check (push) Failing after 14s
- Report (cp1-assessment-report-agent 0.2.0, step s7): a model writes the Assessment report from facts worked out in code (register, gaps, scorecard, rule traces, checks, and what each application is from the Discovery package); code sets the verdict and renders the pages. The disposition_signoff review point moves from the critic to the report. - Package (cp1-assessment-package-agent 0.1.1, step s8): after sign-off, every step's output in one package, with a register, a gaps register, the sign-off record, lineage and a sha256 per part, in code. Page step s9. - scripts/kab_register_agents.py: registers deployed agents in KAB's agent catalog (an in-place workflow update doesn't). - Runs exec_1b1257380df7 (014) and exec_374d5d577077 (015) completed on crucible-kab-dev: critic pass, report figures all in its facts, package complete with every digest recomputing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
104 lines
7.8 KiB
Markdown
104 lines
7.8 KiB
Markdown
# Assessment outputs: proposal
|
||
|
||
| | |
|
||
|---|---|
|
||
| Status | Decisions 1, 2, 4 and 6 accepted 2026-11-08 (ADR 0014). Decision 3 decided 2026-11-08: a model writes the report from the data, before sign-off (ADR 0015). Decision 5: recorded in the ADR, not on the board |
|
||
| Asked by | Product owner, 2026-11-06: "propose outputs for the assessment workflow as currently there are not any specific" |
|
||
| Becomes | An ADR (0014) once decided; Discovery's equivalent is F01 ADR 0020 (package) and ADR 0012 (report page) |
|
||
|
||
## Why
|
||
|
||
The Feature promises an **"Assessment deliverable"** besides the per-application records, but no step makes one:
|
||
|
||
- The workflow ends at the critic's review point (`s6`, `disposition_signoff`). What it leaves is seven separate step
|
||
artifacts (`<exec>_s0` … `_s6`), each in its own shape, plus a review decision that lives only in KAB's activity log.
|
||
- A person has nothing to read: the Engagement Lead signs off from the critic's raw JSON.
|
||
- The next Features have nothing to select. Strategy (F03) needs "signed-off dispositions and risk scores";
|
||
Business Case (F04) needs the TCO models, dispositions and Discovery's cost facts; Target Architecture (F05) and
|
||
Wave Planning (F06) need signed-off dispositions. Each would have to find and join seven artifacts by hand, which is
|
||
the problem Discovery's package solved for F02 (F01 ADR 0020).
|
||
- What Assessment *couldn't* do is never stated in one place. In run `exec_49e355141382` APP-001's risk is a partial
|
||
score (end of life, data and change factors missing) and its confidence 0.50. Nobody is told which Discovery data
|
||
would raise it.
|
||
|
||
## Proposal: three outputs, after the sign-off review point
|
||
|
||
All three run only after the Engagement Lead continues the run at `disposition_signoff`, so they exist only for
|
||
assessments a person let through (as in Discovery). The critic's review stops being the final step.
|
||
|
||
```text
|
||
s0 intake → s1 planner → s2 profiler ┐
|
||
s3 risk ├→ s5 disposition → s6 critic ⏸ disposition_signoff
|
||
s4 security ┘ ├→ s7 report (model + code) → s8 page
|
||
└→ s9 package (code only)
|
||
```
|
||
|
||
### Output 1: the Assessment package (machine hand-off, code only)
|
||
|
||
New agent `cp1-assessment-package-agent`, no model and no tools, like `cp1-discovery-package-agent`. Its output is
|
||
**one artifact** that F03 to F06 select, with every step's fields **copied unchanged** into named parts:
|
||
|
||
| Part | From | Holds |
|
||
|---|---|---|
|
||
| `manifest` | all | engagement, run, execution id; **lineage**: Discovery execution id, the Discovery package's sha256 (as the intake verified it), `rate_card_id` and its sha256; each agent's version; `critic_model` and `executor_models`; critic verdict; a sha256 and item count per part; `complete`; `problems` |
|
||
| `register` | s1–s6, joined in code | **one row per application**, the view the next Features read: disposition, confidence, `signoff_required`, sign-off status, risk score, `partial` and `missing_factors`, high-risk tail, current and target annual cost, posture summary (advisory), `evidence_ref`, and whether the critic rejected it |
|
||
| `plan` | s1 | preconditions, applications, application source, `in_cloud_estate`, coverage gaps |
|
||
| `profiles` | s2 | readiness profiles and gaps |
|
||
| `risk` | s3 | scores with factors, high-risk tail |
|
||
| `security` | s4 | status, posture score, controls, findings, pillars (always advisory) |
|
||
| `dispositions` | s5 | dispositions with rule trace, confidence workings, TCO model, licensing and scoring notes |
|
||
| `critic` | s6 | checks, rejected dispositions, sign-off queue, unverified claims, summary |
|
||
| `gaps` | s1–s5, in code | **a data-improvement register**: per application, each missing input (risk factor, TCO input, readiness signal, telemetry) with the Discovery source or field that would fill it, and what it costs today (for example "−0.10 confidence: P1"). Discovery's source gaps are carried through |
|
||
| `scorecard` | in code | dispositions per 7R category; share above and below 0.80; share of partial risk scores; rejected count; advisory findings by severity; TCO coverage (applications with both current and target cost) |
|
||
|
||
`complete` is true only when every required part is present, the critic's verdict is `pass` and the run was
|
||
continued at the review point. Each part's sha256 is computed the same way as Discovery's (canonical JSON), so F03 can
|
||
verify it the way F02's intake verifies Discovery's package (ADR 0008).
|
||
|
||
### Output 2: the Assessment report (for people)
|
||
|
||
> **Correction, 2026-11-08:** `cp1-discovery-report-agent` calls no model: its `app/report.py` renders the whole
|
||
> report in code. The precedent given below for a model writing prose was wrong, so decision 3 is open again.
|
||
|
||
New agent `cp1-assessment-report-agent` (model plus hooks, like `cp1-discovery-report-agent`), then a
|
||
`format_for_ui` page step. **Every figure is filled in by code** from the step outputs (ADR 0011); the model writes
|
||
only the prose around them. Sections:
|
||
|
||
1. **Summary**: applications assessed, the verdict, how many need sign-off and why.
|
||
2. **Dispositions**: one table row per application (7R, confidence, rule that matched, sign-off), and the 7R mix.
|
||
3. **Sign-off queue**: each low-confidence or interim-rule item with the deductions that lowered it.
|
||
4. **Risk**: scores, the high-risk tail, and which scores are partial.
|
||
5. **Cost**: current against target annual cost per application, with indicative and missing inputs named.
|
||
6. **Security posture (advisory)**: posture score and the top findings; says plainly they changed no disposition.
|
||
7. **What would raise confidence**: the `gaps` register in words, ordered by effect.
|
||
8. **Assurance**: the critic's checks, what is checked in code, what is unverified (K-10) and why.
|
||
9. **Lineage**: the Discovery run, the rate card, agent versions and models.
|
||
|
||
### Output 3: the sign-off record
|
||
|
||
KAB keeps a review decision in its activity log, not on an artifact (see `.agents/CONTEXT.md`), so "signed-off
|
||
dispositions" can't be read from the run today. Proposed: **continuing the run at `disposition_signoff` signs off the
|
||
whole queue as it stands.** The package records each application as `signed_off` (it was in the queue) or
|
||
`not_required`. The next workflow records the approval as a run input, `assessment_approval_ref`, the same way F02 takes
|
||
`discovery_approval_ref` (ADR 0002). Overriding single dispositions (accepting one, changing another) is left for later.
|
||
|
||
## Decisions for the product owner
|
||
|
||
1. **The package** (Output 1): build it as proposed? *Recommended*: it's what F03 to F06 need, and it repeats a
|
||
pattern that already works for Discovery.
|
||
2. **The `register` and `gaps` parts**: include them? *Recommended*. They're the only parts that aren't copies; both
|
||
are worked out in code, so the figures can't drift.
|
||
3. **The report** (Output 2): a model writing prose around code-filled figures, or **code only** (a fixed template, no
|
||
model, cheaper and identical every run, but less readable)? *Recommended*: model plus code, as Discovery does.
|
||
4. **Sign-off** (Output 3): whole-queue sign-off on continue now, per-application overrides later? *Recommended*.
|
||
The alternative is a sign-off input file (one decision per application) on a rerun, which needs a design of its own.
|
||
5. **The board**: the package and report agents are new agents. Add them as User Stories under ALZ-F02 (for example
|
||
ALZ-ASM-07 and ALZ-ASM-08), or record them in this repository only, in an ADR?
|
||
6. **Order**: package first (unblocks F03), then report, then sign-off? *Recommended*.
|
||
|
||
## Not in this proposal
|
||
|
||
- How F03 reads the package (F03 has no agents yet).
|
||
- Telemetry, end-of-life or cost sources that would fill the gaps (Discovery value plan, phases 3 to 7).
|
||
- Changes to the six existing agents' outputs: everything above is assembled from what they already produce.
|