Some checks failed
validation / verify (push) Failing after 11s
A pillar's checklist item the request gives no information about was marked "not addressed", which reads as a fault in the design. It is missing information. The checklist review gains a fourth outcome, to confirm with the customer, which must carry the question to ask. The questions are written into the design recommendations and the guide, and to customer-questions.md; the evaluation's judged check counts such an item as accounted for. Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's retention, training, context and rate limits (iteration 5 step 1), and the owner's labels, which showed the judged question was wrong, not the judges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
69 lines
6.3 KiB
Markdown
69 lines
6.3 KiB
Markdown
# Plan Map
|
|
|
|
**Iteration:** 4, learning and Skills
|
|
**Intent:** Make the agent a learning, adaptable solution through Skills. Fix the cheap defects the 2026-10-06 review found, then close the loop so each Execution can improve the next through a reviewed Skill change, then let new standards, clients and Cloud providers arrive as Skills and configuration.
|
|
**Status:** steps 1 to 8, 11, 13 and 15 to 20 built; case 01 completed on them on 2026-10-07 (`../iter-06-target-architecture-graph/steps/step-11.md`); the other cases have not run, so no criterion is measured across the benchmark; step 9 complete (ADR-0004 accepted 2026-10-08), which unblocks steps 10 and 12; steps 10, 12 and 14 planned. Steps 1 to 6 are in commit `7ee49ab`; the rest are uncommitted.
|
|
**Changed by iteration 6:** `docs/adr/0005-the-target-architecture-is-an-explicit-graph.md` (accepted 2026-10-07) replaces the free-form `findings.json` of step 1 with a verdict per Check, remove the diagrammer, and add a Phase. Its plan is `../iter-06-target-architecture-graph/`, which lists the effect on each step here and should be built before steps 10 to 14.
|
|
**This directory covers:** steps 1 to 27
|
|
|
|
## Sources of truth
|
|
|
|
- Review report (2026-10-06): https://claude.ai/code/artifact/16b37169-1554-4eb7-bfcb-254315fa0518. Each step's `Report ref` names its enhancement number there. The report uses pre-iteration-3 paths.
|
|
- `iter-02-solution-architect-agent/steps/step-8.md`: the evaluation figures quoted as evidence.
|
|
- `docs/adr/0002-skills-are-the-portable-unit-of-behaviour.md`: every step must keep behaviour in Skills and vendored Skills unmodified.
|
|
- `CONTEXT.md`: "every Execution starts clean". Learning arrives as a reviewed Skill change, never as memory carried between Executions.
|
|
|
|
## Status
|
|
|
|
| Step | What | Group | Size | Status |
|
|
|---|---|---|---|---|
|
|
| 1 | Structured validator findings | A | S | replaced by iteration 6 step 9 (verdicts per Check) |
|
|
| 2 | Carry the draft through `needs_input` | A | S | complete |
|
|
| 3 | Reuse pillar advice across Revisions | A | S | built; no run has revised design recommendations, so unexercised by evaluation |
|
|
| 4 | Single source for Deliverable contracts | A | S | complete |
|
|
| 5 | Cap parallel Specialists | A | S | complete |
|
|
| 6 | Orchestrator step order | A | S | built; seen working in one stopped run, no completed evaluation |
|
|
| 7 | Capture lesson candidates | B | M | complete |
|
|
| 8 | Evaluation that can see improvement | B | M | complete: thresholds set and four judged runs done on 2026-10-08 |
|
|
| 9 | ADR: the lessons loop and overlay Skills | B | S | complete: ADR-0004 accepted 2026-10-08 |
|
|
| 10 | Lessons Skill and lesson curator | B | L | the loop is built; six lessons proposed from 21 Executions, approved by the owner and published on 2026-10-09; not yet proved by an evaluation with and without (`steps/step-10.md`) |
|
|
| 11 | IaC patterns Skill | B | M | built; in case 01 the IaC author fell from 57 requests to 11 with no Revision; other cases unmeasured |
|
|
| 12 | Overlay Skills and an organisation standards Skill | C | M | built, with an example standards Skill; case 06 passed on 2026-10-09, failed once on 2026-10-10 for a wrong verdict, and passed again after the fix, now with a check that the code follows the standards |
|
|
| 13 | Trim Specialist context | C | M | built; measured on case 01 only, together with steps 11 and 17 |
|
|
| 14 | Provider-owned contracts | C | M | deferred by its own plan until a second Cloud provider is planned (ADR-0002) |
|
|
| 15 | Group Executions by Customer case | added | S | complete |
|
|
| 16 | Cache Terraform providers between checks | added | S | complete |
|
|
| 17 | A working copy for files and for the graph | added | M | complete; in every run since 2026-10-07 |
|
|
| 18 | Provider schema lookup and one pinned provider version | added | S | built; checked against the real provider |
|
|
| 19 | A Check the code cannot meet goes back to the Target architecture | added | S | built |
|
|
| 20 | A model timeout per role | added | S | built |
|
|
| 21 | A Deliverable stands while what it was written from is unchanged | added | S | complete; the description was kept on Check-only Revisions in runs of 2026-10-07 to 09 |
|
|
| 22 | Validation reports every gap in one pass | added | S | instructions changed; with conditions listed, case 02 ran with 102 Checks and no Revision |
|
|
| 23 | Validation recorded a few verdicts at a time | added | S | built; in case 02 a validator run cut short carried on and submitted in 68 seconds, with no timeout |
|
|
| 24 | The design checked against the provider's schema, in code | added | S | built; measured on two passed designs and the known faults; raised nothing in case 02, which had no unimplementable Check |
|
|
| 25 | An Execution that ends with warnings keeps its files | added | S | complete |
|
|
| 26 | Each pillar's checklist is reviewed and the review is in the Deliverable | added | S | built; a judged run of case 04 scored the checklist check at 100% on every pillar |
|
|
| 27 | A user sees a Deliverable before it is first submitted | added | S | built; case 04 passed the check on 2026-10-09, with the rule not needing to refuse |
|
|
| 28 | Housekeeping: files written as an Execution goes, four more code patterns | none | S | built 2026-10-10 (`steps/step-28.md`) |
|
|
| 29 | What the request does not say is a question for the customer | none | M | built 2026-10-10: a fourth checklist outcome, a questions report, and the judged check counts it; not run (`steps/step-29.md`) |
|
|
|
|
Size: S under a day, M a few days, L a week or more. Group A is cheap fixes that also create signal. Group B is the learning loop. Group C is adaptability.
|
|
|
|
## Dependency graph
|
|
|
|
- Steps 1 to 6: no dependencies; any order
|
|
- Step 7: depends on step 1
|
|
- Step 8: no dependencies
|
|
- Step 9: no dependencies
|
|
- Step 10: depends on step 7, step 8, step 9
|
|
- Step 11: depends on step 8
|
|
- Step 12: depends on step 9
|
|
- Step 13: depends on step 8
|
|
- Step 14: depends on step 4
|
|
- Step 15: no dependencies (added on 2026-10-06 at the owner's request)
|
|
- Step 16: no dependencies (added on 2026-10-07 at the owner's request)
|
|
- Steps 17 to 20: no dependencies (added on 2026-10-07 at the owner's request, from the causes of the 60 minute run recorded in `../iter-06-target-architecture-graph/steps/`)
|
|
- Step 11 uses step 18's pinned provider version; step 19 uses step 18's lookup to find the Checks it reports
|
|
|
|
Step 8 comes before steps 10, 11 and 13 so each of them can be shown to help: same quality, fewer Revisions, seconds or tokens.
|