feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive) - A scripted answer may name who it is for, so facts written for requirements discovery are not given to a Specialist confirming a draft. - A reply that restates what the agreed requirements already say is read as an answer, not a request for changes. - With a user present, a first submission is refused until the user has been shown a draft of that Deliverable (CONFIRM_DRAFTS). Routing and reviews - Intake: a message that starts something is read first. A question is answered and starts nothing; a change to delivered work starts as a new request with a note, until Change requests are built. - Requirements and recommendations list what infrastructure code can show (conditions) apart from what it cannot (outside_code); the latter needs no Check and validation proposes nothing for it. - A requirements review, "names no products", behind its own switch (off). - Each pillar advisor reviews its pillar's checklist; the engine writes the reviews into the design recommendations. The judged checklist check went from 9-45% a pillar to 100% in one run of case 04. Evaluation - Per-case thresholds for Revisions, seconds and tokens, set from every completed run on this build. - A progress file after every agent run, so an interrupted run keeps its time and tokens. - eval/calibrate_judge.py with a 32-question set awaiting a person's labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91. Smaller - An Execution that ends with warnings writes its Deliverables, with the warnings and each status in the manifest. - CI installs the application's dependencies before linting and testing. - A load-balancer pattern in the IaC patterns Skill. Plans record nine runs of 2026-10-08. Iteration 6 is complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
**Iteration:** 6, the Target architecture graph
|
||||
**Intent:** Establish the Target architecture as one explicit graph of Elements, Relationships and Checks before any infrastructure code is written, and derive and check every later Deliverable from it, so that the outcome is understood before code exists and validation has a fixed bar.
|
||||
**Status:** steps 1 to 10 built; cases 01, 02 and 03 completed at 100% with real models on 2026-10-07 and 08 (`steps/step-11.md`); case 04, the interactive one, does not complete
|
||||
**Status:** complete. Steps 1 to 10 built; all four cases, the interactive one included, completed twice with real models on 2026-10-08 and the thresholds are set (`steps/step-11.md`)
|
||||
**This directory covers:** steps 1 to 11
|
||||
|
||||
## Order of work
|
||||
@@ -56,7 +56,7 @@ flowchart TD
|
||||
| 8 | Deployment guidance implements the graph | M | complete |
|
||||
| 9 | Validation gives a verdict per Check | M | complete |
|
||||
| 10 | Revision routing: the code or the graph | S | complete; routing unproven with a real model |
|
||||
| 11 | Evaluation: prove the loop converges | M | cases 01, 02 and 03 pass at 100%; case 04 (interactive) scored 75% once and its rerun did not finish: scripted answers are read as new requirements (`steps/step-11.md`); thresholds outstanding |
|
||||
| 11 | Evaluation: prove the loop converges | M | complete: all four cases pass, each run twice on 2026-10-08; thresholds set (`steps/step-11.md`) |
|
||||
|
||||
Size: S under a day, M a few days.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user