1.4 KiB
1.4 KiB
Step 8 — Implementation Detail
This step has no entry in the spec's comments; this file is its record.
- Added
eval/solution_eval.py: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role. - Added
eval/scoring.pywith 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates. - Added 3 cases in
eval/datasets/solution_cases.json. Results are written toeval/results/<timestamp>/(gitignored).
Recorded runs:
| Run | Case | Score | Seconds | Revisions |
|---|---|---|---|---|
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
No scored run of case-02-high-scale-ingestion is on record.
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
Suite at the end of this step: 226 passed.