This commit is contained in:
@@ -0,0 +1,22 @@
|
||||
# Step 8 — Implementation Detail
|
||||
|
||||
This step has no entry in the spec's comments; this file is its record.
|
||||
|
||||
1. Added `eval/solution_eval.py`: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
|
||||
2. Added `eval/scoring.py` with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
|
||||
3. Added 3 cases in `eval/datasets/solution_cases.json`. Results are written to `eval/results/<timestamp>/` (gitignored).
|
||||
|
||||
Recorded runs:
|
||||
|
||||
| Run | Case | Score | Seconds | Revisions |
|
||||
|---|---|---|---|---|
|
||||
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
|
||||
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
|
||||
|
||||
No scored run of `case-02-high-scale-ingestion` is on record.
|
||||
|
||||
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
|
||||
|
||||
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
|
||||
|
||||
Suite at the end of this step: 226 passed.
|
||||
Reference in New Issue
Block a user