latest refactoring
Some checks failed
validation / verify (push) Failing after 10s

This commit is contained in:
2026-10-06 20:22:37 +01:00
parent e94b2493e7
commit 7ee49ab352
414 changed files with 15768 additions and 3896 deletions

View File

@@ -0,0 +1,22 @@
# Step 8 — Implementation Detail
This step has no entry in the spec's comments; this file is its record.
1. Added `eval/solution_eval.py`: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
2. Added `eval/scoring.py` with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
3. Added 3 cases in `eval/datasets/solution_cases.json`. Results are written to `eval/results/<timestamp>/` (gitignored).
Recorded runs:
| Run | Case | Score | Seconds | Revisions |
|---|---|---|---|---|
| 2026-09-16 10:57Z | case-01-event-driven-http | 1.0 | 2,932 | design_recommendations × 2 |
| 2026-09-17 08:44Z | case-03-internal-web-app | 1.0 | 2,623 | deployment_guidance × 3 |
No scored run of `case-02-high-scale-ingestion` is on record.
Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.
All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.
Suite at the end of this step: 226 passed.