Files
gcp_solution_architecture_a…/.agents/plans/iter-02-solution-architect-agent/steps/step-8.md
Jonathan Boniface 7ee49ab352
Some checks failed
validation / verify (push) Failing after 10s
latest refactoring
2026-10-06 20:22:37 +01:00

1.4 KiB
Raw Blame History

Step 8 — Implementation Detail

This step has no entry in the spec's comments; this file is its record.

  1. Added eval/solution_eval.py: runs each case as a pre-approved Execution through the full assembly with real models, counting agent runs, outcomes, seconds and tokens per role.
  2. Added eval/scoring.py with 8 checks: completed, guide follows the template, required products selected, alternatives listed, documentation cited, requirements name no products, diagram is valid Mermaid, Terraform validates.
  3. Added 3 cases in eval/datasets/solution_cases.json. Results are written to eval/results/<timestamp>/ (gitignored).

Recorded runs:

Run Case Score Seconds Revisions
2026-09-16 10:57Z case-01-event-driven-http 1.0 2,932 design_recommendations × 2
2026-09-17 08:44Z case-03-internal-web-app 1.0 2,623 deployment_guidance × 3

No scored run of case-02-high-scale-ingestion is on record.

Case 03 used 3.27 million tokens over 127 model requests. By role: IaC author 812 thousand, validator 618 thousand, describer 513 thousand, Orchestrator 356 thousand, product selector 345 thousand. These figures are the evidence base for iterations 4 and 5.

All 8 checks are structural, so a better or worse Skill scores the same. Iteration 4 step 8 addresses that.

Suite at the end of this step: 226 passed.