# Validation Runbook Run from the repository root without credentials: ```bash # 1. Pytest unit & integration test suite pytest -v # 2. Real-model evaluation of the engine (needs model credentials; costs usage) python3 -m eval.solution_eval # 3. Optional Terraform formatting & validation terraform -chdir=terraform fmt -check -recursive terraform -chdir=terraform init -backend=false terraform -chdir=terraform validate ``` `init -backend=false` may download the pinned provider constraint but does not create infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review. ## Evaluation options - `python3 -m eval.solution_eval --judge` adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations. - `python3 -m eval.solution_eval case-04-interactive-web-app` runs one case with a scripted user in the loop. - A case's `thresholds` in `eval/datasets/solution_cases.json` bound its revisions, seconds and tokens; exceeding one is reported as a regression.