# Validation Runbook Run from the repository root without credentials: ```bash # 1. Pytest unit & integration test suite pytest -v # 2. Real-model evaluation of the engine (needs model credentials; costs usage) python3 -m eval.solution_eval # 3. Optional: validate the Terraform an Execution generated terraform -chdir=deliverables/cases//executions//terraform init -backend=false terraform -chdir=deliverables/cases//executions//terraform validate ``` The repository holds no Terraform of its own. The tests need the `terraform` binary (1.14.3 in CI) to validate the code patterns in `skills/google-cloud-iac-patterns/`. `init -backend=false` downloads the pinned provider and creates no infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review. ## Evaluation options - `python3 -m eval.solution_eval --judge` adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations. - `python3 -m eval.solution_eval case-04-interactive-web-app` runs one case with a scripted user in the loop. - Several cases run side by side as separate processes; each writes its own folder under `eval/results/`, with a progress file while it runs. - A case with `change` runs its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case with `overlay_skills` names overlay Skills, and `code_standards` lists what the generated code must and must not contain. - `LESSONS_ENABLED=false python3 -m eval.solution_eval ` runs without the lessons Skill. `python3 -m eval.curate_lessons prove --without --with ` compares the two and records the result in the lesson proposals. - `python3 -m eval.calibrate_judge` measures the judges' stability and agreement. Agreement with a person needs the labels in `eval/datasets/judge/labels.json` filled in. - A case's `thresholds` in `eval/datasets/solution_cases.json` bound its revisions, seconds and tokens; exceeding one is reported as a regression.