Files
gcp_solution_architecture_a…/VALIDATION.md
Jonathan Boniface a8ed21b89d
Some checks failed
validation / verify (push) Failing after 11s
docs: bring the README, runbook, glossary and ADR notes up to date
The README now covers the router, Change requests, overlay Skills, Jev's
uses and the current layout. VALIDATION.md no longer points at the removed
root Terraform. ADR-0004 and ADR-0007 gain dated "As built" notes; ADR-0007's
says which uses of Jev send Deliverable content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:33:38 +01:00

2.2 KiB

Validation Runbook

Run from the repository root without credentials:

# 1. Pytest unit & integration test suite
pytest -v

# 2. Real-model evaluation of the engine (needs model credentials; costs usage)
python3 -m eval.solution_eval

# 3. Optional: validate the Terraform an Execution generated
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform init -backend=false
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform validate

The repository holds no Terraform of its own. The tests need the terraform binary (1.14.3 in CI) to validate the code patterns in skills/google-cloud-iac-patterns/. init -backend=false downloads the pinned provider and creates no infrastructure. Do not use terraform apply as part of validation. A deployment candidate additionally requires an operator-supplied terraform plan -out=tfplan and human review.

Evaluation options

  • python3 -m eval.solution_eval --judge adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations.
  • python3 -m eval.solution_eval case-04-interactive-web-app runs one case with a scripted user in the loop.
  • Several cases run side by side as separate processes; each writes its own folder under eval/results/, with a progress file while it runs.
  • A case with change runs its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case with overlay_skills names overlay Skills, and code_standards lists what the generated code must and must not contain.
  • LESSONS_ENABLED=false python3 -m eval.solution_eval <case> runs without the lessons Skill. python3 -m eval.curate_lessons prove --without <report.json> --with <report.json> compares the two and records the result in the lesson proposals.
  • python3 -m eval.calibrate_judge measures the judges' stability and agreement. Agreement with a person needs the labels in eval/datasets/judge/labels.json filled in.
  • A case's thresholds in eval/datasets/solution_cases.json bound its revisions, seconds and tokens; exceeding one is reported as a regression.