Some checks failed
validation / verify (push) Failing after 11s
The README now covers the router, Change requests, overlay Skills, Jev's uses and the current layout. VALIDATION.md no longer points at the removed root Terraform. ADR-0004 and ADR-0007 gain dated "As built" notes; ADR-0007's says which uses of Jev send Deliverable content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.2 KiB
2.2 KiB
Validation Runbook
Run from the repository root without credentials:
# 1. Pytest unit & integration test suite
pytest -v
# 2. Real-model evaluation of the engine (needs model credentials; costs usage)
python3 -m eval.solution_eval
# 3. Optional: validate the Terraform an Execution generated
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform init -backend=false
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform validate
The repository holds no Terraform of its own. The tests need the terraform binary (1.14.3 in CI) to validate the code patterns in skills/google-cloud-iac-patterns/. init -backend=false downloads the pinned provider and creates no infrastructure. Do not use terraform apply as part of validation. A deployment candidate additionally requires an operator-supplied terraform plan -out=tfplan and human review.
Evaluation options
python3 -m eval.solution_eval --judgeadds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations.python3 -m eval.solution_eval case-04-interactive-web-appruns one case with a scripted user in the loop.- Several cases run side by side as separate processes; each writes its own folder under
eval/results/, with a progress file while it runs. - A case with
changeruns its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case withoverlay_skillsnames overlay Skills, andcode_standardslists what the generated code must and must not contain. LESSONS_ENABLED=false python3 -m eval.solution_eval <case>runs without the lessons Skill.python3 -m eval.curate_lessons prove --without <report.json> --with <report.json>compares the two and records the result in the lesson proposals.python3 -m eval.calibrate_judgemeasures the judges' stability and agreement. Agreement with a person needs the labels ineval/datasets/judge/labels.jsonfilled in.- A case's
thresholdsineval/datasets/solution_cases.jsonbound its revisions, seconds and tokens; exceeding one is reported as a regression.