Some checks failed
validation / verify (push) Failing after 11s
The README now covers the router, Change requests, overlay Skills, Jev's uses and the current layout. VALIDATION.md no longer points at the removed root Terraform. ADR-0004 and ADR-0007 gain dated "As built" notes; ADR-0007's says which uses of Jev send Deliverable content. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
28 lines
2.2 KiB
Markdown
28 lines
2.2 KiB
Markdown
# Validation Runbook
|
|
|
|
Run from the repository root without credentials:
|
|
|
|
```bash
|
|
# 1. Pytest unit & integration test suite
|
|
pytest -v
|
|
|
|
# 2. Real-model evaluation of the engine (needs model credentials; costs usage)
|
|
python3 -m eval.solution_eval
|
|
|
|
# 3. Optional: validate the Terraform an Execution generated
|
|
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform init -backend=false
|
|
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform validate
|
|
```
|
|
|
|
The repository holds no Terraform of its own. The tests need the `terraform` binary (1.14.3 in CI) to validate the code patterns in `skills/google-cloud-iac-patterns/`. `init -backend=false` downloads the pinned provider and creates no infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
|
|
|
|
## Evaluation options
|
|
|
|
- `python3 -m eval.solution_eval --judge` adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations.
|
|
- `python3 -m eval.solution_eval case-04-interactive-web-app` runs one case with a scripted user in the loop.
|
|
- Several cases run side by side as separate processes; each writes its own folder under `eval/results/`, with a progress file while it runs.
|
|
- A case with `change` runs its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case with `overlay_skills` names overlay Skills, and `code_standards` lists what the generated code must and must not contain.
|
|
- `LESSONS_ENABLED=false python3 -m eval.solution_eval <case>` runs without the lessons Skill. `python3 -m eval.curate_lessons prove --without <report.json> --with <report.json>` compares the two and records the result in the lesson proposals.
|
|
- `python3 -m eval.calibrate_judge` measures the judges' stability and agreement. Agreement with a person needs the labels in `eval/datasets/judge/labels.json` filled in.
|
|
- A case's `thresholds` in `eval/datasets/solution_cases.json` bound its revisions, seconds and tokens; exceeding one is reported as a regression.
|