docs: bring the README, runbook, glossary and ADR notes up to date
Some checks failed
validation / verify (push) Failing after 11s

The README now covers the router, Change requests, overlay Skills, Jev's
uses and the current layout. VALIDATION.md no longer points at the removed
root Terraform. ADR-0004 and ADR-0007 gain dated "As built" notes; ADR-0007's
says which uses of Jev send Deliverable content.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 21:33:38 +01:00
parent e5e1647ff1
commit a8ed21b89d
7 changed files with 65 additions and 19 deletions

View File

@@ -9,16 +9,19 @@ pytest -v
# 2. Real-model evaluation of the engine (needs model credentials; costs usage)
python3 -m eval.solution_eval
# 3. Optional Terraform formatting & validation
terraform -chdir=terraform fmt -check -recursive
terraform -chdir=terraform init -backend=false
terraform -chdir=terraform validate
# 3. Optional: validate the Terraform an Execution generated
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform init -backend=false
terraform -chdir=deliverables/cases/<customer_case>/executions/<execution_id>/terraform validate
```
`init -backend=false` may download the pinned provider constraint but does not create infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
The repository holds no Terraform of its own. The tests need the `terraform` binary (1.14.3 in CI) to validate the code patterns in `skills/google-cloud-iac-patterns/`. `init -backend=false` downloads the pinned provider and creates no infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
## Evaluation options
- `python3 -m eval.solution_eval --judge` adds the judged checks: coverage of each pillar Skill's validation checklist, and a quality rubric for the design recommendations.
- `python3 -m eval.solution_eval case-04-interactive-web-app` runs one case with a scripted user in the loop.
- Several cases run side by side as separate processes; each writes its own folder under `eval/results/`, with a progress file while it runs.
- A case with `change` runs its request to completion and then that change as a Change request, and checks the change is recorded, reaches Checks, and stays contained. A case with `overlay_skills` names overlay Skills, and `code_standards` lists what the generated code must and must not contain.
- `LESSONS_ENABLED=false python3 -m eval.solution_eval <case>` runs without the lessons Skill. `python3 -m eval.curate_lessons prove --without <report.json> --with <report.json>` compares the two and records the result in the lesson proposals.
- `python3 -m eval.calibrate_judge` measures the judges' stability and agreement. Agreement with a person needs the labels in `eval/datasets/judge/labels.json` filled in.
- A case's `thresholds` in `eval/datasets/solution_cases.json` bound its revisions, seconds and tokens; exceeding one is reported as a regression.