feat: load the env file at start-up, trim schema lookups, remove legacy samples

Start-up
- Export the env file's variables for other libraries (the Cloud provider's
  client) into the process environment; the app's own settings and secrets
  are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
  GOOGLE_GENAI_USE_VERTEXAI.

Target architect
- Look each distinct resource type up once instead of once per setting: in
  the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
  budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.

Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
  tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.

Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-07 20:06:58 +01:00
parent 5bb22d984e
commit f1f6f1b83a
34 changed files with 161 additions and 402 deletions

View File

@@ -56,3 +56,26 @@ Changes this points to, not yet made:
- Have the validator distinguish "the code does not do this" from "this cannot be done to this resource".
- Check a proposed addition against the Element it names before it becomes a Check.
- Raise or tune the model timeout for the target architect.
## Run of 2026-10-07, 16:59 UTC (`eval/results/20261007T165920Z/`)
Case 01 again, after iteration 4 steps 11, 13 and 17 to 20 and the router of iteration 5. Not judged.
| | Run of 12:59 | This run |
|---|---|---|
| Status | completed with warnings | completed |
| Score | 73% | 100% (11 of 11 checks) |
| Seconds | 3,627 | 1,091 |
| Tokens | 4.25 million | 2.38 million |
| Revisions | Revision limit reached | none |
| Model timeouts | target architect, both attempts | none |
| `terraform validate` failures | 13 | 0 |
| IaC author | 57 requests, 2.35 million tokens | 11 requests, 271 thousand tokens |
| Target architect | timed out twice | 55 requests, 1.15 million tokens |
The graph had 13 Elements, 17 Relationships and 42 Checks. The code met all 42 on the first validation; no Check was unimplementable or not verifiable, and nothing was proposed. The Orchestrator was the router: 11 turns, no model chose an action, and no decision needed Jev.
What went wrong: the target architect exhausted its 30 tool calls on its first attempt. It made 39 `terraform_schema` calls for a graph with 8 distinct resource types, one per setting, as its instructions told it to. Its second attempt started from the working graph and submitted in 92 seconds. Changed after the run: it is told to look up each distinct resource type once, `find` takes several names in one call, and its budget is 45. That change has not been run.
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.