feat: load the env file at start-up, trim schema lookups, remove legacy samples
Start-up - Export the env file's variables for other libraries (the Cloud provider's client) into the process environment; the app's own settings and secrets are never exported, and a value already in the environment wins. - Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated GOOGLE_GENAI_USE_VERTEXAI. Target architect - Look each distinct resource type up once instead of once per setting: in the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its budget. terraform_schema's find takes several names; budget 30 to 45. - A cut-off schema outline says so. Removed, as leftovers of the original pipeline that nothing read - terraform/ (no longer passed terraform validate), scripts/validate.sh, tests/test_package.py and the CI steps that ran on them. - The sample outputs under docs/ and requirements.yaml. - CI now pins Terraform 1.14.3, as the container image does. Plans record the case 01 run: completed at 100% in 18 minutes with no Revision, against 60 minutes and warnings before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -56,3 +56,26 @@ Changes this points to, not yet made:
|
||||
- Have the validator distinguish "the code does not do this" from "this cannot be done to this resource".
|
||||
- Check a proposed addition against the Element it names before it becomes a Check.
|
||||
- Raise or tune the model timeout for the target architect.
|
||||
|
||||
## Run of 2026-10-07, 16:59 UTC (`eval/results/20261007T165920Z/`)
|
||||
|
||||
Case 01 again, after iteration 4 steps 11, 13 and 17 to 20 and the router of iteration 5. Not judged.
|
||||
|
||||
| | Run of 12:59 | This run |
|
||||
|---|---|---|
|
||||
| Status | completed with warnings | completed |
|
||||
| Score | 73% | 100% (11 of 11 checks) |
|
||||
| Seconds | 3,627 | 1,091 |
|
||||
| Tokens | 4.25 million | 2.38 million |
|
||||
| Revisions | Revision limit reached | none |
|
||||
| Model timeouts | target architect, both attempts | none |
|
||||
| `terraform validate` failures | 13 | 0 |
|
||||
| IaC author | 57 requests, 2.35 million tokens | 11 requests, 271 thousand tokens |
|
||||
| Target architect | timed out twice | 55 requests, 1.15 million tokens |
|
||||
|
||||
The graph had 13 Elements, 17 Relationships and 42 Checks. The code met all 42 on the first validation; no Check was unimplementable or not verifiable, and nothing was proposed. The Orchestrator was the router: 11 turns, no model chose an action, and no decision needed Jev.
|
||||
|
||||
What went wrong: the target architect exhausted its 30 tool calls on its first attempt. It made 39 `terraform_schema` calls for a graph with 8 distinct resource types, one per setting, as its instructions told it to. Its second attempt started from the working graph and submitted in 92 seconds. Changed after the run: it is told to look up each distinct resource type once, `find` takes several names in one call, and its budget is 45. That change has not been run.
|
||||
|
||||
Acceptance criteria: met for case 01. Still open: cases 02 to 04 have not run on this build, so "all benchmark cases" and the thresholds are unset.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user