A reviewer role reads the requirements and recommendations against the Checks
when the Target architecture is first submitted. Three runs of case 03: it did
not stop the validator sending the design back, and the designs it reviewed
ended with 87 and 93 Checks against 52, so they cost more. It stays behind
DESIGN_COVERAGE_REVIEW, off (iteration 6 step 13).
Also: the schema review no longer reads `true` or `false` in backticks as a
setting name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason sent back to the target architect says to read every Check against
it, and the IaC author records every Check that cannot be met in one
submission. Records the investigation of why the Target architecture is
revised: two causes, about half each (iteration 6 step 12).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
been shown a draft of that Deliverable (CONFIRM_DRAFTS).
Routing and reviews
- Intake: a message that starts something is read first. A question is
answered and starts nothing; a change to delivered work starts as a new
request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
(conditions) apart from what it cannot (outside_code); the latter needs
no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
reviews into the design recommendations. The judged checklist check went
from 9-45% a pillar to 100% in one run of case 04.
Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.
Smaller
- An Execution that ends with warnings writes its Deliverables, with the
warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.
Plans record nine runs of 2026-10-08. Iteration 6 is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 04, the first interactive evaluation, stopped at 100 Orchestrator
turns on two router faults:
- A reply that quoted the agent's question back was judged on that question.
A reply now reaches the router with quoted questions and Q:/A: labels
removed, and the questions are given to the decider as background.
- At a Revision limit the reply was read as another change request for the
same Deliverable, and the refusal repeated. The user is now told once,
the reply is taken as the answer, that Deliverable is not reopened again,
and the Execution carries on with it as it stands.
- Any run that asks the user four times running with no work done in
between now ends with a warning.
The validator is told to confirm with the user in an interactive Execution.
Plans record the fixes and the rerun, which did not finish: both fixes
held, and a third fault appeared that is not fixed here. A scripted answer
restating a requirement, given to a draft-confirmation question, reopens
requirements.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Validator budget 15 to 30: recording verdicts in pieces for 100 Checks
used all 15 calls in one attempt.
- Plans record three runs on 2026-10-08:
case 02 completed at 100% with no Revision in 24 minutes;
case 03 completed at 100% with two Revisions of the design in 40 minutes;
case 04 (interactive) ended with warnings at 75%, on two router faults:
a reply that echoes the question is read as a change request, and the
router loops at the Revision limit. Neither is fixed here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):
- A Deliverable stands while what it was written from is unchanged. The
architecture description is written from the Elements and Relationships,
so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
for each condition, Jev chooses which Check verifies it, or none. Asked as
one broad yes-or-no per requirement it could not tell a complete design
from one with known gaps; asked as a choice per condition it raised all
four known gaps of case 02 and nothing on the complete graph. An opt-in
live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.
Plans record the case 02 run and both forms of the review with figures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Start-up
- Export the env file's variables for other libraries (the Cloud provider's
client) into the process environment; the app's own settings and secrets
are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
GOOGLE_GENAI_USE_VERTEXAI.
Target architect
- Look each distinct resource type up once instead of once per setting: in
the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.
Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.
Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.
Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/
Unproven with a real model: no evaluation run has completed on this design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>