Every Execution now begins by agreeing the scope, before requirements.
- A new Phase and Deliverable, produced by a scope analyst with two tools:
scope_read shows what is settled and what is not; scope_set records fields
with where each value came from (the request, the customer's answer, or an
assumption with its reason). Code checks each value against the catalogue.
- With someone present the analyst asks for what is missing; with nobody
present it assumes it, visibly. With no spec it drafts one from the request.
A value the customer stated in the file is changed only by their own answer.
- A submission cannot leave an essential field unknown. Code writes the
completed spec, the data every later Specialist reads, and the information
register, from what was recorded.
- The spec's status follows the Execution: under review, approved, then
locked as a Baseline. A Change request starts from the delivered scope.
- The requirements analyst derives from the agreed scope and no longer asks
for what it settles.
About 80 tests assumed requirements came first; tests/scoped.py runs scoping
ahead of a test's own script, and tests about order and counts are updated.
Not run with real models.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A pillar's checklist item the request gives no information about was marked
"not addressed", which reads as a fault in the design. It is missing
information. The checklist review gains a fourth outcome, to confirm with the
customer, which must carry the question to ask. The questions are written
into the design recommendations and the guide, and to customer-questions.md;
the evaluation's judged check counts such an item as accounted for.
Records the owner's decisions of 2026-10-10 and the vendor's answers on Jev's
retention, training, context and rate limits (iteration 5 step 1), and the
owner's labels, which showed the judged question was wrong, not the judges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reviewer role reads the requirements and recommendations against the Checks
when the Target architecture is first submitted. Three runs of case 03: it did
not stop the validator sending the design back, and the designs it reviewed
ended with 87 and 93 Checks against 52, so they cost more. It stays behind
DESIGN_COVERAGE_REVIEW, off (iteration 6 step 13).
Also: the schema review no longer reads `true` or `false` in backticks as a
setting name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Infrastructure code is plain resources by default. An Execution may opt in
with terraform_modules; the deployment default is TERRAFORM_MODULES, off.
- A module is accepted only from the registry namespaces the provider bundle
lists (terraform-google-modules), at one exact version. Any other source is
refused before Terraform runs; until now nothing stopped generated code
naming any source for terraform init to download.
- terraform_module looks up a module's inputs, outputs and submodules from
the registry and says when it cannot be used with the pinned provider.
- skills/google-cloud-iac-modules: when a module fits, and four modules with
examples a test validates against hashicorp/google 8.6.0.
- Opted in, the IaC author and validator get the Skill and the lookup. Not
opted in, they get neither and a module block is refused in code.
- A Change request follows its Baseline. execution.json records the choice.
- case-07 runs case 03's request with modules opted into.
Azure (aztfm) and AWS (terraform-aws-modules) sources are recorded in the
plan for when those Cloud providers are added. No case has run with this.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Schema review: a dotted path is judged only when it starts at a name the
resource type has, so a database flag or a service name is no longer read
as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
attempt, named in the provider bundle. Measured on case 03 and left off:
Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
case 03 (no Revision, 22 minutes against 28, quality held).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
Skill from the folders the deployment configures; recorded with its version.
An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
one without, and triage of evidence an existing lesson already covers.
Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
the scores kept in its record. The evaluation takes its rubric from the same
place.
Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A graded question per pillar (not at all, only indirectly, directly) picks
the pillar advisors to consult again on a Revision or a Change request; the
design advisor is told which, and keeps its earlier advice for the rest. As
a yes or no the answer was yes for nearly every pillar of every change.
A completed Execution now writes working.json with each pillar's advice
and checklist review, and a Change request starts from its Baseline's, so
only the pillars a change touches are consulted afresh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A caller restates a request to say how requirements have changed, naming a
completed Execution of the same Customer case as the Baseline.
- app/baseline.py reads a Baseline from its files and refuses one that
does not exist, did not complete or belongs to another case, before
anything runs or is recorded. A Baseline is never modified.
- Requirements carry a change (unchanged, added, changed, removed) and a
reason. Submission is refused when a Baseline requirement disappears, an
id is reused, or an unchanged requirement's text differs. A removed
requirement stays on record and asks nothing downstream.
- Each Specialist starts from its delivered version; the graph and the code
are its working copy, and the description is carried over when the
design's shape is the same.
- app/workflows/changes.py compares versions by id and traces what each
changed requirement reaches; anything else that changed is listed as
unrequested. Approvals show the change for their Phase, and changes.json
is written with the full Deliverables.
- Intake no longer starts a message read as a change: it names the
Execution to give as the Baseline.
- A fifth evaluation case runs a request and then a change to it, with
four checks that the change is recorded, reaches Checks, stays contained
and costs less than its Baseline.
Tested with scripted agents only; no Change request has run with real
models.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The target architect exhausted its 45 tool calls on a first attempt in
three of five runs: 31 schema lookups and 29 graph_put calls for one
design. Each time it carried on from its working graph, at a cost of four
to ten minutes.
- terraform_schema takes resource_types, a list, and outlines every
argument and block each accepts, by name, in one reply.
- The architect is told to make that one call once the Elements are
decided, and to build the graph in a handful of calls of ten to twenty
items.
Case 04 then completed at 100% for the first time, the architect
finishing in one attempt with 11 lookups and 13 graph calls.
Plans record the run and correct three step statuses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
been shown a draft of that Deliverable (CONFIRM_DRAFTS).
Routing and reviews
- Intake: a message that starts something is read first. A question is
answered and starts nothing; a change to delivered work starts as a new
request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
(conditions) apart from what it cannot (outside_code); the latter needs
no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
reviews into the design recommendations. The judged checklist check went
from 9-45% a pillar to 100% in one run of case 04.
Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.
Smaller
- An Execution that ends with warnings writes its Deliverables, with the
warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.
Plans record nine runs of 2026-10-08. Iteration 6 is complete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the case 02 run cancelled at 66 minutes on 2026-10-07:
- The validator records verdicts a few Checks at a time (verdict_put) and
they are attached on submission. It timed out five times writing them in
one reply. Verdicts on the same code survive a run cut short; verdicts on
changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
looked up on its Element's resource type, and an unknown resource type is
reported. Two Revisions in that run were for Checks no code could meet.
On the two designs that passed validation it raises 2 of 96 Checks, both
one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.
Plans record both steps and the cancelled run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):
- A Deliverable stands while what it was written from is unchanged. The
architecture description is written from the Elements and Relationships,
so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
for each condition, Jev chooses which Check verifies it, or none. Asked as
one broad yes-or-no per requirement it could not tell a complete design
from one with known gaps; asked as a choice per condition it raised all
four known gaps of case 02 and nothing on the complete graph. An opt-in
live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.
Plans record the case 02 run and both forms of the review with figures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Start-up
- Export the env file's variables for other libraries (the Cloud provider's
client) into the process environment; the app's own settings and secrets
are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
GOOGLE_GENAI_USE_VERTEXAI.
Target architect
- Look each distinct resource type up once instead of once per setting: in
the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.
Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.
Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.
Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/
Unproven with a real model: no evaluation run has completed on this design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Increment 6 of .scratch/solution-architect-agent/spec.md, test-first.
- app/tools/mcp.py: MCP client and catalog exposing only allowlisted,
readOnlyHint tools as framework-neutral ToolSpecs; unreachable servers
and failed calls become discovery gaps; ADC auth with quota project.
- app/tools/local.py: Mermaid structure check and Terraform fmt,
validate and plan in a scratch copy; apply is never constructed.
- Tool factory gives roles their named local tools and bundle MCP
servers; pillar advisors get general-tier documentation tools.
- providers/gcp.yaml: 42 allowlisted tools reviewed against live
tools/list of 12 Google Cloud MCP servers (snapshot in
providers/reviews/), excluding data-reading and secret-capable tools.
- scripts/list-mcp-tools.py, docs/operations/mcp-access.md (IAM deny
policy, service account roles), cloud-discovery and iac-validation
Skills, MCP and Terraform settings, pytest asyncio auto mode.
- 31 new tests; full suite 189 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These 30 .pyc files were committed before .gitignore covered them and
changed on every test run. Files remain on disk; .gitignore already
excludes __pycache__/ and *.pyc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>