Commit Graph

36 Commits

Author SHA1 Message Date
7ee7054523 docs: record case 06 passing after the verdict and naming fixes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 15:23:34 +01:00
c4c8a1f94e fix: a Check is judged by its own words, and a wrong verdict can be disputed
Some checks failed
validation / verify (push) Failing after 10s
Case 06 reached its Revision limit because the validator failed a Check that
a role is granted, for the identity also holding other roles, and the IaC
author could only answer by recording the Check as unimplementable. The
validator is told what its own words require; the author is told to dispute
such a verdict and change nothing.

The evaluation's naming standard for case 06 accepted only a variable inside
each name; it now accepts names built from a local as well.

Records the reruns: case 04 passed at 12 of 12, case 06 ended with warnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 14:40:31 +01:00
135401e41b feat: module follow-ups, and L-07 reworded
- A Check met through a module is met by a value in the code: the modules
  Skill tells the author to set every input a Check depends on.
- scripts/check-terraform-modules.py reports catalogued modules with a newer
  version, and watched modules that can now be used with the pinned provider.
- L-07 no longer mentions a low-carbon location, which led to a Check about a
  region's carbon score that code cannot show. Reworded with the owner's
  approval.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 13:34:45 +01:00
81ac135cd9 fix: what only the provider publishes is outside code, not a condition
A requirement for a "low-carbon region" was written as a condition for code
to meet, and came back in at least five runs as a gap, an unverifiable Check
or review noise. The requirements-chain Skill and the two roles that write
conditions now put a carbon rating or a service level under outside_code;
the condition is the choice configuration can show, such as the region set.

Records the first runs with lessons L-07 to L-09: the faults they name did
not recur, and Revisions did not fall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 13:31:48 +01:00
5feaaa6271 feat: lessons from what recurs about one setting, and three more approved
Some checks failed
validation / verify (push) Failing after 10s
Evidence was grouped only by the kind of problem, and a group gets one lesson,
so what kept recurring inside "validation proposed a Check" could not be
seen. A proposed or unimplementable Check is now also counted under each
provider setting or resource type it names; a subject seen in two or more
Executions is its own group and reaches the curator past triage.

Over 32 Executions that gave five proposals. The owner approved three, now
published: L-07 location Checks, L-08 Cloud SQL backup encryption, L-09
backend service capabilities. L-10 is held for more evidence and L-11
rejected.

Also: retiring a lesson marks its proposal, so publish no longer brings a
retired lesson back. L-04 had returned to the Skill that way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:47:18 +01:00
ee1ba38867 feat: a design-time coverage review, measured and left off
A reviewer role reads the requirements and recommendations against the Checks
when the Target architecture is first submitted. Three runs of case 03: it did
not stop the validator sending the design back, and the designs it reviewed
ended with 87 and 93 Checks against 52, so they cost more. It stays behind
DESIGN_COVERAGE_REVIEW, off (iteration 6 step 13).

Also: the schema review no longer reads `true` or `false` in backticks as a
setting name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 12:33:06 +01:00
1cf1234143 fix: an unimplementable Check is corrected everywhere it applies, in one Revision
The reason sent back to the target architect says to read every Check against
it, and the IaC author records every Check that cannot be met in one
submission. Records the investigation of why the Target architecture is
revised: two causes, about half each (iteration 6 step 12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:51:46 +01:00
71da0ed504 chore: housekeeping from the runs of 2026-10-09
Some checks failed
validation / verify (push) Failing after 12s
- An Execution's files are written as each Deliverable is submitted, marked
  in_progress, so a killed process or one waiting for its user leaves them on
  disk. A write removes files an earlier write left that it no longer has.
- A question about delivered work is answered from that Execution's solution
  guide by the reply writer; a question about the service is answered as
  before. Nothing is started for either.
- Lessons: prove writes only for lessons the Skill holds; the Skill records
  the vendored Skill versions the lessons were read against, status names any
  that changed, and a test fails until they are reviewed.
- Four more code patterns: Firestore, BigQuery, Eventarc and GKE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:44:38 +01:00
10d23a5f91 docs: record the first runs with and without Terraform modules
Case 03's request once each way, both at 100%. Opted in, a module was used
for private Cloud SQL only and 14 Checks were met through its inputs. The
difference in time and cost between the two runs is their Revisions, not
the modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:47:09 +01:00
ef37f59bad feat: opt-in published Terraform modules from approved sources (iteration 8)
Infrastructure code is plain resources by default. An Execution may opt in
with terraform_modules; the deployment default is TERRAFORM_MODULES, off.

- A module is accepted only from the registry namespaces the provider bundle
  lists (terraform-google-modules), at one exact version. Any other source is
  refused before Terraform runs; until now nothing stopped generated code
  naming any source for terraform init to download.
- terraform_module looks up a module's inputs, outputs and submodules from
  the registry and says when it cannot be used with the pinned provider.
- skills/google-cloud-iac-modules: when a module fits, and four modules with
  examples a test validates against hashicorp/google 8.6.0.
- Opted in, the IaC author and validator get the Skill and the lookup. Not
  opted in, they get neither and a module block is refused in code.
- A Change request follows its Baseline. execution.json records the choice.
- case-07 runs case 03's request with modules opted into.

Azure (aztfm) and AWS (terraform-aws-modules) sources are recorded in the
plan for when those Cloud providers are added. No case has run with this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 22:09:29 +01:00
e5e1647ff1 feat: prove the lessons, check overlays in evaluation, and settle the cost steps
Some checks failed
validation / verify (push) Failing after 13s
- Schema review: a dotted path is judged only when it starts at a name the
  resource type has, so a database flag or a service name is no longer read
  as a missing setting. Lesson L-04, which worked around that, is retired.
- Skill routing: in a Change request the question says what changed.
- Evaluation: a case may list code_standards the infrastructure code must
  meet; case 06 checks its overlay's regions, naming and labels.
- Model tiers (iteration 5 step 6): a lighter model for a role's first
  attempt, named in the provider bundle. Measured on case 03 and left off:
  Flash saved five cents on the IaC author and took twice the requests.
- Steps 7 and 9 decided against on the numbers, recorded in the plans.
- Plans: the runs of cases 05 and 06, and the first proof of the lessons on
  case 03 (no Revision, 22 minutes against 28, quality held).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:27:09 +01:00
567aede5ca feat: overlay Skills, and five more Jev decisions (iterations 4 and 5)
Iteration 4:
- Overlay Skills (step 12): an Execution names an organisation's standards
  Skill from the folders the deployment configures; recorded with its version.
  An example standards Skill and an evaluation case that uses it.
- Lessons (step 10): a prove command that compares a run with lessons against
  one without, and triage of evidence an existing lesson already covers.

Iteration 5:
- Step 8: the references of a Specialist's Skills its task needs are named in
  its context as read_first.
- Step 14: a remote MCP server's reply that instructs the agent is withheld;
  the user's own text is flagged as a risk and nothing more.
- Step 15: in a Change request, items only reworded are listed apart from
  real changes.
- Step 19: each Execution's design recommendations are judged when it ends and
  the scores kept in its record. The evaluation takes its rubric from the same
  place.

Each question was calibrated against live Jev; none has run in a case. Steps
6, 7, 9, 13 and 18 stay planned, with the reasons in the plan map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 17:45:56 +01:00
a7bc5310f3 feat: the lessons loop, and six lessons approved and published
Capture, distil, review, publish, retire (ADR-0004). Lesson candidates of
past Executions are grouped into what recurred; a lesson-curator role,
outside any Execution, proposes a lesson per group worth one; each proposal
is an issue file a person approves or rejects; an approved lesson is
written into the provider's lessons Skill, which a Specialist is given only
once it holds lessons for that Specialist's Deliverable. LESSONS_ENABLED
off runs without them, for an evaluation with and without.

The curator proposed six lessons from 21 Executions. The owner approved all
six on 2026-10-09 and they are published: five for the target architect,
one for the IaC author. None is proved by an evaluation yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 15:19:32 +01:00
59a641fded fix: a changed functional requirement is reached through its components
The change-case check asked for a changed Check whose source is the changed
requirement. A functional requirement is carried by the components that
serve it, so it is reached through a Check on an Element realising one.

Plans record the first Change request with real models: a retention change
made in 18 minutes for a third of its Baseline's tokens, touching seven
Checks on one Element and nothing else, and the regression runs of cases
02, 03 and 04, all at 100%. Iteration 7 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 13:50:18 +01:00
5ef434d00e feat: Change requests against a Baseline (ADR-0006)
A caller restates a request to say how requirements have changed, naming a
completed Execution of the same Customer case as the Baseline.

- app/baseline.py reads a Baseline from its files and refuses one that
  does not exist, did not complete or belongs to another case, before
  anything runs or is recorded. A Baseline is never modified.
- Requirements carry a change (unchanged, added, changed, removed) and a
  reason. Submission is refused when a Baseline requirement disappears, an
  id is reused, or an unchanged requirement's text differs. A removed
  requirement stays on record and asks nothing downstream.
- Each Specialist starts from its delivered version; the graph and the code
  are its working copy, and the description is carried over when the
  design's shape is the same.
- app/workflows/changes.py compares versions by id and traces what each
  changed requirement reaches; anything else that changed is listed as
  unrequested. Approvals show the change for their Phase, and changes.json
  is written with the full Deliverables.
- Intake no longer starts a message read as a change: it names the
  Execution to give as the Baseline.
- A fifth evaluation case runs a request and then a change to it, with
  four checks that the change is recorded, reaches Checks, stays contained
  and costs less than its Baseline.

Tested with scripted agents only; no Change request has run with real
models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 11:38:15 +01:00
db28d02f56 fix: one schema lookup for a whole design, and larger graph batches
The target architect exhausted its 45 tool calls on a first attempt in
three of five runs: 31 schema lookups and 29 graph_put calls for one
design. Each time it carried on from its working graph, at a cost of four
to ten minutes.

- terraform_schema takes resource_types, a list, and outlines every
  argument and block each accepts, by name, in one reply.
- The architect is told to make that one call once the Elements are
  decided, and to build the graph in a handful of calls of ten to twenty
  items.

Case 04 then completed at 100% for the first time, the architect
finishing in one attempt with 11 lookups and 13 graph calls.

Plans record the run and correct three step statuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 10:34:47 +01:00
654370bb46 feat: all four evaluation cases pass; intake, checklist review, thresholds
Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:32:57 +01:00
d41607f505 fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator
turns on two router faults:

- A reply that quoted the agent's question back was judged on that question.
  A reply now reaches the router with quoted questions and Q:/A: labels
  removed, and the questions are given to the decider as background.
- At a Revision limit the reply was read as another change request for the
  same Deliverable, and the refusal repeated. The user is now told once,
  the reply is taken as the answer, that Deliverable is not reopened again,
  and the Execution carries on with it as it stands.
- Any run that asks the user four times running with no work done in
  between now ends with a warning.

The validator is told to confirm with the user in an interactive Execution.

Plans record the fixes and the rerun, which did not finish: both fixes
held, and a third fault appeared that is not fixed here. A scripted answer
restating a requirement, given to a draft-confirmation question, reopens
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:05:03 +01:00
c9f59be274 docs: accept ADR-0004, lessons and overlays arrive as Skills
Accepted by the owner on 2026-10-08 with its six decisions as written:
a person approves every lesson; a lesson must be shown to help by
evaluation; lessons are per Cloud provider; overlay Skills come only from
configured directories; each Execution records the Skills it loaded; and
lesson candidates are client data kept out of the repository.

- CONTEXT.md gains Lesson candidate, Lesson and Overlay Skill.
- Iteration 4 step 9 is complete; steps 10 and 12 name the decisions they
  must meet and are unblocked. Neither is built.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:05:03 +01:00
4e7a16cedd chore: raise the validator's budget and record the runs of cases 02, 03 and 04
- Validator budget 15 to 30: recording verdicts in pieces for 100 Checks
  used all 15 calls in one attempt.
- Plans record three runs on 2026-10-08:
  case 02 completed at 100% with no Revision in 24 minutes;
  case 03 completed at 100% with two Revisions of the design in 40 minutes;
  case 04 (interactive) ended with warnings at 75%, on two router faults:
  a reply that echoes the question is read as a change request, and the
  router loops at the Revision limit. Neither is fixed here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 13:16:36 +01:00
58e778ca7a feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07:

- The validator records verdicts a few Checks at a time (verdict_put) and
  they are attached on submission. It timed out five times writing them in
  one reply. Verdicts on the same code survive a run cut short; verdicts on
  changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
  looked up on its Element's resource type, and an unknown resource type is
  reported. Two Revisions in that run were for Checks no code could meet.
  On the two designs that passed validation it raises 2 of 96 Checks, both
  one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.

Plans record both steps and the cancelled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:39:13 +01:00
5adc911a82 feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:58:38 +01:00
f1f6f1b83a feat: load the env file at start-up, trim schema lookups, remove legacy samples
Start-up
- Export the env file's variables for other libraries (the Cloud provider's
  client) into the process environment; the app's own settings and secrets
  are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
  GOOGLE_GENAI_USE_VERTEXAI.

Target architect
- Look each distinct resource type up once instead of once per setting: in
  the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
  budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.

Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
  tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.

Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:06:58 +01:00
5bb22d984e fixed multile issues
Some checks failed
validation / verify (push) Failing after 8s
2026-10-07 17:33:36 +01:00
4c4d44ef87 fix: show the Orchestrator a validation only while it stands
After deployment guidance was reopened, the Orchestrator's context still
carried the previous validation's verdicts, so it reopened the code for the
same Check three times without a validation in between (evaluation run of
2026-10-07, case 01). Record that run in the iteration 6 plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:08:09 +01:00
e9246828b3 feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:59:09 +01:00
7ee49ab352 latest refactoring
Some checks failed
validation / verify (push) Failing after 10s
2026-10-06 20:22:37 +01:00
43cbd215e2 decomposer: record implementation detail for 8 step(s)
Some checks failed
validation / verify (push) Failing after 13s
2026-09-01 20:04:10 +00:00
146909227b decomposer: record implementation detail for 8 step(s)
Some checks failed
validation / verify (push) Failing after 14s
2026-09-01 20:02:15 +00:00
3d468dcff7 decomposer: record implementation detail for 8 step(s)
Some checks failed
validation / verify (push) Failing after 14s
2026-09-01 20:01:44 +00:00
2d528157b6 decomposer: record implementation detail for 8 step(s)
Some checks failed
validation / verify (push) Failing after 9s
2026-09-01 19:59:38 +00:00
99c3f2d2d7 decomposer: record implementation detail for 3 step(s)
Some checks failed
validation / verify (push) Failing after 8s
2026-09-01 19:57:18 +00:00
bf31dd49b2 decomposer: record implementation detail for 2 step(s)
Some checks failed
validation / verify (push) Failing after 18s
2026-09-01 19:55:54 +00:00
2c158b8d6a decomposer: record implementation detail for 8 step(s) 2026-09-01 19:50:54 +00:00
9bfdf6eca3 decomposer: record implementation detail for 8 step(s) 2026-09-01 19:47:40 +00:00
d73472eed4 decomposer: generate deliverable files for Discover and structure the solution's functional requirements, non-functional requirements, constraints, assumptions, and open questions without selecting cloud products.; Select Google Cloud products from the confirmed requirements and produce the solution architecture, Mermaid diagram, architecture description, and Terraform infrastructure-as-code.; Validate the Terraform infrastructure and architecture artifacts without deploying resources by running formatting checks, Terraform validation, and a dry-run or plan-oriented deployment check.; Package the approved requirements, architecture, Mermaid diagram, Terraform IaC, and validation results into solution-architecture-guide.md in the gcp_solution_architecture_agent repository.; Verify that the gcp_solution_architecture_agent repository contains the packaged solution-architecture-guide.md with the approved workflow outputs.; Verify that the repository is derived from the workflow_agent template and implements the complete four-phase Google Cloud solution architecture workflow alongside the packaged guide.; Publish the verified gcp_solution_architecture_agent repository with its completed workflow implementation and solution architecture guide.; Verify that the published repository revision contains the completed workflow implementation and solution architecture guide. 2026-09-01 19:47:05 +00:00