From the case 02 run cancelled at 66 minutes on 2026-10-07:
- The validator records verdicts a few Checks at a time (verdict_put) and
they are attached on submission. It timed out five times writing them in
one reply. Verdicts on the same code survive a run cut short; verdicts on
changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
looked up on its Element's resource type, and an unknown resource type is
reported. Two Revisions in that run were for Checks no code could meet.
On the two designs that passed validation it raises 2 of 96 Checks, both
one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.
Plans record both steps and the cancelled run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):
- A Deliverable stands while what it was written from is unchanged. The
architecture description is written from the Elements and Relationships,
so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
for each condition, Jev chooses which Check verifies it, or none. Asked as
one broad yes-or-no per requirement it could not tell a complete design
from one with known gaps; asked as a choice per condition it raised all
four known gaps of case 02 and nothing on the complete graph. An opt-in
live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.
Plans record the case 02 run and both forms of the review with figures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Start-up
- Export the env file's variables for other libraries (the Cloud provider's
client) into the process environment; the app's own settings and secrets
are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
GOOGLE_GENAI_USE_VERTEXAI.
Target architect
- Look each distinct resource type up once instead of once per setting: in
the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.
Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.
Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After deployment guidance was reopened, the Orchestrator's context still
carried the previous validation's verdicts, so it reopened the code for the
same Check three times without a validation in between (evaluation run of
2026-10-07, case 01). Record that run in the iteration 6 plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.
Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/
Unproven with a real model: no evaluation run has completed on this design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pytest, mypy and ruff now write to .cache/pytest, .cache/mypy and
.cache/ruff instead of three dot-directories in the repository root.
ruff.toml and mypy.ini set only the cache location, so lint and type
rules are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 7 of .scratch/solution-architect-agent/spec.md.
- app/assembly.py: composition root building the engine from settings
and the provider bundle (roles, ADK runtime, MCP catalog, Terraform
tools, checkpointer, recorder).
- app/a2a: a2a-sdk 1.1 JSON-RPC server (v0.3 compatible) mapping tasks to
Executions; questions, Approvals and permissions pause as
input-required; Approvals accepted only as structured data; completed
tasks return the guide and code as artifacts; agent card derived from
the dependency graph.
- POST /generate runs a pre-approved Execution with the legacy keys.
- Specialist exceptions are relayed to the user; Orchestrator exceptions
fail the task. Engine accepts a caller-chosen execution id.
- Dockerfile installs checksum-verified Terraform and ships agents,
providers and skills; add mcp and a2a-sdk requirements.
- 17 new tests; full suite 206 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 6 of .scratch/solution-architect-agent/spec.md, test-first.
- app/tools/mcp.py: MCP client and catalog exposing only allowlisted,
readOnlyHint tools as framework-neutral ToolSpecs; unreachable servers
and failed calls become discovery gaps; ADC auth with quota project.
- app/tools/local.py: Mermaid structure check and Terraform fmt,
validate and plan in a scratch copy; apply is never constructed.
- Tool factory gives roles their named local tools and bundle MCP
servers; pillar advisors get general-tier documentation tools.
- providers/gcp.yaml: 42 allowlisted tools reviewed against live
tools/list of 12 Google Cloud MCP servers (snapshot in
providers/reviews/), excluding data-reading and secret-capable tools.
- scripts/list-mcp-tools.py, docs/operations/mcp-access.md (IAM deny
policy, service account roles), cloud-discovery and iac-validation
Skills, MCP and Terraform settings, pytest asyncio auto mode.
- 31 new tests; full suite 189 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 5 of .scratch/solution-architect-agent/spec.md, test-first.
- agents/*.md: provider-neutral role files for the Orchestrator, nine
Specialists and the pillar advisor, with a shared Specialist protocol.
- app/agents/roles.py: load and validate role files; Skills, MCP
servers and models come from the provider bundle.
- app/agents/adk_runtime.py: AgentRuntime on google-adk 2.9 with a
per-run LlmAgent, role-scoped SkillToolset (no run_skill_script),
step budgets, early stop on turn-ending tools, LiteLLM models.
- app/execution/pillars.py + toolkit.py: design advisor fans out to one
advisor per WAF pillar in parallel.
- Specialists receive every earlier-Phase Deliverable as input.
- Bundle gains role_skills, pillar_skills and -latest Gemini aliases;
pin google-adk[extensions]==2.9.0.
- 37 new tests; full suite 158 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 4 of .scratch/solution-architect-agent/spec.md, test-first.
- app/agents/runtime.py: framework-neutral AgentRuntime interface
(RoleSpec, AgentTask, ToolSpec, outcomes); fake_runtime.py replays
scripted tool calls through the real tools with step budgets.
- app/execution/graph.py: Orchestrator turn fans out to parallel
Specialist tasks (Send), pauses with interrupt() for questions,
Approvals and cloud-read permission, and finalizes or ends with
warnings. Only the Orchestrator turn writes the Ledger.
- app/execution/tools.py: Orchestrator tools, Specialist submit and
needs_input, and the cloud-read permission gate.
- app/execution/engine.py: ExecutionEngine.start/resume returning an
Interaction; the single interface for A2A, /generate and eval.
- Recorders (in-memory, DatabaseManager), configurable checkpointer
(memory, sqlite, postgres), engine settings, langgraph checkpoint
dependencies.
- 19 new tests; full suite 121 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 3 of .scratch/solution-architect-agent/spec.md, test-first.
- app/execution/deliverables.py: four Phases, nine Deliverables and
their Dependencies declared once, plus per-kind contracts (required
headings, attachments) checked by check_submission.
- app/execution/ledger.py: pure Ledger enforcing both acceptance
thresholds, Phase Approvals (including pre-approved mode and change
requests), Revision cascade with withdrawn Approvals, the revision
cap with escalation and user override, attempt numbers that reject
stale submissions, an ordered event log, and dict round-tripping.
- 37 new tests; full suite 102 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 2 of .scratch/solution-architect-agent/spec.md.
- Vendor google-cloud-solution-architecture and the six
google-cloud-waf-* Skills from google/skills@81a31a6, unmodified,
with LICENSE, SHA256SUMS and VENDOR.md; add
scripts/sync-vendored-skills.sh to re-sync at a pinned sha.
- app/agent_skills.py: discover Skills and validate SKILL.md
frontmatter against the Agent Skills specification.
- app/providers.py + providers/gcp.yaml: load and validate a Cloud
provider bundle (Skills, output template, per-role models, tool
tiers, remote MCP servers). MCP allowlists stay empty until the
read-only review in increment 6.
- Settings CLOUD_PROVIDER and MCP_ENABLE_PREVIEW.
- 35 new tests; full suite 65 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
These 30 .pyc files were committed before .gitignore covered them and
changed on every test run. Files remain on disk; .gitignore already
excludes __pycache__/ and *.pyc.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
30 passed on the current suite; notes resolved dependency versions,
SQLite fallback, and test side effects (execution dirs, tracked pyc).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Record the direction and build plan for replacing the upstream
google-cloud-solution-architecture skill with a portable agent.
- ADR-0002: Agent Skills are the portable unit of behaviour, loaded by
the framework's native skill runtime; discovery and grounding use
each Cloud provider's hosted remote MCP servers.
- ADR-0003: LangGraph holds Execution state, checkpoints and
interrupts; a ReAct Orchestrator and Specialists run as ADK
LlmAgents with SkillToolset. Supersedes ADR-0001.
- CONTEXT.md: domain glossary (Execution, Phase, Approval, Revision,
Orchestrator, Specialist, Deliverable, Dependency, Cloud provider).
- .scratch/solution-architect-agent/spec.md: build spec with the
dependency graph, Revision rules, tool tiers, MCP allowlists, A2A
interaction, tests and nine build increments.
- .scratch/phase-pipeline/spec.md: superseded; kept as decision log.
- CLAUDE.md and docs/agents/: agent skill configuration (local
markdown issue tracker, triage labels, domain docs).
- README.md: direction note; flags current ADK classes as stubs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>