From the case 02 run cancelled at 66 minutes on 2026-10-07:
- The validator records verdicts a few Checks at a time (verdict_put) and
they are attached on submission. It timed out five times writing them in
one reply. Verdicts on the same code survive a run cut short; verdicts on
changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
looked up on its Element's resource type, and an unknown resource type is
reported. Two Revisions in that run were for Checks no code could meet.
On the two designs that passed validation it raises 2 of 96 Checks, both
one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.
Plans record both steps and the cancelled run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.
Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/
Unproven with a real model: no evaluation run has completed on this design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Increment 6 of .scratch/solution-architect-agent/spec.md, test-first.
- app/tools/mcp.py: MCP client and catalog exposing only allowlisted,
readOnlyHint tools as framework-neutral ToolSpecs; unreachable servers
and failed calls become discovery gaps; ADC auth with quota project.
- app/tools/local.py: Mermaid structure check and Terraform fmt,
validate and plan in a scratch copy; apply is never constructed.
- Tool factory gives roles their named local tools and bundle MCP
servers; pillar advisors get general-tier documentation tools.
- providers/gcp.yaml: 42 allowlisted tools reviewed against live
tools/list of 12 Google Cloud MCP servers (snapshot in
providers/reviews/), excluding data-reading and secret-capable tools.
- scripts/list-mcp-tools.py, docs/operations/mcp-access.md (IAM deny
policy, service account roles), cloud-discovery and iac-validation
Skills, MCP and Terraform settings, pytest asyncio auto mode.
- 31 new tests; full suite 189 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 5 of .scratch/solution-architect-agent/spec.md, test-first.
- agents/*.md: provider-neutral role files for the Orchestrator, nine
Specialists and the pillar advisor, with a shared Specialist protocol.
- app/agents/roles.py: load and validate role files; Skills, MCP
servers and models come from the provider bundle.
- app/agents/adk_runtime.py: AgentRuntime on google-adk 2.9 with a
per-run LlmAgent, role-scoped SkillToolset (no run_skill_script),
step budgets, early stop on turn-ending tools, LiteLLM models.
- app/execution/pillars.py + toolkit.py: design advisor fans out to one
advisor per WAF pillar in parallel.
- Specialists receive every earlier-Phase Deliverable as input.
- Bundle gains role_skills, pillar_skills and -latest Gemini aliases;
pin google-adk[extensions]==2.9.0.
- 37 new tests; full suite 158 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Increment 2 of .scratch/solution-architect-agent/spec.md.
- Vendor google-cloud-solution-architecture and the six
google-cloud-waf-* Skills from google/skills@81a31a6, unmodified,
with LICENSE, SHA256SUMS and VENDOR.md; add
scripts/sync-vendored-skills.sh to re-sync at a pinned sha.
- app/agent_skills.py: discover Skills and validate SKILL.md
frontmatter against the Agent Skills specification.
- app/providers.py + providers/gcp.yaml: load and validate a Cloud
provider bundle (Skills, output template, per-role models, tool
tiers, remote MCP servers). MCP allowlists stay empty until the
read-only review in increment 6.
- Settings CLOUD_PROVIDER and MCP_ENABLE_PREVIEW.
- 35 new tests; full suite 65 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>