Commit Graph

18 Commits

Author SHA1 Message Date
d41607f505 fix: read only the user's own words, and stop the router looping at a limit
Case 04, the first interactive evaluation, stopped at 100 Orchestrator
turns on two router faults:

- A reply that quoted the agent's question back was judged on that question.
  A reply now reaches the router with quoted questions and Q:/A: labels
  removed, and the questions are given to the decider as background.
- At a Revision limit the reply was read as another change request for the
  same Deliverable, and the refusal repeated. The user is now told once,
  the reply is taken as the answer, that Deliverable is not reopened again,
  and the Execution carries on with it as it stands.
- Any run that asks the user four times running with no work done in
  between now ends with a warning.

The validator is told to confirm with the user in an interactive Execution.

Plans record the fixes and the rerun, which did not finish: both fixes
held, and a third fault appeared that is not fixed here. A scripted answer
restating a requirement, given to a draft-confirmation question, reopens
requirements.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:05:03 +01:00
58e778ca7a feat: record verdicts in pieces and check the design against the schema
From the case 02 run cancelled at 66 minutes on 2026-10-07:

- The validator records verdicts a few Checks at a time (verdict_put) and
  they are attached on submission. It timed out five times writing them in
  one reply. Verdicts on the same code survive a run cut short; verdicts on
  changed code are dropped. Its model timeout is 300 s.
- A schema review at submission, in code: every setting a Check names is
  looked up on its Element's resource type, and an unknown resource type is
  reported. Two Revisions in that run were for Checks no code could meet.
  On the two designs that passed validation it raises 2 of 96 Checks, both
  one metric name used as a value, so it is returned once and never blocks.
- Reviews run together; the Jev coverage review stays off.

Plans record both steps and the cancelled run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 09:39:13 +01:00
5adc911a82 feat: keep an unchanged description, list conditions, review coverage with Jev
After case 02 (completed at 100%, but with two Revisions of the Target
architecture for gaps validation found one pass at a time):

- A Deliverable stands while what it was written from is unchanged. The
  architecture description is written from the Elements and Relationships,
  so a Revision that only adds or rewords Checks no longer runs the describer.
- Non-functional requirements and recommendations list their conditions,
  one entry per separate thing that must be true; submission requires them.
- A coverage review at submission (JEV_COVERAGE_REVIEW, off by default):
  for each condition, Jev chooses which Check verifies it, or none. Asked as
  one broad yes-or-no per requirement it could not tell a complete design
  from one with known gaps; asked as a choice per condition it raised all
  four known gaps of case 02 and nothing on the complete graph. An opt-in
  live test (LIVE_JEV=1) repeats that.
- The validator is told to report every gap in its first pass.

Plans record the case 02 run and both forms of the review with figures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:58:38 +01:00
f1f6f1b83a feat: load the env file at start-up, trim schema lookups, remove legacy samples
Start-up
- Export the env file's variables for other libraries (the Cloud provider's
  client) into the process environment; the app's own settings and secrets
  are never exported, and a value already in the environment wins.
- Use GOOGLE_GENAI_USE_ENTERPRISE in place of the deprecated
  GOOGLE_GENAI_USE_VERTEXAI.

Target architect
- Look each distinct resource type up once instead of once per setting: in
  the run of 2026-10-07 it made 39 lookups for 8 types and exhausted its
  budget. terraform_schema's find takes several names; budget 30 to 45.
- A cut-off schema outline says so.

Removed, as leftovers of the original pipeline that nothing read
- terraform/ (no longer passed terraform validate), scripts/validate.sh,
  tests/test_package.py and the CI steps that ran on them.
- The sample outputs under docs/ and requirements.yaml.
- CI now pins Terraform 1.14.3, as the container image does.

Plans record the case 01 run: completed at 100% in 18 minutes with no
Revision, against 60 minutes and warnings before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:06:58 +01:00
5bb22d984e fixed multile issues
Some checks failed
validation / verify (push) Failing after 8s
2026-10-07 17:33:36 +01:00
4c4d44ef87 fix: show the Orchestrator a validation only while it stands
After deployment guidance was reopened, the Orchestrator's context still
carried the previous validation's verdicts, so it reopened the code for the
same Check three times without a validation in between (evaluation run of
2026-10-07, case 01). Record that run in the iteration 6 plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:08:09 +01:00
e9246828b3 feat: establish the Target architecture as a graph before code
Design is now approved before any infrastructure code is written (ADR-0005).
A new target-architect Specialist produces one graph of Elements,
Relationships and Checks; the diagram is rendered from it, deployment
guidance must account for every Check, and validation gives a verdict per
Check. Requirements, components, products and recommendations carry ids in
data files, and a submission with a gap in the Requirements chain is refused.

Also in this commit:
- Customer cases group Executions, with an intent history per case
- lesson candidates are captured when an Execution ends
- evaluation gains chain, verdict, judged and interactive checks, and
  per-case thresholds
- Terraform providers are cached between checks
- the requirements analyst proposes defaults instead of asking
- the legacy-baseline case and the checks that only read it are removed
- ADR-0006 (Change requests) accepted; ADR-0004 and ADR-0007 proposed
- plans for iterations 4 to 7 under .agents/plans/

Unproven with a real model: no evaluation run has completed on this design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 13:59:09 +01:00
7ee49ab352 latest refactoring
Some checks failed
validation / verify (push) Failing after 10s
2026-10-06 20:22:37 +01:00
ad371b2ec4 feat: serve Executions over A2A with a pre-approved /generate
Increment 7 of .scratch/solution-architect-agent/spec.md.

- app/assembly.py: composition root building the engine from settings
  and the provider bundle (roles, ADK runtime, MCP catalog, Terraform
  tools, checkpointer, recorder).
- app/a2a: a2a-sdk 1.1 JSON-RPC server (v0.3 compatible) mapping tasks to
  Executions; questions, Approvals and permissions pause as
  input-required; Approvals accepted only as structured data; completed
  tasks return the guide and code as artifacts; agent card derived from
  the dependency graph.
- POST /generate runs a pre-approved Execution with the legacy keys.
- Specialist exceptions are relayed to the user; Orchestrator exceptions
  fail the task. Engine accepts a caller-chosen execution id.
- Dockerfile installs checksum-verified Terraform and ships agents,
  providers and skills; add mcp and a2a-sdk requirements.
- 17 new tests; full suite 206 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 10:23:24 +01:00
ed104e5697 feat: add local tools and read-only remote MCP servers
Increment 6 of .scratch/solution-architect-agent/spec.md, test-first.

- app/tools/mcp.py: MCP client and catalog exposing only allowlisted,
  readOnlyHint tools as framework-neutral ToolSpecs; unreachable servers
  and failed calls become discovery gaps; ADC auth with quota project.
- app/tools/local.py: Mermaid structure check and Terraform fmt,
  validate and plan in a scratch copy; apply is never constructed.
- Tool factory gives roles their named local tools and bundle MCP
  servers; pillar advisors get general-tier documentation tools.
- providers/gcp.yaml: 42 allowlisted tools reviewed against live
  tools/list of 12 Google Cloud MCP servers (snapshot in
  providers/reviews/), excluding data-reading and secret-capable tools.
- scripts/list-mcp-tools.py, docs/operations/mcp-access.md (IAM deny
  policy, service account roles), cloud-discovery and iac-validation
  Skills, MCP and Terraform settings, pytest asyncio auto mode.
- 31 new tests; full suite 189 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 10:14:05 +01:00
f040493583 feat: run role files on the ADK runtime with scoped Skills
Increment 5 of .scratch/solution-architect-agent/spec.md, test-first.

- agents/*.md: provider-neutral role files for the Orchestrator, nine
  Specialists and the pillar advisor, with a shared Specialist protocol.
- app/agents/roles.py: load and validate role files; Skills, MCP
  servers and models come from the provider bundle.
- app/agents/adk_runtime.py: AgentRuntime on google-adk 2.9 with a
  per-run LlmAgent, role-scoped SkillToolset (no run_skill_script),
  step budgets, early stop on turn-ending tools, LiteLLM models.
- app/execution/pillars.py + toolkit.py: design advisor fans out to one
  advisor per WAF pillar in parallel.
- Specialists receive every earlier-Phase Deliverable as input.
- Bundle gains role_skills, pillar_skills and -latest Gemini aliases;
  pin google-adk[extensions]==2.9.0.
- 37 new tests; full suite 158 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 09:51:02 +01:00
d9214c671a feat: add the hierarchical Execution engine on LangGraph
Increment 4 of .scratch/solution-architect-agent/spec.md, test-first.

- app/agents/runtime.py: framework-neutral AgentRuntime interface
  (RoleSpec, AgentTask, ToolSpec, outcomes); fake_runtime.py replays
  scripted tool calls through the real tools with step budgets.
- app/execution/graph.py: Orchestrator turn fans out to parallel
  Specialist tasks (Send), pauses with interrupt() for questions,
  Approvals and cloud-read permission, and finalizes or ends with
  warnings. Only the Orchestrator turn writes the Ledger.
- app/execution/tools.py: Orchestrator tools, Specialist submit and
  needs_input, and the cloud-read permission gate.
- app/execution/engine.py: ExecutionEngine.start/resume returning an
  Interaction; the single interface for A2A, /generate and eval.
- Recorders (in-memory, DatabaseManager), configurable checkpointer
  (memory, sqlite, postgres), engine settings, langgraph checkpoint
  dependencies.
- 19 new tests; full suite 121 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 09:38:13 +01:00
c8410946b8 feat: add the Deliverable dependency graph and Ledger
Increment 3 of .scratch/solution-architect-agent/spec.md, test-first.

- app/execution/deliverables.py: four Phases, nine Deliverables and
  their Dependencies declared once, plus per-kind contracts (required
  headings, attachments) checked by check_submission.
- app/execution/ledger.py: pure Ledger enforcing both acceptance
  thresholds, Phase Approvals (including pre-approved mode and change
  requests), Revision cascade with withdrawn Approvals, the revision
  cap with escalation and user override, attempt numbers that reject
  stale submissions, an ordered event log, and dict round-tripping.
- 37 new tests; full suite 102 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 18:28:04 +01:00
2976dfab21 feat: vendor upstream Skills and add the GCP provider bundle
Increment 2 of .scratch/solution-architect-agent/spec.md.

- Vendor google-cloud-solution-architecture and the six
  google-cloud-waf-* Skills from google/skills@81a31a6, unmodified,
  with LICENSE, SHA256SUMS and VENDOR.md; add
  scripts/sync-vendored-skills.sh to re-sync at a pinned sha.
- app/agent_skills.py: discover Skills and validate SKILL.md
  frontmatter against the Agent Skills specification.
- app/providers.py + providers/gcp.yaml: load and validate a Cloud
  provider bundle (Skills, output template, per-role models, tool
  tiers, remote MCP servers). MCP allowlists stay empty until the
  read-only review in increment 6.
- Settings CLOUD_PROVIDER and MCP_ENABLE_PREVIEW.
- 35 new tests; full suite 65 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 18:20:17 +01:00
69012b2656 chore: stop tracking __pycache__ bytecode
These 30 .pyc files were committed before .gitignore covered them and
changed on every test run. Files remain on disk; .gitignore already
excludes __pycache__/ and *.pyc.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-15 18:13:48 +01:00
a24a44e28c refactored: to utilise the google adk and production grade agent
Some checks failed
validation / verify (push) Failing after 10s
2026-09-02 21:42:10 +01:00
b9a924cf4a fix: refactored manually
Some checks failed
validation / verify (push) Failing after 15s
2026-09-02 16:08:12 +01:00
43f0f5bc79 decomposer: scaffold gcp_solution_architecture_agent 2026-09-01 19:47:04 +00:00