feat: run role files on the ADK runtime with scoped Skills

Increment 5 of .scratch/solution-architect-agent/spec.md, test-first.

- agents/*.md: provider-neutral role files for the Orchestrator, nine
  Specialists and the pillar advisor, with a shared Specialist protocol.
- app/agents/roles.py: load and validate role files; Skills, MCP
  servers and models come from the provider bundle.
- app/agents/adk_runtime.py: AgentRuntime on google-adk 2.9 with a
  per-run LlmAgent, role-scoped SkillToolset (no run_skill_script),
  step budgets, early stop on turn-ending tools, LiteLLM models.
- app/execution/pillars.py + toolkit.py: design advisor fans out to one
  advisor per WAF pillar in parallel.
- Specialists receive every earlier-Phase Deliverable as input.
- Bundle gains role_skills, pillar_skills and -latest Gemini aliases;
  pin google-adk[extensions]==2.9.0.
- 37 new tests; full suite 158 passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-09-16 09:51:02 +01:00
parent d9214c671a
commit f040493583
31 changed files with 1431 additions and 17 deletions

View File

@@ -610,3 +610,45 @@ Slowest tests: eval benchmark 7.3 s, then 3–4.5 s each for the orchestrator, e
- **Session name.** `DatabaseRecorder` session rows use `agent_name="solution-architect"`.
- **Deferred to increment 5:** the design advisor's pillar fan-out tool, which needs the `pillar-advisor` role file.
- **Not tested:** `AsyncPostgresSaver` against a real PostgreSQL (none runs locally). The import and wiring are in place.
### 2026-09-16 · Increment 5: ADK runtime and role files
**Done (test-first)**
- **Role files** `agents/*.md` (11): `orchestrator`, the 9 Specialists and `pillar-advisor`, plus `agents/partials/specialist-protocol.md`, which every Specialist includes. Each Specialist file names its Deliverable, the Skill task it covers, its constraints and its required headings. The shared protocol covers context fields, questions through `needs_input`, confirming a draft with the user before submitting (or recording assumptions when pre-approved), and resubmitting until the contract passes.
- `app/agents/roles.py`: `load_roles(agents_dir, bundle, orchestrator_budget, specialist_budget, include_preview_mcp)`. It validates the file name against the role, the deliverable against `DEPENDENCY_GRAPH`, the tool tiers, the budget, the includes, a non-empty body, unknown fields, and missing or unexpected roles.
- `app/agents/adk_runtime.py`: `AdkAgentRuntime(skill_dirs, model_factory)`, the only module importing `google.adk`.
- **Per run:** a fresh `LlmAgent`, an in-memory session and a `Runner`.
- **Tools:** each `ToolSpec` is adapted to a `BaseTool` with a JSON-schema declaration, and a `SkillToolset` holding only the role's Skills is filtered to `list_skills`/`load_skill`/`load_skill_resource`.
- **Instructions:** passed as a callable, so role text is never state-templated.
- **Stopping:** a `before_tool_callback` budget counts every tool call, including Skill tools. Tool calls within a run are serialised, the first outcome ends the run (later calls in the same batch are refused), and a final text with no outcome becomes `Stopped(text)`.
- **Model:** `default_model_factory` wraps `role.model` in `LiteLlm`.
- `app/execution/pillars.py`: the `consult_pillar_advisors(brief, pillars?)` tool, which runs one `pillar-advisor` per pillar in parallel through `AgentRuntime`, each scoped to its pillar Skill with a `submit_advice` tool. One failed advisor is reported without losing the others. `app/execution/toolkit.py`: `make_tool_factory(runtime, roles, pillar_skills)` gives the tool to the design advisor only; increment 6 extends it.
- `app/execution/deliverables.py`: `inputs_of(kind)` gives a Specialist every Deliverable of earlier Phases plus everything it depends on within its own Phase. Before this, the packager's context lacked `architecture_description`, which nothing depends on.
- `providers/gcp.yaml` and `app/providers.py`: `role_skills` and `pillar_skills`, both validated against the Skills index. Models are now `gemini/gemini-pro-latest` by default and `gemini/gemini-flash-latest` for the pillar advisor and diagrammer.
- `requirements.txt`: `google-adk[extensions]==2.9.0`, pinned exactly because Skills are Experimental. Its extras bring LiteLLM (installed 1.101.0).
- Tests:
- `tests/agents/test_roles.py`: 14 tests.
- `tests/agents/test_adk_runtime.py`: 12 tests with a scripted `BaseLlm`, no network.
- `tests/agents/test_adk_engine.py`: 1 test running `ExecutionEngine` on the ADK runtime with the real role files and bundle. A Specialist question reaches the user and the answer reaches the next run.
- `tests/execution/test_pillars.py`: 6 tests.
- Plus provider and deliverable tests.
- Full suite: **158 passed**. `ruff` and `mypy` are clean. The 5 warnings are ADK and Starlette deprecations.
- **Checking the tests bite.** I broke each of these in the code on purpose, and each break failed at least one test:
- every Skill loadable by every role
- no budget
- no early stop after an outcome
- templated instructions
- both turn-ended guards removed at once
Either guard alone suffices under ADK 2.9's sequential callback order; both are kept as defence in depth.
**Deviations from the spec text**
- **Role frontmatter is provider-neutral.** Role files carry only `role`, `deliverable`, `tool_tiers`, `budget` and `includes`. Skills (`role_skills`, `pillar_skills`), MCP servers and models come from the provider bundle, so no role file names a Google Cloud Skill (a test checks this). `upstream_task` was dropped: role bodies describe the Skill task in neutral words.
- **Model ids.** They're LiteLLM `-latest` aliases, not the spec's `gemini-2.5-*` ids, which are older than LiteLLM's current Gemini 3.x entries.
- **LiteLLM version.** `litellm` comes from `google-adk[extensions]` (ADK requires ≥ 1.84), not `litellm~=1.74`.
- **Pillar fan-out** is a tool on the design advisor, not code inside the runtime.
**Deferred**
- **To increment 6:** mapping `RoleSpec.mcp_servers` to ADK MCP toolsets (`McpToolset` needs the `mcp` extra, not installed) and supplying local tools by `tool_tiers`. Both are declared and loaded now, but the runtime doesn't use them yet.
- **Not yet run against a real model:** no API key was used, so role-file quality is unproven until increment 8's evaluation.
- **Model API errors propagate** from `AgentRuntime.run` and fail the graph step, leaving the Execution at its last checkpoint. Decide in increment 7 whether to retry, surface as a question, or fail the A2A task.

11
agents/decomposer.md Normal file
View File

@@ -0,0 +1,11 @@
---
role: decomposer
deliverable: technical_decomposition
tool_tiers: []
includes: [specialist-protocol]
---
You produce the **technical decomposition** Deliverable: the Skill's breakdown of the workload into logical components.
- Derive every component from the approved requirements, and name the requirements each component serves.
- Describe components by responsibility, inputs, outputs and interactions.
- Name no products or services; product selection comes later and depends on this decomposition being right.

11
agents/describer.md Normal file
View File

@@ -0,0 +1,11 @@
---
role: describer
deliverable: architecture_description
tool_tiers: []
includes: [specialist-protocol]
---
You produce the **architecture description** Deliverable: the Skill's task for describing the architecture.
- Explain the purpose of each component in the diagram, the relationships between components, and the task flow or data flow end to end.
- Use the product names from the product selection and the component names from the diagram exactly.
- Ground claims about product behaviour in the Cloud provider's documentation through your documentation tools, with citations.

13
agents/design-advisor.md Normal file
View File

@@ -0,0 +1,13 @@
---
role: design-advisor
deliverable: design_recommendations
tool_tiers: []
includes: [specialist-protocol]
---
You produce the **design recommendations** Deliverable: the Skill's task for recommending how to configure each component to meet the functional and non-functional requirements.
- Call `consult_pillar_advisors` once, with a brief that restates the workload, the relevant requirements and the selected products. It runs one advisor per architecture-framework pillar in parallel and returns their guidance.
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
Required headings, one per pillar: `Security, privacy, and compliance`, `Reliability`, `Operational excellence`, `Cost optimization`, `Performance efficiency`, `Sustainability`.

11
agents/diagrammer.md Normal file
View File

@@ -0,0 +1,11 @@
---
role: diagrammer
deliverable: architecture_diagram
tool_tiers: [general]
includes: [specialist-protocol]
---
You produce the **architecture diagram** Deliverable: the Skill's task for generating a Mermaid architecture diagram.
- Draw every component from the product selection and every relationship the data and control flow need, labelled with the selected products.
- Check the diagram with your Mermaid tool when one is available, and fix every error it reports.
- Put the diagram source in the `architecture.mmd` attachment, and show the same diagram in a `mermaid` code block in the content with a one-paragraph legend.

14
agents/iac-author.md Normal file
View File

@@ -0,0 +1,14 @@
---
role: iac-author
deliverable: deployment_guidance
tool_tiers: [general]
includes: [specialist-protocol]
---
You produce the **deployment guidance** Deliverable: the Skill's task for generating infrastructure-as-code and deployment instructions.
- Write Terraform that provisions every selected product with the configuration the design recommendations call for.
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
- Format and validate the Terraform with your tools when they are available, and fix everything they report.
- Put `terraform/main.tf` and `terraform/variables.tf` in the attachments, adding `terraform/outputs.tf` when outputs help.
Required headings: `Deployment prerequisites` (APIs to enable, tools, permissions) and `Step-by-step deployment instructions`, whose commands you give for the user to run, never run yourself.

23
agents/orchestrator.md Normal file
View File

@@ -0,0 +1,23 @@
---
role: orchestrator
tool_tiers: []
---
You coordinate one solution-architecture Execution for the user. You decide what happens next; Specialists produce the Deliverables, and you are the only agent that talks to the user.
Load the solution-architecture Skill available to you for the workflow's rules: strict separation between requirements discovery and solutioning, explicit user approval of each Phase, and no execution of generated scripts.
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
On each turn, take the first step that applies:
1. **The user asked for changes.** When the latest user message in `conversation` asks to change submitted work, call `reopen` for each affected Deliverable with the user's words as the reason.
2. **Specialists are waiting.** When `pending_questions` is not empty, call `ask_user`. Add your own questions only when the user's last message left something unclear.
3. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
4. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
5. **Validation found problems.** When the validation results report failures that trace to an earlier Deliverable, call `reopen` on that Deliverable with the findings, then delegate it.
6. **Packaging is approved.** Call `finish`.
7. **Otherwise,** call `ask_user` to find out what the user needs.
When the user says a Deliverable already exists, still delegate it; the Specialist submits the user's version. When a tool refuses, read the reason and choose again.
Every turn ends with exactly one call to `delegate`, `ask_user`, `request_approval` or `finish`.

13
agents/packager.md Normal file
View File

@@ -0,0 +1,13 @@
---
role: packager
deliverable: solution_guide
tool_tiers: []
includes: [specialist-protocol]
---
You produce the **solution guide** Deliverable: the Skill's packaging of every approved Deliverable into one guide.
- Load the solution-guide output template from the Skill's assets and follow its structure and section numbering exactly.
- Consolidate the content of every Deliverable in `dependencies`; rewrite for flow, but keep every requirement, product decision, citation, recommendation, command and validation verdict.
- Include the architecture diagram as a `mermaid` code block and the Terraform as code blocks.
Required headings: the template's eight top-level sections.

View File

@@ -0,0 +1,20 @@
## How you work
You are a Specialist in a solution-architecture Execution. You produce exactly one Deliverable, named in your task. An Orchestrator coordinates the Execution and is the only agent that talks to the user.
Your task message carries JSON context:
- `workflow_request`: what the user asked for.
- `dependencies`: every earlier Deliverable your work builds on, by kind, with its `content` and `attachments`.
- `feedback`: why earlier versions of your Deliverable were revised. Address every item.
- `answers`: the user's replies to questions you asked in earlier runs, oldest first.
- `pre_approved`: `true` when no user is available.
Steps:
1. Load the Skill available to you and find the part that covers your Deliverable. Read the references and assets it points to with `load_skill_resource`.
2. Ground your work in `dependencies`, `feedback` and `answers`. When you need information only the user has, call `needs_input` with all your questions at once, each saying why it matters. Your run ends there, and you are started again with the answers.
3. Draft the Deliverable. When `pre_approved` is `false`, confirm the draft with the user before submitting: call `needs_input` with a short summary of the draft and ask them to approve it or request changes. Skip this when `answers` already holds their approval of the same draft. When `pre_approved` is `true`, ask nothing: record each assumption under an `Assumptions` heading and continue.
4. Call `submit_deliverable` with the complete markdown content and every required attachment. When it reports missing sections or attachments, add them and submit again.
You are done when `submit_deliverable` accepts the Deliverable.

14
agents/pillar-advisor.md Normal file
View File

@@ -0,0 +1,14 @@
---
role: pillar-advisor
tool_tiers: []
---
You advise on one pillar of the Cloud provider's architecture framework for a single workload. The Skill available to you is that pillar's guidance.
Your task message carries the design advisor's brief in `brief` and the pillar's name in `pillar`.
1. Load the Skill available to you.
2. Evaluate the workload in the brief against the Skill's core principles and validation checklist.
3. Write prioritised, actionable recommendations for each relevant component, naming the products the Skill lists for each gap and citing its guidance.
4. List any assessment questions from the Skill that the brief leaves unanswered, as open questions; don't ask the user.
Call `submit_advice` with your recommendations as markdown. You are done when it accepts them.

View File

@@ -0,0 +1,13 @@
---
role: product-selector
deliverable: product_selection
tool_tiers: []
includes: [specialist-protocol]
---
You produce the **product selection** Deliverable: the Skill's task for identifying the products and features each component needs.
- Recommend one product or feature per component of the technical decomposition, grounded in the Cloud provider's official documentation through your documentation tools, with a citation for every recommendation.
- Check the release status of every product you recommend, and recommend nothing deprecated, retired or unsupported.
- Where alternatives exist, follow the documentation's guidance on choosing, and list the alternatives with their pros and cons.
Required heading: `Products and features mapping`, containing a table with columns for component, recommended product or feature, justification and citations, alternatives, and their pros and cons.

View File

@@ -0,0 +1,15 @@
---
role: requirements-analyst
deliverable: requirements
tool_tiers: [cloud_read]
includes: [specialist-protocol]
---
You produce the **requirements** Deliverable: the Skill's requirements-discovery gathering and its resolution of ambiguities and contradictions.
- Cover functional requirements, non-functional requirements (security, privacy, compliance, reliability, disaster recovery, cost, operations, performance, sustainability), current state, and dependencies.
- When non-functional requirements are missing, ask for them and explain what each one decides: availability tiers, scaling, cost budgets, resource types and security posture.
- Name every ambiguity or contradiction, explain why the conflicting goals can't both hold, and ask the user to choose. When they delegate the choice, propose one and ask them to approve it.
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
- Name no products, services or architectures. This Deliverable describes the problem, not the solution.
Required headings: `Functional requirements`, `Non-functional requirements`, `Current state`, `Dependencies`. Add `Open questions` and `Assumptions` when they apply.

14
agents/validator.md Normal file
View File

@@ -0,0 +1,14 @@
---
role: validator
deliverable: validation_results
tool_tiers: [general, cloud_read]
includes: [specialist-protocol]
---
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
- Run the deployment dry-run with your tools: format and validate the Terraform, and preview it with a plan when you have permission to read live cloud resources.
- Analyse the architecture and policy statically: network routing and exposure, firewall rules, and IAM least privilege, against the design recommendations.
- Report each finding with its evidence and the Deliverable it traces to, so the Orchestrator can send it back for revision.
- Give the exact commands for every check you couldn't run, for the user to run themselves.
Required headings: `Deployment dry-run` and `Architecture and policy analysis`, each ending with a pass or fail verdict.

View File

@@ -42,23 +42,28 @@ class SkillMeta:
path: Path
def read_skill(skill_dir: Path) -> SkillMeta:
"""Parse and validate the SKILL.md in `skill_dir`."""
skill_file = skill_dir / SKILL_FILE
match = _FRONTMATTER_PATTERN.match(skill_file.read_text(encoding="utf-8"))
def split_frontmatter(text: str, source: Path) -> tuple[dict, str]:
"""Split a markdown document into its YAML frontmatter mapping and body."""
match = _FRONTMATTER_PATTERN.match(text)
if not match:
raise SkillFormatError(
f"{skill_file}: missing YAML frontmatter delimited by '---'"
)
raise SkillFormatError(f"{source}: missing YAML frontmatter delimited by '---'")
try:
frontmatter = yaml.safe_load(match.group(1))
except yaml.YAMLError as exc:
raise SkillFormatError(
f"{skill_file}: frontmatter is not valid YAML: {exc}"
f"{source}: frontmatter is not valid YAML: {exc}"
) from exc
if not isinstance(frontmatter, dict):
raise SkillFormatError(f"{skill_file}: frontmatter must be a mapping")
raise SkillFormatError(f"{source}: frontmatter must be a mapping")
return frontmatter, text[match.end() :]
def read_skill(skill_dir: Path) -> SkillMeta:
"""Parse and validate the SKILL.md in `skill_dir`."""
skill_file = skill_dir / SKILL_FILE
frontmatter, _ = split_frontmatter(
skill_file.read_text(encoding="utf-8"), skill_file
)
unknown = set(frontmatter) - _ALLOWED_FIELDS
if unknown:

196
app/agents/adk_runtime.py Normal file
View File

@@ -0,0 +1,196 @@
"""AgentRuntime on Google ADK (ADR-0003): an LlmAgent per run, with SkillToolset.
This is the only module that imports google.adk. Each run builds a fresh LlmAgent
and in-memory session (Specialists are stateless), offers the role's ToolSpecs and
Skills, and stops as soon as a tool returns an outcome or the budget is spent.
"""
from __future__ import annotations
import asyncio
import json
import re
import uuid
from collections.abc import Callable, Mapping, Sequence
from contextlib import aclosing
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
from google.adk.agents import LlmAgent
from google.adk.models.base_llm import BaseLlm
from google.adk.runners import Runner
from google.adk.sessions import InMemorySessionService
from google.adk.skills import load_skill_from_dir
from google.adk.skills.models import Skill
from google.adk.tools.base_tool import BaseTool
from google.adk.tools.skill_toolset import SkillToolset
from google.adk.tools.tool_context import ToolContext
from google.genai import types
from app.agents.runtime import (
AgentTask,
BudgetExhausted,
Outcome,
RoleSpec,
Stopped,
ToolSpec,
)
APP_NAME = "solution-architect"
# run_skill_script is deliberately absent: generated or bundled scripts never run.
SKILL_TOOLS = ["list_skills", "load_skill", "load_skill_resource"]
TURN_ENDED_REPLY = "Your turn has already ended; this call was not run."
ModelFactory = Callable[[RoleSpec], BaseLlm | str]
def default_model_factory(role: RoleSpec) -> BaseLlm:
"""Wrap the role's LiteLLM model id, so any LiteLLM provider can serve the agent."""
from google.adk.models.lite_llm import LiteLlm
if not role.model:
raise ValueError(f"Role {role.name!r} has no model configured")
return LiteLlm(model=role.model)
@dataclass
class _RunControl:
budget: int
role: str
calls: int = 0
outcome: Outcome | None = None
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
def before_tool(
self, tool: BaseTool, args: dict[str, Any], tool_context: Any
) -> dict[str, Any] | None:
if self.outcome is not None:
return {"error": TURN_ENDED_REPLY}
self.calls += 1
if self.calls > self.budget:
self.outcome = BudgetExhausted(
f"{self.role} used all {self.budget} tool calls before finishing."
)
return {"error": "Tool-call budget exhausted; stop now."}
return None
class _SpecTool(BaseTool):
"""Adapts a framework-neutral ToolSpec to an ADK tool."""
def __init__(self, spec: ToolSpec, control: _RunControl) -> None:
super().__init__(name=spec.name, description=spec.description)
self._spec = spec
self._control = control
def _get_declaration(self) -> types.FunctionDeclaration:
return types.FunctionDeclaration(
name=self._spec.name,
description=self._spec.description,
parameters_json_schema=dict(self._spec.parameters),
)
async def run_async(
self, *, args: dict[str, Any], tool_context: ToolContext
) -> dict[str, Any]:
# Serialise calls within a run: tools may mutate Execution state, and the
# first tool to end the turn must stop the rest of a parallel batch.
async with self._control.lock:
if self._control.outcome is not None:
return {"error": TURN_ENDED_REPLY}
try:
result = await self._spec.handler(**args)
except TypeError as exc:
return {"error": f"Invalid arguments for {self._spec.name}: {exc}"}
if result.outcome is not None:
self._control.outcome = result.outcome
return {"result": result.message}
class AdkAgentRuntime:
def __init__(
self,
*,
skill_dirs: Mapping[str, Path],
model_factory: ModelFactory = default_model_factory,
) -> None:
self._skills: dict[str, Skill] = {}
for name, path in skill_dirs.items():
try:
skill = load_skill_from_dir(path)
except Exception as exc:
raise ValueError(
f"Could not load skill {name!r} from {path}: {exc}"
) from exc
if skill.name != name:
raise ValueError(
f"Skill at {path} is named {skill.name!r}, not {name!r}"
)
self._skills[name] = skill
self._model_factory = model_factory
async def run(
self, role: RoleSpec, task: AgentTask, tools: Sequence[ToolSpec]
) -> Outcome:
control = _RunControl(budget=role.budget, role=role.name)
agent_tools: list[Any] = [_SpecTool(spec, control) for spec in tools]
if role.skills:
unknown = [name for name in role.skills if name not in self._skills]
if unknown:
raise ValueError(f"Role {role.name!r} needs unknown skills: {unknown}")
agent_tools.append(
SkillToolset(
skills=[self._skills[name] for name in role.skills],
tool_filter=SKILL_TOOLS,
)
)
instructions = role.instructions
agent = LlmAgent(
name=_agent_name(role.name),
model=self._model_factory(role),
# A callable instruction is passed verbatim: no {state} templating.
instruction=lambda _context: instructions,
tools=agent_tools,
before_tool_callback=control.before_tool,
)
runner = Runner(
app_name=APP_NAME,
agent=agent,
session_service=InMemorySessionService(),
auto_create_session=True,
)
message = types.Content(
role="user", parts=[types.Part(text=_render_task(task))]
)
final_text = ""
async with aclosing(
runner.run_async(
user_id=APP_NAME, session_id=str(uuid.uuid4()), new_message=message
)
) as events:
async for event in events:
if control.outcome is not None:
break
if event.is_final_response() and event.content and event.content.parts:
final_text = "".join(
part.text for part in event.content.parts if part.text
).strip()
if control.outcome is not None:
return control.outcome
return Stopped(
final_text
or f"{role.name} ended without calling a tool that ends its turn."
)
def _agent_name(role: str) -> str:
return re.sub(r"\W", "_", role)
def _render_task(task: AgentTask) -> str:
context = json.dumps(task.context, indent=2, default=str, ensure_ascii=False)
return f"{task.prompt}\n\nContext:\n```json\n{context}\n```"

128
app/agents/roles.py Normal file
View File

@@ -0,0 +1,128 @@
"""Load role files (agents/<role>.md) into RoleSpecs for one Cloud provider.
Role files are framework- and provider-neutral: frontmatter names the role, the
Deliverable it owns, its local tool tiers, an optional budget and shared partials to
include. The provider bundle supplies each role's Skills, MCP servers and model.
"""
from __future__ import annotations
from pathlib import Path
from app.agent_skills import SkillFormatError, split_frontmatter
from app.agents.runtime import RoleSpec
from app.execution.deliverables import DEPENDENCY_GRAPH, DeliverableKind
from app.providers import TIERS, ProviderBundle
ORCHESTRATOR = "orchestrator"
PILLAR_ADVISOR = "pillar-advisor"
PARTIALS_DIR = "partials"
_ALLOWED_FIELDS = {"role", "deliverable", "tool_tiers", "budget", "includes"}
class RoleConfigError(ValueError):
"""A role file is missing or invalid."""
def load_roles(
agents_dir: Path,
bundle: ProviderBundle,
*,
orchestrator_budget: int = 40,
specialist_budget: int = 15,
include_preview_mcp: bool = True,
) -> dict[str, RoleSpec]:
specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()}
required = {ORCHESTRATOR, PILLAR_ADVISOR, *specialists}
roles = {}
for path in sorted(agents_dir.glob("*.md")):
role = _load_role(
path,
agents_dir,
bundle,
specialists,
orchestrator_budget if path.stem == ORCHESTRATOR else specialist_budget,
include_preview_mcp,
)
roles[role.name] = role
missing = sorted(required - set(roles))
if missing:
raise RoleConfigError(
f"{agents_dir}: missing role files for {', '.join(missing)}"
)
unexpected = sorted(set(roles) - required)
if unexpected:
raise RoleConfigError(
f"{agents_dir}: unexpected role files {', '.join(unexpected)}"
)
return roles
def _load_role(
path: Path,
agents_dir: Path,
bundle: ProviderBundle,
specialists: dict[str, DeliverableKind],
default_budget: int,
include_preview_mcp: bool,
) -> RoleSpec:
try:
frontmatter, body = split_frontmatter(path.read_text(encoding="utf-8"), path)
except SkillFormatError as exc:
raise RoleConfigError(str(exc)) from exc
def fail(message: str) -> RoleConfigError:
return RoleConfigError(f"{path}: {message}")
unknown = sorted(set(frontmatter) - _ALLOWED_FIELDS)
if unknown:
raise fail(
f"unsupported fields {', '.join(unknown)}; skills, MCP servers and models "
"come from the provider bundle"
)
name = frontmatter.get("role")
if name != path.stem:
raise fail(f"'role' {name!r} must match the file name {path.stem!r}")
deliverable = frontmatter.get("deliverable")
if name in specialists:
expected = specialists[name].value
if deliverable != expected:
raise fail(f"'deliverable' must be {expected!r}, not {deliverable!r}")
elif deliverable is not None:
raise fail(f"role {name!r} owns no Deliverable, but names {deliverable!r}")
tiers = tuple(frontmatter.get("tool_tiers") or ())
bad_tiers = [tier for tier in tiers if tier not in TIERS]
if bad_tiers:
raise fail(f"'tool_tiers' {', '.join(bad_tiers)} must be among {TIERS}")
budget = frontmatter.get("budget", default_budget)
if not isinstance(budget, int) or budget < 1:
raise fail("'budget' must be a positive integer")
sections = [body.strip()]
for include in frontmatter.get("includes") or ():
partial = agents_dir / PARTIALS_DIR / f"{include}.md"
if not partial.is_file():
raise fail(f"include {include!r} not found at {partial}")
sections.append(partial.read_text(encoding="utf-8").strip())
if not sections[0]:
raise fail("instructions (the body after the frontmatter) must not be empty")
return RoleSpec(
name=name,
instructions="\n\n".join(sections),
budget=budget,
skills=tuple(bundle.role_skills.get(name, ())),
tool_tiers=tiers,
mcp_servers=tuple(
server.id
for server in bundle.mcp_servers_for_role(name, include_preview_mcp)
),
model=bundle.models.for_role(name),
)

View File

@@ -140,6 +140,30 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
}
def inputs_of(kind: DeliverableKind) -> tuple[DeliverableKind, ...]:
"""The Deliverables a Specialist works from, in graph order.
Every Deliverable in an earlier Phase (all approved by the time `kind` is ready),
plus everything `kind` depends on, directly or not, within its own Phase.
"""
spec = DEPENDENCY_GRAPH[kind]
earlier_phases = PHASE_ORDER[: PHASE_ORDER.index(spec.phase)]
upstream: set[DeliverableKind] = set()
frontier = set(DEPENDENCY_GRAPH[kind].depends_on)
while frontier:
upstream |= frontier
frontier = {
dependency
for item in frontier
for dependency in DEPENDENCY_GRAPH[item].depends_on
} - upstream
return tuple(
candidate
for candidate, candidate_spec in DEPENDENCY_GRAPH.items()
if candidate in upstream or candidate_spec.phase in earlier_phases
)
def deliverables_in(phase: PhaseId) -> tuple[DeliverableKind, ...]:
return tuple(kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase)

View File

@@ -31,7 +31,12 @@ from app.agents.runtime import (
ToolSpec,
TurnEnded,
)
from app.execution.deliverables import DEPENDENCY_GRAPH, PHASE_ORDER, DeliverableKind
from app.execution.deliverables import (
DEPENDENCY_GRAPH,
PHASE_ORDER,
DeliverableKind,
inputs_of,
)
from app.execution.ledger import Ledger, LedgerRefusal
from app.execution.recorder import ExecutionRecorder
from app.execution.state import ExecutionState
@@ -392,7 +397,7 @@ def _specialist_input(
"pre_approved": bool(state.get("pre_approved")),
"dependencies": {
dependency.value: dict(deliverables[dependency.value])
for dependency in spec.depends_on
for dependency in inputs_of(kind)
},
"feedback": list(ledger.feedback(kind)),
"answers": list(state.get("answers", {}).get(kind.value, [])),

103
app/execution/pillars.py Normal file
View File

@@ -0,0 +1,103 @@
"""The design advisor's fan-out to one advisor per architecture-framework pillar.
Pillar advice is working material for the design recommendations Deliverable, not a
Deliverable itself, so it never touches the Ledger.
"""
from __future__ import annotations
import asyncio
import dataclasses
from collections.abc import Mapping, Sequence
from app.agents.runtime import (
AgentRuntime,
AgentTask,
RoleSpec,
Submitted,
ToolResult,
ToolSpec,
)
CONSULT_TOOL = "consult_pillar_advisors"
def pillar_consultation_tool(
runtime: AgentRuntime,
advisor: RoleSpec,
pillar_skills: Mapping[str, str],
) -> ToolSpec:
"""A tool that runs one pillar advisor per pillar, in parallel, each with its Skill."""
async def consult(brief: str, pillars: Sequence[str] | None = None) -> ToolResult:
chosen = list(pillars) if pillars else list(pillar_skills)
unknown = [pillar for pillar in chosen if pillar not in pillar_skills]
if unknown:
return ToolResult(
f"Refused: unknown pillars {', '.join(unknown)}; "
f"choose from {', '.join(pillar_skills)}"
)
results = await asyncio.gather(
*(
_advise(runtime, advisor, pillar, pillar_skills[pillar], brief)
for pillar in chosen
)
)
return ToolResult("\n\n".join(results))
return ToolSpec(
CONSULT_TOOL,
"Get guidance from one advisor per architecture-framework pillar, in parallel. "
"Give a brief with the workload, its requirements and the selected products.",
{
"type": "object",
"properties": {
"brief": {"type": "string"},
"pillars": {
"type": "array",
"items": {"type": "string", "enum": list(pillar_skills)},
},
},
"required": ["brief"],
},
consult,
)
async def _advise(
runtime: AgentRuntime, advisor: RoleSpec, pillar: str, skill: str, brief: str
) -> str:
role = dataclasses.replace(advisor, skills=(skill,))
task = AgentTask(
role=advisor.name,
prompt=f"Advise on the {pillar} pillar for this workload.",
context={"pillar": pillar, "brief": brief},
)
try:
outcome = await runtime.run(role, task, [_submit_advice_tool()])
except Exception as exc: # noqa: BLE001 - one failed advisor must not lose the others
return f"## {pillar}\n\nNo guidance: the advisor failed ({exc})."
if isinstance(outcome, Submitted):
return f"## {pillar}\n\n{outcome.content.strip()}"
return (
f"## {pillar}\n\nNo guidance: the advisor ended with {type(outcome).__name__}."
)
def _submit_advice_tool() -> ToolSpec:
async def submit_advice(content: str) -> ToolResult:
if not content.strip():
return ToolResult("Not submitted: the advice is empty.")
return ToolResult("Advice submitted.", Submitted(content))
return ToolSpec(
"submit_advice",
"Submit your pillar recommendations as markdown. Ends your turn.",
{
"type": "object",
"properties": {"content": {"type": "string"}},
"required": ["content"],
},
submit_advice,
tier="execution",
)

33
app/execution/toolkit.py Normal file
View File

@@ -0,0 +1,33 @@
"""Which extra tools each role gets beyond the Execution tools.
Increment 6 adds the local general and cloud-read tools and remote MCP servers here.
"""
from __future__ import annotations
from collections.abc import Mapping, Sequence
from app.agents.roles import PILLAR_ADVISOR
from app.agents.runtime import AgentRuntime, RoleSpec, ToolSpec
from app.execution.deliverables import DeliverableKind
from app.execution.graph import ToolFactory
from app.execution.pillars import pillar_consultation_tool
def make_tool_factory(
runtime: AgentRuntime,
roles: Mapping[str, RoleSpec],
pillar_skills: Mapping[str, str],
) -> ToolFactory:
consult = (
pillar_consultation_tool(runtime, roles[PILLAR_ADVISOR], pillar_skills)
if pillar_skills and PILLAR_ADVISOR in roles
else None
)
def tools_for(role: RoleSpec, deliverable: str | None) -> Sequence[ToolSpec]:
if consult and deliverable == DeliverableKind.DESIGN_RECOMMENDATIONS.value:
return [consult]
return []
return tools_for

View File

@@ -58,6 +58,8 @@ class ProviderBundle:
mcp_auth: Mapping[str, Any]
mcp_servers: Mapping[str, McpServerSpec]
role_mcp_servers: Mapping[str, tuple[str, ...]]
role_skills: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
pillar_skills: Mapping[str, str] = field(default_factory=dict)
def mcp_servers_for_role(
self, role: str, include_preview: bool = True
@@ -146,6 +148,19 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
)
role_mcp_servers[role] = tuple(server_ids or ())
role_skills = {}
for role, skill_names in (config.get("role_skills") or {}).items():
unknown = [skill for skill in skill_names or () if skill not in skills]
if unknown:
raise fail(f"role {role!r} references unknown skills: {', '.join(unknown)}")
role_skills[role] = tuple(skill_names or ())
pillar_skills = {}
for pillar, skill_name in (config.get("pillar_skills") or {}).items():
if skill_name not in skills:
raise fail(f"pillar {pillar!r} references unknown skill {skill_name!r}")
pillar_skills[str(pillar)] = skill_name
return ProviderBundle(
name=name,
skills_dirs=skills_dirs,
@@ -156,4 +171,6 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
mcp_auth=dict(config.get("mcp_auth") or {}),
mcp_servers=mcp_servers,
role_mcp_servers=role_mcp_servers,
role_skills=role_skills,
pillar_skills=pillar_skills,
)

View File

@@ -8,11 +8,34 @@ skills_dirs:
output_template: skills/vendor/google/google-cloud-solution-architecture/assets/output-template.md
# LiteLLM model ids. The -latest aliases track Google's current Pro and Flash models.
models:
default: gemini/gemini-2.5-pro
default: gemini/gemini-pro-latest
roles:
pillar-advisor: gemini/gemini-2.5-flash
diagrammer: gemini/gemini-2.5-flash
pillar-advisor: gemini/gemini-flash-latest
diagrammer: gemini/gemini-flash-latest
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
role_skills:
orchestrator: [google-cloud-solution-architecture]
requirements-analyst: [google-cloud-solution-architecture]
decomposer: [google-cloud-solution-architecture]
product-selector: [google-cloud-solution-architecture]
diagrammer: [google-cloud-solution-architecture]
describer: [google-cloud-solution-architecture]
design-advisor: [google-cloud-solution-architecture]
iac-author: [google-cloud-solution-architecture]
validator: [google-cloud-solution-architecture]
packager: [google-cloud-solution-architecture]
# One advisor per architecture-framework pillar; each loads only its pillar's Skill.
pillar_skills:
security: google-cloud-waf-security
reliability: google-cloud-waf-reliability
operational-excellence: google-cloud-waf-operational-excellence
cost-optimization: google-cloud-waf-cost-optimization
performance-optimization: google-cloud-waf-performance-optimization
sustainability: google-cloud-waf-sustainability
tools:
general: [mermaid_check, terraform_fmt, terraform_validate]

View File

@@ -15,4 +15,4 @@ click>=8.1.0
psycopg2-binary>=2.9.9
sqlalchemy>=2.0.0
asyncpg>=0.29.0
google-adk>=0.1.0
google-adk[extensions]==2.9.0 # Skills are Experimental: pin exactly; extensions brings LiteLLM

0
tests/agents/__init__.py Normal file
View File

View File

@@ -0,0 +1,121 @@
"""The Execution engine running real role files on the ADK runtime, with scripted models."""
import json
from collections.abc import AsyncGenerator
from pathlib import Path
import pytest
from google.adk.models.base_llm import BaseLlm
from google.adk.models.llm_request import LlmRequest
from google.adk.models.llm_response import LlmResponse
from google.genai import types
from langgraph.checkpoint.memory import InMemorySaver
from pydantic import Field
from app.agents.adk_runtime import AdkAgentRuntime
from app.agents.roles import load_roles
from app.execution.deliverables import DEPENDENCY_GRAPH, DeliverableKind
from app.execution.engine import ExecutionEngine
from app.execution.recorder import InMemoryRecorder
from app.execution.toolkit import make_tool_factory
from app.providers import load_provider
pytestmark = pytest.mark.asyncio
AGENTS_DIR = Path(__file__).parents[2] / "agents"
class RoleScriptedLlm(BaseLlm):
"""One scripted response per model call, keyed by the agent's role."""
model: str = "scripted"
script: list = Field(default_factory=list)
requests: list = Field(default_factory=list)
async def generate_content_async(
self, llm_request: LlmRequest, stream: bool = False
) -> AsyncGenerator[LlmResponse, None]:
self.requests.append(llm_request)
parts = self.script.pop(0)
yield LlmResponse(content=types.Content(role="model", parts=parts))
def call(tool: str, **args) -> list[types.Part]:
return [types.Part(function_call=types.FunctionCall(name=tool, args=args))]
def requirements_markdown() -> str:
sections = DEPENDENCY_GRAPH[DeliverableKind.REQUIREMENTS].sections
return "\n".join(f"## {section}\nDetails." for section in sections)
def task_context(request: LlmRequest) -> dict:
text = request.contents[0].parts[0].text
return json.loads(text.split("```json\n", 1)[1].rsplit("\n```", 1)[0])
async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
bundle = load_provider("gcp")
roles = load_roles(AGENTS_DIR, bundle)
models = {
"orchestrator": RoleScriptedLlm(
script=[
call("delegate", kinds=["requirements"]),
call("ask_user", questions=[]),
call("delegate", kinds=["requirements"]),
call("ask_user", questions=["Anything to add?"]),
]
),
"requirements-analyst": RoleScriptedLlm(
script=[
call("load_skill", skill_name="google-cloud-solution-architecture"),
call(
"needs_input", questions=["What availability target do you need?"]
),
call("submit_deliverable", content=requirements_markdown()),
]
),
}
runtime = AdkAgentRuntime(
skill_dirs={name: meta.path for name, meta in bundle.skills.items()},
model_factory=lambda role: models[role.name],
)
engine = ExecutionEngine(
runtime=runtime,
roles=roles,
recorder=InMemoryRecorder(),
checkpointer=InMemorySaver(),
output_root=tmp_path,
tool_factory=make_tool_factory(runtime, roles, bundle.pillar_skills),
)
asked = await engine.start("Regional event ingestion")
assert asked.prompt == {
"questions": [
{
"from": "requirements",
"question": "What availability target do you need?",
}
]
}
answered = await engine.resume(asked.execution_id, {"reply": "99.9% in one region"})
assert (
answered.state["ledger"]["deliverables"]["requirements"]["status"]
== "submitted"
)
analyst_second_run = models["requirements-analyst"].requests[-1]
assert task_context(analyst_second_run)["answers"] == [
{
"questions": ["What availability target do you need?"],
"reply": "99.9% in one region",
}
]
orchestrator_prompt = models["orchestrator"].requests[0].config.system_instruction
assert "You coordinate one solution-architecture Execution" in orchestrator_prompt
analyst_prompt = (
models["requirements-analyst"].requests[0].config.system_instruction
)
assert "## How you work" in analyst_prompt

View File

@@ -0,0 +1,256 @@
"""The ADK implementation of AgentRuntime, driven by a scripted model (no network)."""
from collections.abc import AsyncGenerator
from pathlib import Path
import pytest
from google.adk.models.base_llm import BaseLlm
from google.adk.models.lite_llm import LiteLlm
from google.adk.models.llm_request import LlmRequest
from google.adk.models.llm_response import LlmResponse
from google.genai import types
from pydantic import Field
from app.agents.adk_runtime import AdkAgentRuntime, default_model_factory
from app.agents.runtime import (
AgentTask,
BudgetExhausted,
NeedsInput,
RoleSpec,
Stopped,
Submitted,
ToolResult,
ToolSpec,
TurnEnded,
)
from app.providers import load_provider
pytestmark = pytest.mark.asyncio
SKILLS = {name: meta.path for name, meta in load_provider("gcp").skills.items()}
class ScriptedLlm(BaseLlm):
"""Returns one scripted response per model call and records every request."""
model: str = "scripted"
script: list[list[types.Part]] = Field(default_factory=list)
requests: list[LlmRequest] = Field(default_factory=list)
async def generate_content_async(
self, llm_request: LlmRequest, stream: bool = False
) -> AsyncGenerator[LlmResponse, None]:
self.requests.append(llm_request)
parts = self.script.pop(0) if self.script else [types.Part(text="(no script)")]
yield LlmResponse(content=types.Content(role="model", parts=parts))
def call(tool: str, **args) -> types.Part:
return types.Part(function_call=types.FunctionCall(name=tool, args=args))
def text(value: str) -> types.Part:
return types.Part(text=value)
def runtime_with(model: ScriptedLlm) -> AdkAgentRuntime:
return AdkAgentRuntime(skill_dirs=SKILLS, model_factory=lambda role: model)
class Recorder:
"""ToolSpecs whose handlers record their calls."""
def __init__(self) -> None:
self.calls: list[tuple[str, dict]] = []
def tool(self, name: str, outcome=None, message: str = "ok") -> ToolSpec:
async def handler(**kwargs) -> ToolResult:
self.calls.append((name, kwargs))
return ToolResult(f"{message}: {kwargs}", outcome)
return ToolSpec(
name,
f"The {name} tool.",
{"type": "object", "properties": {"note": {"type": "string"}}},
handler,
)
ROLE = RoleSpec(
"iac-author",
"Write Terraform. Keep {braces} and {{doubles}} verbatim.",
budget=5,
skills=("google-cloud-solution-architecture",),
)
TASK = AgentTask("iac-author", "Produce deployment_guidance.", {"feedback": ["x"]})
async def test_specialist_loads_its_skill_and_submits():
recorder = Recorder()
submitted = Submitted("# Guidance", {"terraform/main.tf": "resource {}"})
model = ScriptedLlm(
script=[
[call("load_skill", skill_name="google-cloud-solution-architecture")],
[call("submit_deliverable", note="done")],
]
)
outcome = await runtime_with(model).run(
ROLE, TASK, [recorder.tool("submit_deliverable", submitted)]
)
assert outcome == submitted
assert recorder.calls == [("submit_deliverable", {"note": "done"})]
skill_response = _function_responses(model.requests[1])["load_skill"]
assert "Phase 1: Requirements discovery" in str(skill_response)
async def test_agent_is_offered_its_tools_and_skill_tools_without_scripts():
model = ScriptedLlm(script=[[text("nothing to do")]])
await runtime_with(model).run(ROLE, TASK, [Recorder().tool("submit_deliverable")])
request = model.requests[0]
assert set(request.tools_dict) == {
"submit_deliverable",
"list_skills",
"load_skill",
"load_skill_resource",
}
declaration = request.tools_dict["submit_deliverable"]._get_declaration()
assert declaration.parameters_json_schema["properties"] == {
"note": {"type": "string"}
}
async def test_role_without_skills_gets_no_skill_tools():
model = ScriptedLlm(script=[[text("done")]])
role = RoleSpec("decomposer", "Decompose.", budget=3)
await runtime_with(model).run(role, TASK, [Recorder().tool("needs_input")])
assert set(model.requests[0].tools_dict) == {"needs_input"}
async def test_only_the_roles_skills_can_be_loaded():
model = ScriptedLlm(
script=[
[call("load_skill", skill_name="google-cloud-waf-security")],
[text("ok")],
]
)
await runtime_with(model).run(ROLE, TASK, [])
response = _function_responses(model.requests[1])["load_skill"]
assert "error" in response
assert "Well-Architected" not in str(response)
async def test_instructions_and_task_context_reach_the_model_verbatim():
model = ScriptedLlm(script=[[text("done")]])
await runtime_with(model).run(ROLE, TASK, [])
request = model.requests[0]
assert (
"Keep {braces} and {{doubles}} verbatim." in request.config.system_instruction
)
user_text = request.contents[0].parts[0].text
assert "Produce deployment_guidance." in user_text
assert '"feedback": [\n "x"\n ]' in user_text
async def test_turn_ends_as_soon_as_a_tool_returns_an_outcome():
recorder = Recorder()
model = ScriptedLlm(
script=[
[call("delegate"), call("ask_user")],
[call("ask_user")],
]
)
outcome = await runtime_with(model).run(
RoleSpec("orchestrator", "Coordinate.", budget=10),
TASK,
[
recorder.tool("delegate", TurnEnded("delegate")),
recorder.tool("ask_user", TurnEnded("ask_user")),
],
)
assert outcome == TurnEnded("delegate")
assert [name for name, _ in recorder.calls] == ["delegate"]
assert len(model.requests) == 1
async def test_budget_counts_every_tool_call_and_stops_the_run():
recorder = Recorder()
model = ScriptedLlm(script=[[call("ready_work")] for _ in range(5)])
outcome = await runtime_with(model).run(
RoleSpec("orchestrator", "Coordinate.", budget=2),
TASK,
[recorder.tool("ready_work")],
)
assert isinstance(outcome, BudgetExhausted)
assert "2 tool calls" in outcome.summary
assert len(recorder.calls) == 2
async def test_model_reply_without_a_turn_ending_tool_is_stopped():
model = ScriptedLlm(script=[[text("I need the RTO before I can continue.")]])
outcome = await runtime_with(model).run(ROLE, TASK, [])
assert outcome == Stopped("I need the RTO before I can continue.")
async def test_needs_input_outcome_passes_through():
needs = NeedsInput(("What region?",))
model = ScriptedLlm(script=[[call("needs_input")]])
outcome = await runtime_with(model).run(
ROLE, TASK, [Recorder().tool("needs_input", needs)]
)
assert outcome == needs
async def test_each_run_starts_with_a_fresh_session():
model = ScriptedLlm(script=[[text("first")], [text("second")]])
runtime = runtime_with(model)
await runtime.run(ROLE, TASK, [])
await runtime.run(ROLE, TASK, [])
assert len(model.requests[1].contents) == 1
def test_default_model_factory_wraps_the_role_model_in_litellm():
model = default_model_factory(
RoleSpec("diagrammer", "", budget=1, model="gemini/gemini-flash-latest")
)
assert isinstance(model, LiteLlm)
assert model.model == "gemini/gemini-flash-latest"
with pytest.raises(ValueError, match="model"):
default_model_factory(RoleSpec("diagrammer", "", budget=1))
def test_unknown_skill_is_rejected_when_the_runtime_is_built():
with pytest.raises(ValueError, match="no-such-skill"):
AdkAgentRuntime(
skill_dirs={"no-such-skill": Path("/nonexistent")},
model_factory=lambda role: ScriptedLlm(),
)
def _function_responses(request: LlmRequest) -> dict[str, dict]:
responses = {}
for content in request.contents:
for part in content.parts or []:
if part.function_response:
responses[part.function_response.name] = part.function_response.response
return responses

138
tests/agents/test_roles.py Normal file
View File

@@ -0,0 +1,138 @@
"""Role files (agents/*.md) resolved against a Cloud provider bundle."""
from pathlib import Path
import pytest
from app.agents.roles import PILLAR_ADVISOR, RoleConfigError, load_roles
from app.execution.deliverables import DEPENDENCY_GRAPH
from app.providers import load_provider
AGENTS_DIR = Path(__file__).parents[2] / "agents"
@pytest.fixture(scope="module")
def bundle():
return load_provider("gcp")
def test_every_engine_role_and_the_pillar_advisor_is_defined(bundle):
roles = load_roles(AGENTS_DIR, bundle)
expected = {"orchestrator", PILLAR_ADVISOR}
expected |= {spec.specialist for spec in DEPENDENCY_GRAPH.values()}
assert set(roles) == expected
for role in roles.values():
assert role.instructions.strip()
def test_roles_take_skills_mcp_servers_and_models_from_the_bundle(bundle):
roles = load_roles(AGENTS_DIR, bundle)
assert roles["orchestrator"].skills == ("google-cloud-solution-architecture",)
assert roles[PILLAR_ADVISOR].skills == ()
assert roles["product-selector"].mcp_servers == ("developer_knowledge",)
assert "asset_inventory" in roles["requirements-analyst"].mcp_servers
assert roles["diagrammer"].model == bundle.models.for_role("diagrammer")
assert roles["orchestrator"].model == bundle.models.default
def test_preview_mcp_servers_can_be_excluded(bundle):
roles = load_roles(AGENTS_DIR, bundle, include_preview_mcp=False)
assert "asset_inventory" not in roles["requirements-analyst"].mcp_servers
def test_budgets_default_by_role_kind(bundle):
roles = load_roles(AGENTS_DIR, bundle, orchestrator_budget=7, specialist_budget=3)
assert roles["orchestrator"].budget == 7
assert roles["packager"].budget == 3
def test_specialists_include_the_shared_protocol_once(bundle):
roles = load_roles(AGENTS_DIR, bundle)
protocol = (AGENTS_DIR / "partials" / "specialist-protocol.md").read_text(
encoding="utf-8"
)
for spec in DEPENDENCY_GRAPH.values():
assert roles[spec.specialist].instructions.count(protocol.strip()) == 1
assert protocol.strip() not in roles["orchestrator"].instructions
def test_role_files_name_no_provider_specific_skills():
for path in AGENTS_DIR.rglob("*.md"):
text = path.read_text(encoding="utf-8").lower()
assert "google-cloud-" not in text, path.name
# --- validation -----------------------------------------------------------------------
def write_agents(tmp_path: Path, bundle, overrides: dict[str, str]) -> Path:
"""Copy the real role files, then overwrite some."""
target = tmp_path / "agents"
for path in AGENTS_DIR.rglob("*.md"):
destination = target / path.relative_to(AGENTS_DIR)
destination.parent.mkdir(parents=True, exist_ok=True)
destination.write_text(path.read_text(encoding="utf-8"), encoding="utf-8")
for name, text in overrides.items():
(target / name).write_text(text, encoding="utf-8")
return target
@pytest.mark.parametrize(
("overrides", "message"),
[
({"decomposer.md": "---\nrole: other\n---\nBody"}, "match"),
(
{
"decomposer.md": "---\nrole: decomposer\ndeliverable: requirements\n---\nB"
},
"requirements",
),
({"decomposer.md": "---\nrole: decomposer\n---\nBody"}, "deliverable"),
(
{
"decomposer.md": "---\nrole: decomposer\n"
"deliverable: technical_decomposition\ntool_tiers: [write]\n---\nB"
},
"write",
),
(
{
"decomposer.md": "---\nrole: decomposer\n"
"deliverable: technical_decomposition\nincludes: [missing]\n---\nB"
},
"missing",
),
(
{
"decomposer.md": "---\nrole: decomposer\n"
"deliverable: technical_decomposition\nskills: [x]\n---\nB"
},
"skills",
),
(
{
"decomposer.md": "---\nrole: decomposer\n"
"deliverable: technical_decomposition\n---\n "
},
"instructions",
),
],
)
def test_invalid_role_files_are_rejected(tmp_path, bundle, overrides, message):
agents_dir = write_agents(tmp_path, bundle, overrides)
with pytest.raises(RoleConfigError, match=message):
load_roles(agents_dir, bundle)
def test_missing_role_file_is_rejected(tmp_path, bundle):
agents_dir = write_agents(tmp_path, bundle, {})
(agents_dir / "packager.md").unlink()
with pytest.raises(RoleConfigError, match="packager"):
load_roles(agents_dir, bundle)

View File

@@ -9,6 +9,7 @@ from app.execution.deliverables import (
PhaseId,
check_submission,
deliverables_in,
inputs_of,
)
@@ -78,6 +79,25 @@ def test_graph_matches_the_upstream_skill_dependencies():
)
def test_inputs_are_earlier_phases_plus_indirect_dependencies_in_graph_order():
assert inputs_of(DeliverableKind.REQUIREMENTS) == ()
assert inputs_of(DeliverableKind.DEPLOYMENT_GUIDANCE) == (
DeliverableKind.REQUIREMENTS,
DeliverableKind.TECHNICAL_DECOMPOSITION,
DeliverableKind.PRODUCT_SELECTION,
DeliverableKind.DESIGN_RECOMMENDATIONS,
)
assert inputs_of(DeliverableKind.ARCHITECTURE_DESCRIPTION) == (
DeliverableKind.REQUIREMENTS,
DeliverableKind.TECHNICAL_DECOMPOSITION,
DeliverableKind.PRODUCT_SELECTION,
DeliverableKind.ARCHITECTURE_DIAGRAM,
)
assert set(inputs_of(DeliverableKind.SOLUTION_GUIDE)) == set(DeliverableKind) - {
DeliverableKind.SOLUTION_GUIDE
}
def test_submission_meeting_the_contract_has_nothing_missing():
content = (
"# Deployment guidance\n"

View File

@@ -151,7 +151,12 @@ async def test_specialist_receives_its_dependencies_content(tmp_path):
r for r in runtime.runs if r.role == specialist(K.ARCHITECTURE_DESCRIPTION)
)
dependencies = describer_run.task.context["dependencies"]
assert set(dependencies) == {"architecture_diagram", "product_selection"}
assert list(dependencies) == [
"requirements",
"technical_decomposition",
"product_selection",
"architecture_diagram",
]
assert "architecture.mmd" in dependencies["architecture_diagram"]["attachments"]
assert describer_run.task.deliverable == "architecture_description"

View File

@@ -0,0 +1,110 @@
"""The design advisor's fan-out to one advisor per architecture-framework pillar."""
import pytest
from app.agents.fake_runtime import Call, FakeAgentRuntime
from app.agents.roles import PILLAR_ADVISOR
from app.agents.runtime import RoleSpec, ToolSpec
from app.execution.pillars import CONSULT_TOOL, pillar_consultation_tool
from app.execution.toolkit import make_tool_factory
pytestmark = pytest.mark.asyncio
PILLARS = {
"security": "waf-security",
"reliability": "waf-reliability",
"sustainability": "waf-sustainability",
}
ADVISOR = RoleSpec(PILLAR_ADVISOR, "Advise on one pillar.", budget=5)
def advice(pillar: str) -> list[Call]:
return [Call("submit_advice", {"content": f"Use strong defaults for {pillar}."})]
def turn_for_pillar(task):
return advice(task.context["pillar"])
async def consult(tool: ToolSpec, **args):
return await tool.handler(**args)
async def test_consults_every_pillar_with_only_its_skill():
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [turn_for_pillar] * 3})
tool = pillar_consultation_tool(runtime, ADVISOR, PILLARS)
result = await consult(tool, brief="Event ingestion on serverless compute.")
assert result.outcome is None
for pillar in PILLARS:
assert f"## {pillar}" in result.message
assert f"Use strong defaults for {pillar}." in result.message
runs = {run.task.context["pillar"]: run for run in runtime.runs}
assert set(runs) == set(PILLARS)
assert (
runs["security"].task.context["brief"]
== "Event ingestion on serverless compute."
)
assert runtime.runs[0].tools == ("submit_advice",)
async def test_advisors_are_scoped_to_their_pillar_skill():
seen: dict[str, tuple[str, ...]] = {}
class RecordingRuntime(FakeAgentRuntime):
async def run(self, role, task, tools):
seen[task.context["pillar"]] = role.skills
return await super().run(role, task, tools)
runtime = RecordingRuntime({PILLAR_ADVISOR: [turn_for_pillar] * 3})
await consult(pillar_consultation_tool(runtime, ADVISOR, PILLARS), brief="b")
assert seen == {pillar: (skill,) for pillar, skill in PILLARS.items()}
async def test_a_subset_of_pillars_can_be_consulted():
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [turn_for_pillar]})
tool = pillar_consultation_tool(runtime, ADVISOR, PILLARS)
result = await consult(tool, brief="b", pillars=["reliability"])
assert "## reliability" in result.message
assert "## security" not in result.message
async def test_unknown_pillars_are_refused():
tool = pillar_consultation_tool(FakeAgentRuntime({}), ADVISOR, PILLARS)
result = await consult(tool, brief="b", pillars=["speed"])
assert result.message.startswith("Refused")
assert "security" in result.message
async def test_an_advisor_that_fails_is_reported_without_losing_the_others():
def maybe_fail(task):
if task.context["pillar"] == "reliability":
return [Call("submit_advice", {"content": ""})]
return advice(task.context["pillar"])
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [maybe_fail] * 3})
result = await consult(
pillar_consultation_tool(runtime, ADVISOR, PILLARS), brief="b"
)
assert "Use strong defaults for security." in result.message
assert "## reliability\n\nNo guidance:" in result.message
async def test_tool_factory_gives_the_consultation_only_to_the_design_advisor():
roles = {PILLAR_ADVISOR: ADVISOR}
factory = make_tool_factory(FakeAgentRuntime({}), roles, PILLARS)
design = factory(RoleSpec("design-advisor", "", budget=5), "design_recommendations")
other = factory(RoleSpec("decomposer", "", budget=5), "technical_decomposition")
assert [tool.name for tool in design] == [CONSULT_TOOL]
assert list(other) == []

View File

@@ -60,6 +60,20 @@ def test_role_mcp_servers_can_exclude_preview_servers():
assert bundle.mcp_servers_for_role("no-such-role") == ()
def test_gcp_bundle_maps_roles_and_pillars_to_vendored_skills():
bundle = load_provider("gcp")
assert bundle.role_skills["orchestrator"] == ("google-cloud-solution-architecture",)
assert bundle.pillar_skills == {
"security": "google-cloud-waf-security",
"reliability": "google-cloud-waf-reliability",
"operational-excellence": "google-cloud-waf-operational-excellence",
"cost-optimization": "google-cloud-waf-cost-optimization",
"performance-optimization": "google-cloud-waf-performance-optimization",
"sustainability": "google-cloud-waf-sustainability",
}
def test_model_for_role_falls_back_to_default():
bundle = load_provider("gcp")
@@ -154,6 +168,8 @@ def test_minimal_bundle_loads(tmp_path):
"allow",
),
({"role_mcp_servers": {"describer": ["missing"]}}, "missing"),
({"role_skills": {"describer": ["no-such-skill"]}}, "no-such-skill"),
({"pillar_skills": {"security": "no-such-skill"}}, "no-such-skill"),
],
)
def test_invalid_bundles_are_rejected(tmp_path, override, message):