feat: run role files on the ADK runtime with scoped Skills
Increment 5 of .scratch/solution-architect-agent/spec.md, test-first. - agents/*.md: provider-neutral role files for the Orchestrator, nine Specialists and the pillar advisor, with a shared Specialist protocol. - app/agents/roles.py: load and validate role files; Skills, MCP servers and models come from the provider bundle. - app/agents/adk_runtime.py: AgentRuntime on google-adk 2.9 with a per-run LlmAgent, role-scoped SkillToolset (no run_skill_script), step budgets, early stop on turn-ending tools, LiteLLM models. - app/execution/pillars.py + toolkit.py: design advisor fans out to one advisor per WAF pillar in parallel. - Specialists receive every earlier-Phase Deliverable as input. - Bundle gains role_skills, pillar_skills and -latest Gemini aliases; pin google-adk[extensions]==2.9.0. - 37 new tests; full suite 158 passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -610,3 +610,45 @@ Slowest tests: eval benchmark 7.3 s, then 3–4.5 s each for the orchestrator, e
|
|||||||
- **Session name.** `DatabaseRecorder` session rows use `agent_name="solution-architect"`.
|
- **Session name.** `DatabaseRecorder` session rows use `agent_name="solution-architect"`.
|
||||||
- **Deferred to increment 5:** the design advisor's pillar fan-out tool, which needs the `pillar-advisor` role file.
|
- **Deferred to increment 5:** the design advisor's pillar fan-out tool, which needs the `pillar-advisor` role file.
|
||||||
- **Not tested:** `AsyncPostgresSaver` against a real PostgreSQL (none runs locally). The import and wiring are in place.
|
- **Not tested:** `AsyncPostgresSaver` against a real PostgreSQL (none runs locally). The import and wiring are in place.
|
||||||
|
|
||||||
|
### 2026-09-16 · Increment 5: ADK runtime and role files
|
||||||
|
|
||||||
|
**Done (test-first)**
|
||||||
|
- **Role files** `agents/*.md` (11): `orchestrator`, the 9 Specialists and `pillar-advisor`, plus `agents/partials/specialist-protocol.md`, which every Specialist includes. Each Specialist file names its Deliverable, the Skill task it covers, its constraints and its required headings. The shared protocol covers context fields, questions through `needs_input`, confirming a draft with the user before submitting (or recording assumptions when pre-approved), and resubmitting until the contract passes.
|
||||||
|
- `app/agents/roles.py`: `load_roles(agents_dir, bundle, orchestrator_budget, specialist_budget, include_preview_mcp)`. It validates the file name against the role, the deliverable against `DEPENDENCY_GRAPH`, the tool tiers, the budget, the includes, a non-empty body, unknown fields, and missing or unexpected roles.
|
||||||
|
- `app/agents/adk_runtime.py`: `AdkAgentRuntime(skill_dirs, model_factory)`, the only module importing `google.adk`.
|
||||||
|
- **Per run:** a fresh `LlmAgent`, an in-memory session and a `Runner`.
|
||||||
|
- **Tools:** each `ToolSpec` is adapted to a `BaseTool` with a JSON-schema declaration, and a `SkillToolset` holding only the role's Skills is filtered to `list_skills`/`load_skill`/`load_skill_resource`.
|
||||||
|
- **Instructions:** passed as a callable, so role text is never state-templated.
|
||||||
|
- **Stopping:** a `before_tool_callback` budget counts every tool call, including Skill tools. Tool calls within a run are serialised, the first outcome ends the run (later calls in the same batch are refused), and a final text with no outcome becomes `Stopped(text)`.
|
||||||
|
- **Model:** `default_model_factory` wraps `role.model` in `LiteLlm`.
|
||||||
|
- `app/execution/pillars.py`: the `consult_pillar_advisors(brief, pillars?)` tool, which runs one `pillar-advisor` per pillar in parallel through `AgentRuntime`, each scoped to its pillar Skill with a `submit_advice` tool. One failed advisor is reported without losing the others. `app/execution/toolkit.py`: `make_tool_factory(runtime, roles, pillar_skills)` gives the tool to the design advisor only; increment 6 extends it.
|
||||||
|
- `app/execution/deliverables.py`: `inputs_of(kind)` gives a Specialist every Deliverable of earlier Phases plus everything it depends on within its own Phase. Before this, the packager's context lacked `architecture_description`, which nothing depends on.
|
||||||
|
- `providers/gcp.yaml` and `app/providers.py`: `role_skills` and `pillar_skills`, both validated against the Skills index. Models are now `gemini/gemini-pro-latest` by default and `gemini/gemini-flash-latest` for the pillar advisor and diagrammer.
|
||||||
|
- `requirements.txt`: `google-adk[extensions]==2.9.0`, pinned exactly because Skills are Experimental. Its extras bring LiteLLM (installed 1.101.0).
|
||||||
|
- Tests:
|
||||||
|
- `tests/agents/test_roles.py`: 14 tests.
|
||||||
|
- `tests/agents/test_adk_runtime.py`: 12 tests with a scripted `BaseLlm`, no network.
|
||||||
|
- `tests/agents/test_adk_engine.py`: 1 test running `ExecutionEngine` on the ADK runtime with the real role files and bundle. A Specialist question reaches the user and the answer reaches the next run.
|
||||||
|
- `tests/execution/test_pillars.py`: 6 tests.
|
||||||
|
- Plus provider and deliverable tests.
|
||||||
|
- Full suite: **158 passed**. `ruff` and `mypy` are clean. The 5 warnings are ADK and Starlette deprecations.
|
||||||
|
- **Checking the tests bite.** I broke each of these in the code on purpose, and each break failed at least one test:
|
||||||
|
- every Skill loadable by every role
|
||||||
|
- no budget
|
||||||
|
- no early stop after an outcome
|
||||||
|
- templated instructions
|
||||||
|
- both turn-ended guards removed at once
|
||||||
|
|
||||||
|
Either guard alone suffices under ADK 2.9's sequential callback order; both are kept as defence in depth.
|
||||||
|
|
||||||
|
**Deviations from the spec text**
|
||||||
|
- **Role frontmatter is provider-neutral.** Role files carry only `role`, `deliverable`, `tool_tiers`, `budget` and `includes`. Skills (`role_skills`, `pillar_skills`), MCP servers and models come from the provider bundle, so no role file names a Google Cloud Skill (a test checks this). `upstream_task` was dropped: role bodies describe the Skill task in neutral words.
|
||||||
|
- **Model ids.** They're LiteLLM `-latest` aliases, not the spec's `gemini-2.5-*` ids, which are older than LiteLLM's current Gemini 3.x entries.
|
||||||
|
- **LiteLLM version.** `litellm` comes from `google-adk[extensions]` (ADK requires ≥ 1.84), not `litellm~=1.74`.
|
||||||
|
- **Pillar fan-out** is a tool on the design advisor, not code inside the runtime.
|
||||||
|
|
||||||
|
**Deferred**
|
||||||
|
- **To increment 6:** mapping `RoleSpec.mcp_servers` to ADK MCP toolsets (`McpToolset` needs the `mcp` extra, not installed) and supplying local tools by `tool_tiers`. Both are declared and loaded now, but the runtime doesn't use them yet.
|
||||||
|
- **Not yet run against a real model:** no API key was used, so role-file quality is unproven until increment 8's evaluation.
|
||||||
|
- **Model API errors propagate** from `AgentRuntime.run` and fail the graph step, leaving the Execution at its last checkpoint. Decide in increment 7 whether to retry, surface as a question, or fail the A2A task.
|
||||||
|
|||||||
11
agents/decomposer.md
Normal file
11
agents/decomposer.md
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
role: decomposer
|
||||||
|
deliverable: technical_decomposition
|
||||||
|
tool_tiers: []
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **technical decomposition** Deliverable: the Skill's breakdown of the workload into logical components.
|
||||||
|
|
||||||
|
- Derive every component from the approved requirements, and name the requirements each component serves.
|
||||||
|
- Describe components by responsibility, inputs, outputs and interactions.
|
||||||
|
- Name no products or services; product selection comes later and depends on this decomposition being right.
|
||||||
11
agents/describer.md
Normal file
11
agents/describer.md
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
role: describer
|
||||||
|
deliverable: architecture_description
|
||||||
|
tool_tiers: []
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **architecture description** Deliverable: the Skill's task for describing the architecture.
|
||||||
|
|
||||||
|
- Explain the purpose of each component in the diagram, the relationships between components, and the task flow or data flow end to end.
|
||||||
|
- Use the product names from the product selection and the component names from the diagram exactly.
|
||||||
|
- Ground claims about product behaviour in the Cloud provider's documentation through your documentation tools, with citations.
|
||||||
13
agents/design-advisor.md
Normal file
13
agents/design-advisor.md
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
role: design-advisor
|
||||||
|
deliverable: design_recommendations
|
||||||
|
tool_tiers: []
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **design recommendations** Deliverable: the Skill's task for recommending how to configure each component to meet the functional and non-functional requirements.
|
||||||
|
|
||||||
|
- Call `consult_pillar_advisors` once, with a brief that restates the workload, the relevant requirements and the selected products. It runs one advisor per architecture-framework pillar in parallel and returns their guidance.
|
||||||
|
- Reconcile the pillars' guidance into recommendations for each component. Where pillars conflict, name the trade-off and recommend a resolution tied to the requirements.
|
||||||
|
- When an advisor's guidance is missing, derive that pillar's recommendations from the best-practices references in the solution-architecture Skill.
|
||||||
|
|
||||||
|
Required headings, one per pillar: `Security, privacy, and compliance`, `Reliability`, `Operational excellence`, `Cost optimization`, `Performance efficiency`, `Sustainability`.
|
||||||
11
agents/diagrammer.md
Normal file
11
agents/diagrammer.md
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
---
|
||||||
|
role: diagrammer
|
||||||
|
deliverable: architecture_diagram
|
||||||
|
tool_tiers: [general]
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **architecture diagram** Deliverable: the Skill's task for generating a Mermaid architecture diagram.
|
||||||
|
|
||||||
|
- Draw every component from the product selection and every relationship the data and control flow need, labelled with the selected products.
|
||||||
|
- Check the diagram with your Mermaid tool when one is available, and fix every error it reports.
|
||||||
|
- Put the diagram source in the `architecture.mmd` attachment, and show the same diagram in a `mermaid` code block in the content with a one-paragraph legend.
|
||||||
14
agents/iac-author.md
Normal file
14
agents/iac-author.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
role: iac-author
|
||||||
|
deliverable: deployment_guidance
|
||||||
|
tool_tiers: [general]
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **deployment guidance** Deliverable: the Skill's task for generating infrastructure-as-code and deployment instructions.
|
||||||
|
|
||||||
|
- Write Terraform that provisions every selected product with the configuration the design recommendations call for.
|
||||||
|
- Take every environment-specific value (project, region, names, image references) from variables; embed no credentials or secrets.
|
||||||
|
- Format and validate the Terraform with your tools when they are available, and fix everything they report.
|
||||||
|
- Put `terraform/main.tf` and `terraform/variables.tf` in the attachments, adding `terraform/outputs.tf` when outputs help.
|
||||||
|
|
||||||
|
Required headings: `Deployment prerequisites` (APIs to enable, tools, permissions) and `Step-by-step deployment instructions`, whose commands you give for the user to run, never run yourself.
|
||||||
23
agents/orchestrator.md
Normal file
23
agents/orchestrator.md
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
---
|
||||||
|
role: orchestrator
|
||||||
|
tool_tiers: []
|
||||||
|
---
|
||||||
|
You coordinate one solution-architecture Execution for the user. You decide what happens next; Specialists produce the Deliverables, and you are the only agent that talks to the user.
|
||||||
|
|
||||||
|
Load the solution-architecture Skill available to you for the workflow's rules: strict separation between requirements discovery and solutioning, explicit user approval of each Phase, and no execution of generated scripts.
|
||||||
|
|
||||||
|
Your task message carries JSON context: `workflow_request`, each Deliverable's `status`, `revisions` and `feedback`, each Phase's status, `ready_work`, `phases_ready_for_approval`, `pending_questions` from Specialists, the recent `conversation`, `notes` about discarded results, `pre_approved` and `allow_cloud_read`.
|
||||||
|
|
||||||
|
On each turn, take the first step that applies:
|
||||||
|
|
||||||
|
1. **The user asked for changes.** When the latest user message in `conversation` asks to change submitted work, call `reopen` for each affected Deliverable with the user's words as the reason.
|
||||||
|
2. **Specialists are waiting.** When `pending_questions` is not empty, call `ask_user`. Add your own questions only when the user's last message left something unclear.
|
||||||
|
3. **A Phase is complete.** When `phases_ready_for_approval` names a Phase, read its Deliverables with `read_deliverable` and call `request_approval` with a summary a busy architect can approve from. In a pre-approved Execution the approval is granted immediately, and you continue with the next step in the same turn.
|
||||||
|
4. **Work is ready.** When `ready_work` is not empty, call `delegate` once with every ready Deliverable, so independent work runs in parallel.
|
||||||
|
5. **Validation found problems.** When the validation results report failures that trace to an earlier Deliverable, call `reopen` on that Deliverable with the findings, then delegate it.
|
||||||
|
6. **Packaging is approved.** Call `finish`.
|
||||||
|
7. **Otherwise,** call `ask_user` to find out what the user needs.
|
||||||
|
|
||||||
|
When the user says a Deliverable already exists, still delegate it; the Specialist submits the user's version. When a tool refuses, read the reason and choose again.
|
||||||
|
|
||||||
|
Every turn ends with exactly one call to `delegate`, `ask_user`, `request_approval` or `finish`.
|
||||||
13
agents/packager.md
Normal file
13
agents/packager.md
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
role: packager
|
||||||
|
deliverable: solution_guide
|
||||||
|
tool_tiers: []
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **solution guide** Deliverable: the Skill's packaging of every approved Deliverable into one guide.
|
||||||
|
|
||||||
|
- Load the solution-guide output template from the Skill's assets and follow its structure and section numbering exactly.
|
||||||
|
- Consolidate the content of every Deliverable in `dependencies`; rewrite for flow, but keep every requirement, product decision, citation, recommendation, command and validation verdict.
|
||||||
|
- Include the architecture diagram as a `mermaid` code block and the Terraform as code blocks.
|
||||||
|
|
||||||
|
Required headings: the template's eight top-level sections.
|
||||||
20
agents/partials/specialist-protocol.md
Normal file
20
agents/partials/specialist-protocol.md
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
## How you work
|
||||||
|
|
||||||
|
You are a Specialist in a solution-architecture Execution. You produce exactly one Deliverable, named in your task. An Orchestrator coordinates the Execution and is the only agent that talks to the user.
|
||||||
|
|
||||||
|
Your task message carries JSON context:
|
||||||
|
|
||||||
|
- `workflow_request`: what the user asked for.
|
||||||
|
- `dependencies`: every earlier Deliverable your work builds on, by kind, with its `content` and `attachments`.
|
||||||
|
- `feedback`: why earlier versions of your Deliverable were revised. Address every item.
|
||||||
|
- `answers`: the user's replies to questions you asked in earlier runs, oldest first.
|
||||||
|
- `pre_approved`: `true` when no user is available.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
1. Load the Skill available to you and find the part that covers your Deliverable. Read the references and assets it points to with `load_skill_resource`.
|
||||||
|
2. Ground your work in `dependencies`, `feedback` and `answers`. When you need information only the user has, call `needs_input` with all your questions at once, each saying why it matters. Your run ends there, and you are started again with the answers.
|
||||||
|
3. Draft the Deliverable. When `pre_approved` is `false`, confirm the draft with the user before submitting: call `needs_input` with a short summary of the draft and ask them to approve it or request changes. Skip this when `answers` already holds their approval of the same draft. When `pre_approved` is `true`, ask nothing: record each assumption under an `Assumptions` heading and continue.
|
||||||
|
4. Call `submit_deliverable` with the complete markdown content and every required attachment. When it reports missing sections or attachments, add them and submit again.
|
||||||
|
|
||||||
|
You are done when `submit_deliverable` accepts the Deliverable.
|
||||||
14
agents/pillar-advisor.md
Normal file
14
agents/pillar-advisor.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
role: pillar-advisor
|
||||||
|
tool_tiers: []
|
||||||
|
---
|
||||||
|
You advise on one pillar of the Cloud provider's architecture framework for a single workload. The Skill available to you is that pillar's guidance.
|
||||||
|
|
||||||
|
Your task message carries the design advisor's brief in `brief` and the pillar's name in `pillar`.
|
||||||
|
|
||||||
|
1. Load the Skill available to you.
|
||||||
|
2. Evaluate the workload in the brief against the Skill's core principles and validation checklist.
|
||||||
|
3. Write prioritised, actionable recommendations for each relevant component, naming the products the Skill lists for each gap and citing its guidance.
|
||||||
|
4. List any assessment questions from the Skill that the brief leaves unanswered, as open questions; don't ask the user.
|
||||||
|
|
||||||
|
Call `submit_advice` with your recommendations as markdown. You are done when it accepts them.
|
||||||
13
agents/product-selector.md
Normal file
13
agents/product-selector.md
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
---
|
||||||
|
role: product-selector
|
||||||
|
deliverable: product_selection
|
||||||
|
tool_tiers: []
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **product selection** Deliverable: the Skill's task for identifying the products and features each component needs.
|
||||||
|
|
||||||
|
- Recommend one product or feature per component of the technical decomposition, grounded in the Cloud provider's official documentation through your documentation tools, with a citation for every recommendation.
|
||||||
|
- Check the release status of every product you recommend, and recommend nothing deprecated, retired or unsupported.
|
||||||
|
- Where alternatives exist, follow the documentation's guidance on choosing, and list the alternatives with their pros and cons.
|
||||||
|
|
||||||
|
Required heading: `Products and features mapping`, containing a table with columns for component, recommended product or feature, justification and citations, alternatives, and their pros and cons.
|
||||||
15
agents/requirements-analyst.md
Normal file
15
agents/requirements-analyst.md
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
---
|
||||||
|
role: requirements-analyst
|
||||||
|
deliverable: requirements
|
||||||
|
tool_tiers: [cloud_read]
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **requirements** Deliverable: the Skill's requirements-discovery gathering and its resolution of ambiguities and contradictions.
|
||||||
|
|
||||||
|
- Cover functional requirements, non-functional requirements (security, privacy, compliance, reliability, disaster recovery, cost, operations, performance, sustainability), current state, and dependencies.
|
||||||
|
- When non-functional requirements are missing, ask for them and explain what each one decides: availability tiers, scaling, cost budgets, resource types and security posture.
|
||||||
|
- Name every ambiguity or contradiction, explain why the conflicting goals can't both hold, and ask the user to choose. When they delegate the choice, propose one and ask them to approve it.
|
||||||
|
- When the user runs workloads in a cloud environment today, use your cloud discovery tools to describe the current state from what actually exists, and record anything you couldn't read as a discovery gap.
|
||||||
|
- Name no products, services or architectures. This Deliverable describes the problem, not the solution.
|
||||||
|
|
||||||
|
Required headings: `Functional requirements`, `Non-functional requirements`, `Current state`, `Dependencies`. Add `Open questions` and `Assumptions` when they apply.
|
||||||
14
agents/validator.md
Normal file
14
agents/validator.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
---
|
||||||
|
role: validator
|
||||||
|
deliverable: validation_results
|
||||||
|
tool_tiers: [general, cloud_read]
|
||||||
|
includes: [specialist-protocol]
|
||||||
|
---
|
||||||
|
You produce the **validation results** Deliverable: the Skill's pre-deployment validation, done statically without provisioning anything.
|
||||||
|
|
||||||
|
- Run the deployment dry-run with your tools: format and validate the Terraform, and preview it with a plan when you have permission to read live cloud resources.
|
||||||
|
- Analyse the architecture and policy statically: network routing and exposure, firewall rules, and IAM least privilege, against the design recommendations.
|
||||||
|
- Report each finding with its evidence and the Deliverable it traces to, so the Orchestrator can send it back for revision.
|
||||||
|
- Give the exact commands for every check you couldn't run, for the user to run themselves.
|
||||||
|
|
||||||
|
Required headings: `Deployment dry-run` and `Architecture and policy analysis`, each ending with a pass or fail verdict.
|
||||||
@@ -42,23 +42,28 @@ class SkillMeta:
|
|||||||
path: Path
|
path: Path
|
||||||
|
|
||||||
|
|
||||||
def read_skill(skill_dir: Path) -> SkillMeta:
|
def split_frontmatter(text: str, source: Path) -> tuple[dict, str]:
|
||||||
"""Parse and validate the SKILL.md in `skill_dir`."""
|
"""Split a markdown document into its YAML frontmatter mapping and body."""
|
||||||
skill_file = skill_dir / SKILL_FILE
|
match = _FRONTMATTER_PATTERN.match(text)
|
||||||
match = _FRONTMATTER_PATTERN.match(skill_file.read_text(encoding="utf-8"))
|
|
||||||
if not match:
|
if not match:
|
||||||
raise SkillFormatError(
|
raise SkillFormatError(f"{source}: missing YAML frontmatter delimited by '---'")
|
||||||
f"{skill_file}: missing YAML frontmatter delimited by '---'"
|
|
||||||
)
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
frontmatter = yaml.safe_load(match.group(1))
|
frontmatter = yaml.safe_load(match.group(1))
|
||||||
except yaml.YAMLError as exc:
|
except yaml.YAMLError as exc:
|
||||||
raise SkillFormatError(
|
raise SkillFormatError(
|
||||||
f"{skill_file}: frontmatter is not valid YAML: {exc}"
|
f"{source}: frontmatter is not valid YAML: {exc}"
|
||||||
) from exc
|
) from exc
|
||||||
if not isinstance(frontmatter, dict):
|
if not isinstance(frontmatter, dict):
|
||||||
raise SkillFormatError(f"{skill_file}: frontmatter must be a mapping")
|
raise SkillFormatError(f"{source}: frontmatter must be a mapping")
|
||||||
|
return frontmatter, text[match.end() :]
|
||||||
|
|
||||||
|
|
||||||
|
def read_skill(skill_dir: Path) -> SkillMeta:
|
||||||
|
"""Parse and validate the SKILL.md in `skill_dir`."""
|
||||||
|
skill_file = skill_dir / SKILL_FILE
|
||||||
|
frontmatter, _ = split_frontmatter(
|
||||||
|
skill_file.read_text(encoding="utf-8"), skill_file
|
||||||
|
)
|
||||||
|
|
||||||
unknown = set(frontmatter) - _ALLOWED_FIELDS
|
unknown = set(frontmatter) - _ALLOWED_FIELDS
|
||||||
if unknown:
|
if unknown:
|
||||||
|
|||||||
196
app/agents/adk_runtime.py
Normal file
196
app/agents/adk_runtime.py
Normal file
@@ -0,0 +1,196 @@
|
|||||||
|
"""AgentRuntime on Google ADK (ADR-0003): an LlmAgent per run, with SkillToolset.
|
||||||
|
|
||||||
|
This is the only module that imports google.adk. Each run builds a fresh LlmAgent
|
||||||
|
and in-memory session (Specialists are stateless), offers the role's ToolSpecs and
|
||||||
|
Skills, and stops as soon as a tool returns an outcome or the budget is spent.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import uuid
|
||||||
|
from collections.abc import Callable, Mapping, Sequence
|
||||||
|
from contextlib import aclosing
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from google.adk.agents import LlmAgent
|
||||||
|
from google.adk.models.base_llm import BaseLlm
|
||||||
|
from google.adk.runners import Runner
|
||||||
|
from google.adk.sessions import InMemorySessionService
|
||||||
|
from google.adk.skills import load_skill_from_dir
|
||||||
|
from google.adk.skills.models import Skill
|
||||||
|
from google.adk.tools.base_tool import BaseTool
|
||||||
|
from google.adk.tools.skill_toolset import SkillToolset
|
||||||
|
from google.adk.tools.tool_context import ToolContext
|
||||||
|
from google.genai import types
|
||||||
|
|
||||||
|
from app.agents.runtime import (
|
||||||
|
AgentTask,
|
||||||
|
BudgetExhausted,
|
||||||
|
Outcome,
|
||||||
|
RoleSpec,
|
||||||
|
Stopped,
|
||||||
|
ToolSpec,
|
||||||
|
)
|
||||||
|
|
||||||
|
APP_NAME = "solution-architect"
|
||||||
|
# run_skill_script is deliberately absent: generated or bundled scripts never run.
|
||||||
|
SKILL_TOOLS = ["list_skills", "load_skill", "load_skill_resource"]
|
||||||
|
TURN_ENDED_REPLY = "Your turn has already ended; this call was not run."
|
||||||
|
|
||||||
|
ModelFactory = Callable[[RoleSpec], BaseLlm | str]
|
||||||
|
|
||||||
|
|
||||||
|
def default_model_factory(role: RoleSpec) -> BaseLlm:
|
||||||
|
"""Wrap the role's LiteLLM model id, so any LiteLLM provider can serve the agent."""
|
||||||
|
from google.adk.models.lite_llm import LiteLlm
|
||||||
|
|
||||||
|
if not role.model:
|
||||||
|
raise ValueError(f"Role {role.name!r} has no model configured")
|
||||||
|
return LiteLlm(model=role.model)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _RunControl:
|
||||||
|
budget: int
|
||||||
|
role: str
|
||||||
|
calls: int = 0
|
||||||
|
outcome: Outcome | None = None
|
||||||
|
lock: asyncio.Lock = field(default_factory=asyncio.Lock)
|
||||||
|
|
||||||
|
def before_tool(
|
||||||
|
self, tool: BaseTool, args: dict[str, Any], tool_context: Any
|
||||||
|
) -> dict[str, Any] | None:
|
||||||
|
if self.outcome is not None:
|
||||||
|
return {"error": TURN_ENDED_REPLY}
|
||||||
|
self.calls += 1
|
||||||
|
if self.calls > self.budget:
|
||||||
|
self.outcome = BudgetExhausted(
|
||||||
|
f"{self.role} used all {self.budget} tool calls before finishing."
|
||||||
|
)
|
||||||
|
return {"error": "Tool-call budget exhausted; stop now."}
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class _SpecTool(BaseTool):
|
||||||
|
"""Adapts a framework-neutral ToolSpec to an ADK tool."""
|
||||||
|
|
||||||
|
def __init__(self, spec: ToolSpec, control: _RunControl) -> None:
|
||||||
|
super().__init__(name=spec.name, description=spec.description)
|
||||||
|
self._spec = spec
|
||||||
|
self._control = control
|
||||||
|
|
||||||
|
def _get_declaration(self) -> types.FunctionDeclaration:
|
||||||
|
return types.FunctionDeclaration(
|
||||||
|
name=self._spec.name,
|
||||||
|
description=self._spec.description,
|
||||||
|
parameters_json_schema=dict(self._spec.parameters),
|
||||||
|
)
|
||||||
|
|
||||||
|
async def run_async(
|
||||||
|
self, *, args: dict[str, Any], tool_context: ToolContext
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
# Serialise calls within a run: tools may mutate Execution state, and the
|
||||||
|
# first tool to end the turn must stop the rest of a parallel batch.
|
||||||
|
async with self._control.lock:
|
||||||
|
if self._control.outcome is not None:
|
||||||
|
return {"error": TURN_ENDED_REPLY}
|
||||||
|
try:
|
||||||
|
result = await self._spec.handler(**args)
|
||||||
|
except TypeError as exc:
|
||||||
|
return {"error": f"Invalid arguments for {self._spec.name}: {exc}"}
|
||||||
|
if result.outcome is not None:
|
||||||
|
self._control.outcome = result.outcome
|
||||||
|
return {"result": result.message}
|
||||||
|
|
||||||
|
|
||||||
|
class AdkAgentRuntime:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
skill_dirs: Mapping[str, Path],
|
||||||
|
model_factory: ModelFactory = default_model_factory,
|
||||||
|
) -> None:
|
||||||
|
self._skills: dict[str, Skill] = {}
|
||||||
|
for name, path in skill_dirs.items():
|
||||||
|
try:
|
||||||
|
skill = load_skill_from_dir(path)
|
||||||
|
except Exception as exc:
|
||||||
|
raise ValueError(
|
||||||
|
f"Could not load skill {name!r} from {path}: {exc}"
|
||||||
|
) from exc
|
||||||
|
if skill.name != name:
|
||||||
|
raise ValueError(
|
||||||
|
f"Skill at {path} is named {skill.name!r}, not {name!r}"
|
||||||
|
)
|
||||||
|
self._skills[name] = skill
|
||||||
|
self._model_factory = model_factory
|
||||||
|
|
||||||
|
async def run(
|
||||||
|
self, role: RoleSpec, task: AgentTask, tools: Sequence[ToolSpec]
|
||||||
|
) -> Outcome:
|
||||||
|
control = _RunControl(budget=role.budget, role=role.name)
|
||||||
|
agent_tools: list[Any] = [_SpecTool(spec, control) for spec in tools]
|
||||||
|
if role.skills:
|
||||||
|
unknown = [name for name in role.skills if name not in self._skills]
|
||||||
|
if unknown:
|
||||||
|
raise ValueError(f"Role {role.name!r} needs unknown skills: {unknown}")
|
||||||
|
agent_tools.append(
|
||||||
|
SkillToolset(
|
||||||
|
skills=[self._skills[name] for name in role.skills],
|
||||||
|
tool_filter=SKILL_TOOLS,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
instructions = role.instructions
|
||||||
|
agent = LlmAgent(
|
||||||
|
name=_agent_name(role.name),
|
||||||
|
model=self._model_factory(role),
|
||||||
|
# A callable instruction is passed verbatim: no {state} templating.
|
||||||
|
instruction=lambda _context: instructions,
|
||||||
|
tools=agent_tools,
|
||||||
|
before_tool_callback=control.before_tool,
|
||||||
|
)
|
||||||
|
runner = Runner(
|
||||||
|
app_name=APP_NAME,
|
||||||
|
agent=agent,
|
||||||
|
session_service=InMemorySessionService(),
|
||||||
|
auto_create_session=True,
|
||||||
|
)
|
||||||
|
message = types.Content(
|
||||||
|
role="user", parts=[types.Part(text=_render_task(task))]
|
||||||
|
)
|
||||||
|
|
||||||
|
final_text = ""
|
||||||
|
async with aclosing(
|
||||||
|
runner.run_async(
|
||||||
|
user_id=APP_NAME, session_id=str(uuid.uuid4()), new_message=message
|
||||||
|
)
|
||||||
|
) as events:
|
||||||
|
async for event in events:
|
||||||
|
if control.outcome is not None:
|
||||||
|
break
|
||||||
|
if event.is_final_response() and event.content and event.content.parts:
|
||||||
|
final_text = "".join(
|
||||||
|
part.text for part in event.content.parts if part.text
|
||||||
|
).strip()
|
||||||
|
|
||||||
|
if control.outcome is not None:
|
||||||
|
return control.outcome
|
||||||
|
return Stopped(
|
||||||
|
final_text
|
||||||
|
or f"{role.name} ended without calling a tool that ends its turn."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _agent_name(role: str) -> str:
|
||||||
|
return re.sub(r"\W", "_", role)
|
||||||
|
|
||||||
|
|
||||||
|
def _render_task(task: AgentTask) -> str:
|
||||||
|
context = json.dumps(task.context, indent=2, default=str, ensure_ascii=False)
|
||||||
|
return f"{task.prompt}\n\nContext:\n```json\n{context}\n```"
|
||||||
128
app/agents/roles.py
Normal file
128
app/agents/roles.py
Normal file
@@ -0,0 +1,128 @@
|
|||||||
|
"""Load role files (agents/<role>.md) into RoleSpecs for one Cloud provider.
|
||||||
|
|
||||||
|
Role files are framework- and provider-neutral: frontmatter names the role, the
|
||||||
|
Deliverable it owns, its local tool tiers, an optional budget and shared partials to
|
||||||
|
include. The provider bundle supplies each role's Skills, MCP servers and model.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from app.agent_skills import SkillFormatError, split_frontmatter
|
||||||
|
from app.agents.runtime import RoleSpec
|
||||||
|
from app.execution.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||||
|
from app.providers import TIERS, ProviderBundle
|
||||||
|
|
||||||
|
ORCHESTRATOR = "orchestrator"
|
||||||
|
PILLAR_ADVISOR = "pillar-advisor"
|
||||||
|
PARTIALS_DIR = "partials"
|
||||||
|
|
||||||
|
_ALLOWED_FIELDS = {"role", "deliverable", "tool_tiers", "budget", "includes"}
|
||||||
|
|
||||||
|
|
||||||
|
class RoleConfigError(ValueError):
|
||||||
|
"""A role file is missing or invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
def load_roles(
|
||||||
|
agents_dir: Path,
|
||||||
|
bundle: ProviderBundle,
|
||||||
|
*,
|
||||||
|
orchestrator_budget: int = 40,
|
||||||
|
specialist_budget: int = 15,
|
||||||
|
include_preview_mcp: bool = True,
|
||||||
|
) -> dict[str, RoleSpec]:
|
||||||
|
specialists = {spec.specialist: kind for kind, spec in DEPENDENCY_GRAPH.items()}
|
||||||
|
required = {ORCHESTRATOR, PILLAR_ADVISOR, *specialists}
|
||||||
|
|
||||||
|
roles = {}
|
||||||
|
for path in sorted(agents_dir.glob("*.md")):
|
||||||
|
role = _load_role(
|
||||||
|
path,
|
||||||
|
agents_dir,
|
||||||
|
bundle,
|
||||||
|
specialists,
|
||||||
|
orchestrator_budget if path.stem == ORCHESTRATOR else specialist_budget,
|
||||||
|
include_preview_mcp,
|
||||||
|
)
|
||||||
|
roles[role.name] = role
|
||||||
|
|
||||||
|
missing = sorted(required - set(roles))
|
||||||
|
if missing:
|
||||||
|
raise RoleConfigError(
|
||||||
|
f"{agents_dir}: missing role files for {', '.join(missing)}"
|
||||||
|
)
|
||||||
|
unexpected = sorted(set(roles) - required)
|
||||||
|
if unexpected:
|
||||||
|
raise RoleConfigError(
|
||||||
|
f"{agents_dir}: unexpected role files {', '.join(unexpected)}"
|
||||||
|
)
|
||||||
|
return roles
|
||||||
|
|
||||||
|
|
||||||
|
def _load_role(
|
||||||
|
path: Path,
|
||||||
|
agents_dir: Path,
|
||||||
|
bundle: ProviderBundle,
|
||||||
|
specialists: dict[str, DeliverableKind],
|
||||||
|
default_budget: int,
|
||||||
|
include_preview_mcp: bool,
|
||||||
|
) -> RoleSpec:
|
||||||
|
try:
|
||||||
|
frontmatter, body = split_frontmatter(path.read_text(encoding="utf-8"), path)
|
||||||
|
except SkillFormatError as exc:
|
||||||
|
raise RoleConfigError(str(exc)) from exc
|
||||||
|
|
||||||
|
def fail(message: str) -> RoleConfigError:
|
||||||
|
return RoleConfigError(f"{path}: {message}")
|
||||||
|
|
||||||
|
unknown = sorted(set(frontmatter) - _ALLOWED_FIELDS)
|
||||||
|
if unknown:
|
||||||
|
raise fail(
|
||||||
|
f"unsupported fields {', '.join(unknown)}; skills, MCP servers and models "
|
||||||
|
"come from the provider bundle"
|
||||||
|
)
|
||||||
|
|
||||||
|
name = frontmatter.get("role")
|
||||||
|
if name != path.stem:
|
||||||
|
raise fail(f"'role' {name!r} must match the file name {path.stem!r}")
|
||||||
|
|
||||||
|
deliverable = frontmatter.get("deliverable")
|
||||||
|
if name in specialists:
|
||||||
|
expected = specialists[name].value
|
||||||
|
if deliverable != expected:
|
||||||
|
raise fail(f"'deliverable' must be {expected!r}, not {deliverable!r}")
|
||||||
|
elif deliverable is not None:
|
||||||
|
raise fail(f"role {name!r} owns no Deliverable, but names {deliverable!r}")
|
||||||
|
|
||||||
|
tiers = tuple(frontmatter.get("tool_tiers") or ())
|
||||||
|
bad_tiers = [tier for tier in tiers if tier not in TIERS]
|
||||||
|
if bad_tiers:
|
||||||
|
raise fail(f"'tool_tiers' {', '.join(bad_tiers)} must be among {TIERS}")
|
||||||
|
|
||||||
|
budget = frontmatter.get("budget", default_budget)
|
||||||
|
if not isinstance(budget, int) or budget < 1:
|
||||||
|
raise fail("'budget' must be a positive integer")
|
||||||
|
|
||||||
|
sections = [body.strip()]
|
||||||
|
for include in frontmatter.get("includes") or ():
|
||||||
|
partial = agents_dir / PARTIALS_DIR / f"{include}.md"
|
||||||
|
if not partial.is_file():
|
||||||
|
raise fail(f"include {include!r} not found at {partial}")
|
||||||
|
sections.append(partial.read_text(encoding="utf-8").strip())
|
||||||
|
if not sections[0]:
|
||||||
|
raise fail("instructions (the body after the frontmatter) must not be empty")
|
||||||
|
|
||||||
|
return RoleSpec(
|
||||||
|
name=name,
|
||||||
|
instructions="\n\n".join(sections),
|
||||||
|
budget=budget,
|
||||||
|
skills=tuple(bundle.role_skills.get(name, ())),
|
||||||
|
tool_tiers=tiers,
|
||||||
|
mcp_servers=tuple(
|
||||||
|
server.id
|
||||||
|
for server in bundle.mcp_servers_for_role(name, include_preview_mcp)
|
||||||
|
),
|
||||||
|
model=bundle.models.for_role(name),
|
||||||
|
)
|
||||||
@@ -140,6 +140,30 @@ DEPENDENCY_GRAPH: Mapping[DeliverableKind, DeliverableSpec] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def inputs_of(kind: DeliverableKind) -> tuple[DeliverableKind, ...]:
|
||||||
|
"""The Deliverables a Specialist works from, in graph order.
|
||||||
|
|
||||||
|
Every Deliverable in an earlier Phase (all approved by the time `kind` is ready),
|
||||||
|
plus everything `kind` depends on, directly or not, within its own Phase.
|
||||||
|
"""
|
||||||
|
spec = DEPENDENCY_GRAPH[kind]
|
||||||
|
earlier_phases = PHASE_ORDER[: PHASE_ORDER.index(spec.phase)]
|
||||||
|
upstream: set[DeliverableKind] = set()
|
||||||
|
frontier = set(DEPENDENCY_GRAPH[kind].depends_on)
|
||||||
|
while frontier:
|
||||||
|
upstream |= frontier
|
||||||
|
frontier = {
|
||||||
|
dependency
|
||||||
|
for item in frontier
|
||||||
|
for dependency in DEPENDENCY_GRAPH[item].depends_on
|
||||||
|
} - upstream
|
||||||
|
return tuple(
|
||||||
|
candidate
|
||||||
|
for candidate, candidate_spec in DEPENDENCY_GRAPH.items()
|
||||||
|
if candidate in upstream or candidate_spec.phase in earlier_phases
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def deliverables_in(phase: PhaseId) -> tuple[DeliverableKind, ...]:
|
def deliverables_in(phase: PhaseId) -> tuple[DeliverableKind, ...]:
|
||||||
return tuple(kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase)
|
return tuple(kind for kind, spec in DEPENDENCY_GRAPH.items() if spec.phase is phase)
|
||||||
|
|
||||||
|
|||||||
@@ -31,7 +31,12 @@ from app.agents.runtime import (
|
|||||||
ToolSpec,
|
ToolSpec,
|
||||||
TurnEnded,
|
TurnEnded,
|
||||||
)
|
)
|
||||||
from app.execution.deliverables import DEPENDENCY_GRAPH, PHASE_ORDER, DeliverableKind
|
from app.execution.deliverables import (
|
||||||
|
DEPENDENCY_GRAPH,
|
||||||
|
PHASE_ORDER,
|
||||||
|
DeliverableKind,
|
||||||
|
inputs_of,
|
||||||
|
)
|
||||||
from app.execution.ledger import Ledger, LedgerRefusal
|
from app.execution.ledger import Ledger, LedgerRefusal
|
||||||
from app.execution.recorder import ExecutionRecorder
|
from app.execution.recorder import ExecutionRecorder
|
||||||
from app.execution.state import ExecutionState
|
from app.execution.state import ExecutionState
|
||||||
@@ -392,7 +397,7 @@ def _specialist_input(
|
|||||||
"pre_approved": bool(state.get("pre_approved")),
|
"pre_approved": bool(state.get("pre_approved")),
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
dependency.value: dict(deliverables[dependency.value])
|
dependency.value: dict(deliverables[dependency.value])
|
||||||
for dependency in spec.depends_on
|
for dependency in inputs_of(kind)
|
||||||
},
|
},
|
||||||
"feedback": list(ledger.feedback(kind)),
|
"feedback": list(ledger.feedback(kind)),
|
||||||
"answers": list(state.get("answers", {}).get(kind.value, [])),
|
"answers": list(state.get("answers", {}).get(kind.value, [])),
|
||||||
|
|||||||
103
app/execution/pillars.py
Normal file
103
app/execution/pillars.py
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
"""The design advisor's fan-out to one advisor per architecture-framework pillar.
|
||||||
|
|
||||||
|
Pillar advice is working material for the design recommendations Deliverable, not a
|
||||||
|
Deliverable itself, so it never touches the Ledger.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import dataclasses
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
|
||||||
|
from app.agents.runtime import (
|
||||||
|
AgentRuntime,
|
||||||
|
AgentTask,
|
||||||
|
RoleSpec,
|
||||||
|
Submitted,
|
||||||
|
ToolResult,
|
||||||
|
ToolSpec,
|
||||||
|
)
|
||||||
|
|
||||||
|
CONSULT_TOOL = "consult_pillar_advisors"
|
||||||
|
|
||||||
|
|
||||||
|
def pillar_consultation_tool(
|
||||||
|
runtime: AgentRuntime,
|
||||||
|
advisor: RoleSpec,
|
||||||
|
pillar_skills: Mapping[str, str],
|
||||||
|
) -> ToolSpec:
|
||||||
|
"""A tool that runs one pillar advisor per pillar, in parallel, each with its Skill."""
|
||||||
|
|
||||||
|
async def consult(brief: str, pillars: Sequence[str] | None = None) -> ToolResult:
|
||||||
|
chosen = list(pillars) if pillars else list(pillar_skills)
|
||||||
|
unknown = [pillar for pillar in chosen if pillar not in pillar_skills]
|
||||||
|
if unknown:
|
||||||
|
return ToolResult(
|
||||||
|
f"Refused: unknown pillars {', '.join(unknown)}; "
|
||||||
|
f"choose from {', '.join(pillar_skills)}"
|
||||||
|
)
|
||||||
|
results = await asyncio.gather(
|
||||||
|
*(
|
||||||
|
_advise(runtime, advisor, pillar, pillar_skills[pillar], brief)
|
||||||
|
for pillar in chosen
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return ToolResult("\n\n".join(results))
|
||||||
|
|
||||||
|
return ToolSpec(
|
||||||
|
CONSULT_TOOL,
|
||||||
|
"Get guidance from one advisor per architecture-framework pillar, in parallel. "
|
||||||
|
"Give a brief with the workload, its requirements and the selected products.",
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"brief": {"type": "string"},
|
||||||
|
"pillars": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {"type": "string", "enum": list(pillar_skills)},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["brief"],
|
||||||
|
},
|
||||||
|
consult,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def _advise(
|
||||||
|
runtime: AgentRuntime, advisor: RoleSpec, pillar: str, skill: str, brief: str
|
||||||
|
) -> str:
|
||||||
|
role = dataclasses.replace(advisor, skills=(skill,))
|
||||||
|
task = AgentTask(
|
||||||
|
role=advisor.name,
|
||||||
|
prompt=f"Advise on the {pillar} pillar for this workload.",
|
||||||
|
context={"pillar": pillar, "brief": brief},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
outcome = await runtime.run(role, task, [_submit_advice_tool()])
|
||||||
|
except Exception as exc: # noqa: BLE001 - one failed advisor must not lose the others
|
||||||
|
return f"## {pillar}\n\nNo guidance: the advisor failed ({exc})."
|
||||||
|
if isinstance(outcome, Submitted):
|
||||||
|
return f"## {pillar}\n\n{outcome.content.strip()}"
|
||||||
|
return (
|
||||||
|
f"## {pillar}\n\nNo guidance: the advisor ended with {type(outcome).__name__}."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _submit_advice_tool() -> ToolSpec:
|
||||||
|
async def submit_advice(content: str) -> ToolResult:
|
||||||
|
if not content.strip():
|
||||||
|
return ToolResult("Not submitted: the advice is empty.")
|
||||||
|
return ToolResult("Advice submitted.", Submitted(content))
|
||||||
|
|
||||||
|
return ToolSpec(
|
||||||
|
"submit_advice",
|
||||||
|
"Submit your pillar recommendations as markdown. Ends your turn.",
|
||||||
|
{
|
||||||
|
"type": "object",
|
||||||
|
"properties": {"content": {"type": "string"}},
|
||||||
|
"required": ["content"],
|
||||||
|
},
|
||||||
|
submit_advice,
|
||||||
|
tier="execution",
|
||||||
|
)
|
||||||
33
app/execution/toolkit.py
Normal file
33
app/execution/toolkit.py
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
"""Which extra tools each role gets beyond the Execution tools.
|
||||||
|
|
||||||
|
Increment 6 adds the local general and cloud-read tools and remote MCP servers here.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
|
||||||
|
from app.agents.roles import PILLAR_ADVISOR
|
||||||
|
from app.agents.runtime import AgentRuntime, RoleSpec, ToolSpec
|
||||||
|
from app.execution.deliverables import DeliverableKind
|
||||||
|
from app.execution.graph import ToolFactory
|
||||||
|
from app.execution.pillars import pillar_consultation_tool
|
||||||
|
|
||||||
|
|
||||||
|
def make_tool_factory(
|
||||||
|
runtime: AgentRuntime,
|
||||||
|
roles: Mapping[str, RoleSpec],
|
||||||
|
pillar_skills: Mapping[str, str],
|
||||||
|
) -> ToolFactory:
|
||||||
|
consult = (
|
||||||
|
pillar_consultation_tool(runtime, roles[PILLAR_ADVISOR], pillar_skills)
|
||||||
|
if pillar_skills and PILLAR_ADVISOR in roles
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
|
||||||
|
def tools_for(role: RoleSpec, deliverable: str | None) -> Sequence[ToolSpec]:
|
||||||
|
if consult and deliverable == DeliverableKind.DESIGN_RECOMMENDATIONS.value:
|
||||||
|
return [consult]
|
||||||
|
return []
|
||||||
|
|
||||||
|
return tools_for
|
||||||
@@ -58,6 +58,8 @@ class ProviderBundle:
|
|||||||
mcp_auth: Mapping[str, Any]
|
mcp_auth: Mapping[str, Any]
|
||||||
mcp_servers: Mapping[str, McpServerSpec]
|
mcp_servers: Mapping[str, McpServerSpec]
|
||||||
role_mcp_servers: Mapping[str, tuple[str, ...]]
|
role_mcp_servers: Mapping[str, tuple[str, ...]]
|
||||||
|
role_skills: Mapping[str, tuple[str, ...]] = field(default_factory=dict)
|
||||||
|
pillar_skills: Mapping[str, str] = field(default_factory=dict)
|
||||||
|
|
||||||
def mcp_servers_for_role(
|
def mcp_servers_for_role(
|
||||||
self, role: str, include_preview: bool = True
|
self, role: str, include_preview: bool = True
|
||||||
@@ -146,6 +148,19 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
|||||||
)
|
)
|
||||||
role_mcp_servers[role] = tuple(server_ids or ())
|
role_mcp_servers[role] = tuple(server_ids or ())
|
||||||
|
|
||||||
|
role_skills = {}
|
||||||
|
for role, skill_names in (config.get("role_skills") or {}).items():
|
||||||
|
unknown = [skill for skill in skill_names or () if skill not in skills]
|
||||||
|
if unknown:
|
||||||
|
raise fail(f"role {role!r} references unknown skills: {', '.join(unknown)}")
|
||||||
|
role_skills[role] = tuple(skill_names or ())
|
||||||
|
|
||||||
|
pillar_skills = {}
|
||||||
|
for pillar, skill_name in (config.get("pillar_skills") or {}).items():
|
||||||
|
if skill_name not in skills:
|
||||||
|
raise fail(f"pillar {pillar!r} references unknown skill {skill_name!r}")
|
||||||
|
pillar_skills[str(pillar)] = skill_name
|
||||||
|
|
||||||
return ProviderBundle(
|
return ProviderBundle(
|
||||||
name=name,
|
name=name,
|
||||||
skills_dirs=skills_dirs,
|
skills_dirs=skills_dirs,
|
||||||
@@ -156,4 +171,6 @@ def load_provider(name: str, root: Path | None = None) -> ProviderBundle:
|
|||||||
mcp_auth=dict(config.get("mcp_auth") or {}),
|
mcp_auth=dict(config.get("mcp_auth") or {}),
|
||||||
mcp_servers=mcp_servers,
|
mcp_servers=mcp_servers,
|
||||||
role_mcp_servers=role_mcp_servers,
|
role_mcp_servers=role_mcp_servers,
|
||||||
|
role_skills=role_skills,
|
||||||
|
pillar_skills=pillar_skills,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -8,11 +8,34 @@ skills_dirs:
|
|||||||
|
|
||||||
output_template: skills/vendor/google/google-cloud-solution-architecture/assets/output-template.md
|
output_template: skills/vendor/google/google-cloud-solution-architecture/assets/output-template.md
|
||||||
|
|
||||||
|
# LiteLLM model ids. The -latest aliases track Google's current Pro and Flash models.
|
||||||
models:
|
models:
|
||||||
default: gemini/gemini-2.5-pro
|
default: gemini/gemini-pro-latest
|
||||||
roles:
|
roles:
|
||||||
pillar-advisor: gemini/gemini-2.5-flash
|
pillar-advisor: gemini/gemini-flash-latest
|
||||||
diagrammer: gemini/gemini-2.5-flash
|
diagrammer: gemini/gemini-flash-latest
|
||||||
|
|
||||||
|
# Skills each role may load. Role files stay provider-neutral; the bundle names Skills.
|
||||||
|
role_skills:
|
||||||
|
orchestrator: [google-cloud-solution-architecture]
|
||||||
|
requirements-analyst: [google-cloud-solution-architecture]
|
||||||
|
decomposer: [google-cloud-solution-architecture]
|
||||||
|
product-selector: [google-cloud-solution-architecture]
|
||||||
|
diagrammer: [google-cloud-solution-architecture]
|
||||||
|
describer: [google-cloud-solution-architecture]
|
||||||
|
design-advisor: [google-cloud-solution-architecture]
|
||||||
|
iac-author: [google-cloud-solution-architecture]
|
||||||
|
validator: [google-cloud-solution-architecture]
|
||||||
|
packager: [google-cloud-solution-architecture]
|
||||||
|
|
||||||
|
# One advisor per architecture-framework pillar; each loads only its pillar's Skill.
|
||||||
|
pillar_skills:
|
||||||
|
security: google-cloud-waf-security
|
||||||
|
reliability: google-cloud-waf-reliability
|
||||||
|
operational-excellence: google-cloud-waf-operational-excellence
|
||||||
|
cost-optimization: google-cloud-waf-cost-optimization
|
||||||
|
performance-optimization: google-cloud-waf-performance-optimization
|
||||||
|
sustainability: google-cloud-waf-sustainability
|
||||||
|
|
||||||
tools:
|
tools:
|
||||||
general: [mermaid_check, terraform_fmt, terraform_validate]
|
general: [mermaid_check, terraform_fmt, terraform_validate]
|
||||||
|
|||||||
@@ -15,4 +15,4 @@ click>=8.1.0
|
|||||||
psycopg2-binary>=2.9.9
|
psycopg2-binary>=2.9.9
|
||||||
sqlalchemy>=2.0.0
|
sqlalchemy>=2.0.0
|
||||||
asyncpg>=0.29.0
|
asyncpg>=0.29.0
|
||||||
google-adk>=0.1.0
|
google-adk[extensions]==2.9.0 # Skills are Experimental: pin exactly; extensions brings LiteLLM
|
||||||
|
|||||||
0
tests/agents/__init__.py
Normal file
0
tests/agents/__init__.py
Normal file
121
tests/agents/test_adk_engine.py
Normal file
121
tests/agents/test_adk_engine.py
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
"""The Execution engine running real role files on the ADK runtime, with scripted models."""
|
||||||
|
|
||||||
|
import json
|
||||||
|
from collections.abc import AsyncGenerator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from google.adk.models.base_llm import BaseLlm
|
||||||
|
from google.adk.models.llm_request import LlmRequest
|
||||||
|
from google.adk.models.llm_response import LlmResponse
|
||||||
|
from google.genai import types
|
||||||
|
from langgraph.checkpoint.memory import InMemorySaver
|
||||||
|
from pydantic import Field
|
||||||
|
|
||||||
|
from app.agents.adk_runtime import AdkAgentRuntime
|
||||||
|
from app.agents.roles import load_roles
|
||||||
|
from app.execution.deliverables import DEPENDENCY_GRAPH, DeliverableKind
|
||||||
|
from app.execution.engine import ExecutionEngine
|
||||||
|
from app.execution.recorder import InMemoryRecorder
|
||||||
|
from app.execution.toolkit import make_tool_factory
|
||||||
|
from app.providers import load_provider
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.asyncio
|
||||||
|
|
||||||
|
AGENTS_DIR = Path(__file__).parents[2] / "agents"
|
||||||
|
|
||||||
|
|
||||||
|
class RoleScriptedLlm(BaseLlm):
|
||||||
|
"""One scripted response per model call, keyed by the agent's role."""
|
||||||
|
|
||||||
|
model: str = "scripted"
|
||||||
|
script: list = Field(default_factory=list)
|
||||||
|
requests: list = Field(default_factory=list)
|
||||||
|
|
||||||
|
async def generate_content_async(
|
||||||
|
self, llm_request: LlmRequest, stream: bool = False
|
||||||
|
) -> AsyncGenerator[LlmResponse, None]:
|
||||||
|
self.requests.append(llm_request)
|
||||||
|
parts = self.script.pop(0)
|
||||||
|
yield LlmResponse(content=types.Content(role="model", parts=parts))
|
||||||
|
|
||||||
|
|
||||||
|
def call(tool: str, **args) -> list[types.Part]:
|
||||||
|
return [types.Part(function_call=types.FunctionCall(name=tool, args=args))]
|
||||||
|
|
||||||
|
|
||||||
|
def requirements_markdown() -> str:
|
||||||
|
sections = DEPENDENCY_GRAPH[DeliverableKind.REQUIREMENTS].sections
|
||||||
|
return "\n".join(f"## {section}\nDetails." for section in sections)
|
||||||
|
|
||||||
|
|
||||||
|
def task_context(request: LlmRequest) -> dict:
|
||||||
|
text = request.contents[0].parts[0].text
|
||||||
|
return json.loads(text.split("```json\n", 1)[1].rsplit("\n```", 1)[0])
|
||||||
|
|
||||||
|
|
||||||
|
async def test_engine_runs_role_files_through_adk_with_one_voice(tmp_path):
|
||||||
|
bundle = load_provider("gcp")
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle)
|
||||||
|
models = {
|
||||||
|
"orchestrator": RoleScriptedLlm(
|
||||||
|
script=[
|
||||||
|
call("delegate", kinds=["requirements"]),
|
||||||
|
call("ask_user", questions=[]),
|
||||||
|
call("delegate", kinds=["requirements"]),
|
||||||
|
call("ask_user", questions=["Anything to add?"]),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
"requirements-analyst": RoleScriptedLlm(
|
||||||
|
script=[
|
||||||
|
call("load_skill", skill_name="google-cloud-solution-architecture"),
|
||||||
|
call(
|
||||||
|
"needs_input", questions=["What availability target do you need?"]
|
||||||
|
),
|
||||||
|
call("submit_deliverable", content=requirements_markdown()),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
}
|
||||||
|
runtime = AdkAgentRuntime(
|
||||||
|
skill_dirs={name: meta.path for name, meta in bundle.skills.items()},
|
||||||
|
model_factory=lambda role: models[role.name],
|
||||||
|
)
|
||||||
|
engine = ExecutionEngine(
|
||||||
|
runtime=runtime,
|
||||||
|
roles=roles,
|
||||||
|
recorder=InMemoryRecorder(),
|
||||||
|
checkpointer=InMemorySaver(),
|
||||||
|
output_root=tmp_path,
|
||||||
|
tool_factory=make_tool_factory(runtime, roles, bundle.pillar_skills),
|
||||||
|
)
|
||||||
|
|
||||||
|
asked = await engine.start("Regional event ingestion")
|
||||||
|
|
||||||
|
assert asked.prompt == {
|
||||||
|
"questions": [
|
||||||
|
{
|
||||||
|
"from": "requirements",
|
||||||
|
"question": "What availability target do you need?",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
answered = await engine.resume(asked.execution_id, {"reply": "99.9% in one region"})
|
||||||
|
|
||||||
|
assert (
|
||||||
|
answered.state["ledger"]["deliverables"]["requirements"]["status"]
|
||||||
|
== "submitted"
|
||||||
|
)
|
||||||
|
analyst_second_run = models["requirements-analyst"].requests[-1]
|
||||||
|
assert task_context(analyst_second_run)["answers"] == [
|
||||||
|
{
|
||||||
|
"questions": ["What availability target do you need?"],
|
||||||
|
"reply": "99.9% in one region",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
orchestrator_prompt = models["orchestrator"].requests[0].config.system_instruction
|
||||||
|
assert "You coordinate one solution-architecture Execution" in orchestrator_prompt
|
||||||
|
analyst_prompt = (
|
||||||
|
models["requirements-analyst"].requests[0].config.system_instruction
|
||||||
|
)
|
||||||
|
assert "## How you work" in analyst_prompt
|
||||||
256
tests/agents/test_adk_runtime.py
Normal file
256
tests/agents/test_adk_runtime.py
Normal file
@@ -0,0 +1,256 @@
|
|||||||
|
"""The ADK implementation of AgentRuntime, driven by a scripted model (no network)."""
|
||||||
|
|
||||||
|
from collections.abc import AsyncGenerator
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from google.adk.models.base_llm import BaseLlm
|
||||||
|
from google.adk.models.lite_llm import LiteLlm
|
||||||
|
from google.adk.models.llm_request import LlmRequest
|
||||||
|
from google.adk.models.llm_response import LlmResponse
|
||||||
|
from google.genai import types
|
||||||
|
from pydantic import Field
|
||||||
|
|
||||||
|
from app.agents.adk_runtime import AdkAgentRuntime, default_model_factory
|
||||||
|
from app.agents.runtime import (
|
||||||
|
AgentTask,
|
||||||
|
BudgetExhausted,
|
||||||
|
NeedsInput,
|
||||||
|
RoleSpec,
|
||||||
|
Stopped,
|
||||||
|
Submitted,
|
||||||
|
ToolResult,
|
||||||
|
ToolSpec,
|
||||||
|
TurnEnded,
|
||||||
|
)
|
||||||
|
from app.providers import load_provider
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.asyncio
|
||||||
|
|
||||||
|
SKILLS = {name: meta.path for name, meta in load_provider("gcp").skills.items()}
|
||||||
|
|
||||||
|
|
||||||
|
class ScriptedLlm(BaseLlm):
|
||||||
|
"""Returns one scripted response per model call and records every request."""
|
||||||
|
|
||||||
|
model: str = "scripted"
|
||||||
|
script: list[list[types.Part]] = Field(default_factory=list)
|
||||||
|
requests: list[LlmRequest] = Field(default_factory=list)
|
||||||
|
|
||||||
|
async def generate_content_async(
|
||||||
|
self, llm_request: LlmRequest, stream: bool = False
|
||||||
|
) -> AsyncGenerator[LlmResponse, None]:
|
||||||
|
self.requests.append(llm_request)
|
||||||
|
parts = self.script.pop(0) if self.script else [types.Part(text="(no script)")]
|
||||||
|
yield LlmResponse(content=types.Content(role="model", parts=parts))
|
||||||
|
|
||||||
|
|
||||||
|
def call(tool: str, **args) -> types.Part:
|
||||||
|
return types.Part(function_call=types.FunctionCall(name=tool, args=args))
|
||||||
|
|
||||||
|
|
||||||
|
def text(value: str) -> types.Part:
|
||||||
|
return types.Part(text=value)
|
||||||
|
|
||||||
|
|
||||||
|
def runtime_with(model: ScriptedLlm) -> AdkAgentRuntime:
|
||||||
|
return AdkAgentRuntime(skill_dirs=SKILLS, model_factory=lambda role: model)
|
||||||
|
|
||||||
|
|
||||||
|
class Recorder:
|
||||||
|
"""ToolSpecs whose handlers record their calls."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.calls: list[tuple[str, dict]] = []
|
||||||
|
|
||||||
|
def tool(self, name: str, outcome=None, message: str = "ok") -> ToolSpec:
|
||||||
|
async def handler(**kwargs) -> ToolResult:
|
||||||
|
self.calls.append((name, kwargs))
|
||||||
|
return ToolResult(f"{message}: {kwargs}", outcome)
|
||||||
|
|
||||||
|
return ToolSpec(
|
||||||
|
name,
|
||||||
|
f"The {name} tool.",
|
||||||
|
{"type": "object", "properties": {"note": {"type": "string"}}},
|
||||||
|
handler,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
ROLE = RoleSpec(
|
||||||
|
"iac-author",
|
||||||
|
"Write Terraform. Keep {braces} and {{doubles}} verbatim.",
|
||||||
|
budget=5,
|
||||||
|
skills=("google-cloud-solution-architecture",),
|
||||||
|
)
|
||||||
|
TASK = AgentTask("iac-author", "Produce deployment_guidance.", {"feedback": ["x"]})
|
||||||
|
|
||||||
|
|
||||||
|
async def test_specialist_loads_its_skill_and_submits():
|
||||||
|
recorder = Recorder()
|
||||||
|
submitted = Submitted("# Guidance", {"terraform/main.tf": "resource {}"})
|
||||||
|
model = ScriptedLlm(
|
||||||
|
script=[
|
||||||
|
[call("load_skill", skill_name="google-cloud-solution-architecture")],
|
||||||
|
[call("submit_deliverable", note="done")],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
outcome = await runtime_with(model).run(
|
||||||
|
ROLE, TASK, [recorder.tool("submit_deliverable", submitted)]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert outcome == submitted
|
||||||
|
assert recorder.calls == [("submit_deliverable", {"note": "done"})]
|
||||||
|
skill_response = _function_responses(model.requests[1])["load_skill"]
|
||||||
|
assert "Phase 1: Requirements discovery" in str(skill_response)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_agent_is_offered_its_tools_and_skill_tools_without_scripts():
|
||||||
|
model = ScriptedLlm(script=[[text("nothing to do")]])
|
||||||
|
|
||||||
|
await runtime_with(model).run(ROLE, TASK, [Recorder().tool("submit_deliverable")])
|
||||||
|
|
||||||
|
request = model.requests[0]
|
||||||
|
assert set(request.tools_dict) == {
|
||||||
|
"submit_deliverable",
|
||||||
|
"list_skills",
|
||||||
|
"load_skill",
|
||||||
|
"load_skill_resource",
|
||||||
|
}
|
||||||
|
declaration = request.tools_dict["submit_deliverable"]._get_declaration()
|
||||||
|
assert declaration.parameters_json_schema["properties"] == {
|
||||||
|
"note": {"type": "string"}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_role_without_skills_gets_no_skill_tools():
|
||||||
|
model = ScriptedLlm(script=[[text("done")]])
|
||||||
|
role = RoleSpec("decomposer", "Decompose.", budget=3)
|
||||||
|
|
||||||
|
await runtime_with(model).run(role, TASK, [Recorder().tool("needs_input")])
|
||||||
|
|
||||||
|
assert set(model.requests[0].tools_dict) == {"needs_input"}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_only_the_roles_skills_can_be_loaded():
|
||||||
|
model = ScriptedLlm(
|
||||||
|
script=[
|
||||||
|
[call("load_skill", skill_name="google-cloud-waf-security")],
|
||||||
|
[text("ok")],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
await runtime_with(model).run(ROLE, TASK, [])
|
||||||
|
|
||||||
|
response = _function_responses(model.requests[1])["load_skill"]
|
||||||
|
assert "error" in response
|
||||||
|
assert "Well-Architected" not in str(response)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_instructions_and_task_context_reach_the_model_verbatim():
|
||||||
|
model = ScriptedLlm(script=[[text("done")]])
|
||||||
|
|
||||||
|
await runtime_with(model).run(ROLE, TASK, [])
|
||||||
|
|
||||||
|
request = model.requests[0]
|
||||||
|
assert (
|
||||||
|
"Keep {braces} and {{doubles}} verbatim." in request.config.system_instruction
|
||||||
|
)
|
||||||
|
user_text = request.contents[0].parts[0].text
|
||||||
|
assert "Produce deployment_guidance." in user_text
|
||||||
|
assert '"feedback": [\n "x"\n ]' in user_text
|
||||||
|
|
||||||
|
|
||||||
|
async def test_turn_ends_as_soon_as_a_tool_returns_an_outcome():
|
||||||
|
recorder = Recorder()
|
||||||
|
model = ScriptedLlm(
|
||||||
|
script=[
|
||||||
|
[call("delegate"), call("ask_user")],
|
||||||
|
[call("ask_user")],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
outcome = await runtime_with(model).run(
|
||||||
|
RoleSpec("orchestrator", "Coordinate.", budget=10),
|
||||||
|
TASK,
|
||||||
|
[
|
||||||
|
recorder.tool("delegate", TurnEnded("delegate")),
|
||||||
|
recorder.tool("ask_user", TurnEnded("ask_user")),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert outcome == TurnEnded("delegate")
|
||||||
|
assert [name for name, _ in recorder.calls] == ["delegate"]
|
||||||
|
assert len(model.requests) == 1
|
||||||
|
|
||||||
|
|
||||||
|
async def test_budget_counts_every_tool_call_and_stops_the_run():
|
||||||
|
recorder = Recorder()
|
||||||
|
model = ScriptedLlm(script=[[call("ready_work")] for _ in range(5)])
|
||||||
|
|
||||||
|
outcome = await runtime_with(model).run(
|
||||||
|
RoleSpec("orchestrator", "Coordinate.", budget=2),
|
||||||
|
TASK,
|
||||||
|
[recorder.tool("ready_work")],
|
||||||
|
)
|
||||||
|
|
||||||
|
assert isinstance(outcome, BudgetExhausted)
|
||||||
|
assert "2 tool calls" in outcome.summary
|
||||||
|
assert len(recorder.calls) == 2
|
||||||
|
|
||||||
|
|
||||||
|
async def test_model_reply_without_a_turn_ending_tool_is_stopped():
|
||||||
|
model = ScriptedLlm(script=[[text("I need the RTO before I can continue.")]])
|
||||||
|
|
||||||
|
outcome = await runtime_with(model).run(ROLE, TASK, [])
|
||||||
|
|
||||||
|
assert outcome == Stopped("I need the RTO before I can continue.")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_needs_input_outcome_passes_through():
|
||||||
|
needs = NeedsInput(("What region?",))
|
||||||
|
model = ScriptedLlm(script=[[call("needs_input")]])
|
||||||
|
|
||||||
|
outcome = await runtime_with(model).run(
|
||||||
|
ROLE, TASK, [Recorder().tool("needs_input", needs)]
|
||||||
|
)
|
||||||
|
|
||||||
|
assert outcome == needs
|
||||||
|
|
||||||
|
|
||||||
|
async def test_each_run_starts_with_a_fresh_session():
|
||||||
|
model = ScriptedLlm(script=[[text("first")], [text("second")]])
|
||||||
|
runtime = runtime_with(model)
|
||||||
|
|
||||||
|
await runtime.run(ROLE, TASK, [])
|
||||||
|
await runtime.run(ROLE, TASK, [])
|
||||||
|
|
||||||
|
assert len(model.requests[1].contents) == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_model_factory_wraps_the_role_model_in_litellm():
|
||||||
|
model = default_model_factory(
|
||||||
|
RoleSpec("diagrammer", "", budget=1, model="gemini/gemini-flash-latest")
|
||||||
|
)
|
||||||
|
|
||||||
|
assert isinstance(model, LiteLlm)
|
||||||
|
assert model.model == "gemini/gemini-flash-latest"
|
||||||
|
with pytest.raises(ValueError, match="model"):
|
||||||
|
default_model_factory(RoleSpec("diagrammer", "", budget=1))
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_skill_is_rejected_when_the_runtime_is_built():
|
||||||
|
with pytest.raises(ValueError, match="no-such-skill"):
|
||||||
|
AdkAgentRuntime(
|
||||||
|
skill_dirs={"no-such-skill": Path("/nonexistent")},
|
||||||
|
model_factory=lambda role: ScriptedLlm(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _function_responses(request: LlmRequest) -> dict[str, dict]:
|
||||||
|
responses = {}
|
||||||
|
for content in request.contents:
|
||||||
|
for part in content.parts or []:
|
||||||
|
if part.function_response:
|
||||||
|
responses[part.function_response.name] = part.function_response.response
|
||||||
|
return responses
|
||||||
138
tests/agents/test_roles.py
Normal file
138
tests/agents/test_roles.py
Normal file
@@ -0,0 +1,138 @@
|
|||||||
|
"""Role files (agents/*.md) resolved against a Cloud provider bundle."""
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.agents.roles import PILLAR_ADVISOR, RoleConfigError, load_roles
|
||||||
|
from app.execution.deliverables import DEPENDENCY_GRAPH
|
||||||
|
from app.providers import load_provider
|
||||||
|
|
||||||
|
AGENTS_DIR = Path(__file__).parents[2] / "agents"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def bundle():
|
||||||
|
return load_provider("gcp")
|
||||||
|
|
||||||
|
|
||||||
|
def test_every_engine_role_and_the_pillar_advisor_is_defined(bundle):
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle)
|
||||||
|
|
||||||
|
expected = {"orchestrator", PILLAR_ADVISOR}
|
||||||
|
expected |= {spec.specialist for spec in DEPENDENCY_GRAPH.values()}
|
||||||
|
assert set(roles) == expected
|
||||||
|
for role in roles.values():
|
||||||
|
assert role.instructions.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def test_roles_take_skills_mcp_servers_and_models_from_the_bundle(bundle):
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle)
|
||||||
|
|
||||||
|
assert roles["orchestrator"].skills == ("google-cloud-solution-architecture",)
|
||||||
|
assert roles[PILLAR_ADVISOR].skills == ()
|
||||||
|
assert roles["product-selector"].mcp_servers == ("developer_knowledge",)
|
||||||
|
assert "asset_inventory" in roles["requirements-analyst"].mcp_servers
|
||||||
|
assert roles["diagrammer"].model == bundle.models.for_role("diagrammer")
|
||||||
|
assert roles["orchestrator"].model == bundle.models.default
|
||||||
|
|
||||||
|
|
||||||
|
def test_preview_mcp_servers_can_be_excluded(bundle):
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle, include_preview_mcp=False)
|
||||||
|
|
||||||
|
assert "asset_inventory" not in roles["requirements-analyst"].mcp_servers
|
||||||
|
|
||||||
|
|
||||||
|
def test_budgets_default_by_role_kind(bundle):
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle, orchestrator_budget=7, specialist_budget=3)
|
||||||
|
|
||||||
|
assert roles["orchestrator"].budget == 7
|
||||||
|
assert roles["packager"].budget == 3
|
||||||
|
|
||||||
|
|
||||||
|
def test_specialists_include_the_shared_protocol_once(bundle):
|
||||||
|
roles = load_roles(AGENTS_DIR, bundle)
|
||||||
|
protocol = (AGENTS_DIR / "partials" / "specialist-protocol.md").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
for spec in DEPENDENCY_GRAPH.values():
|
||||||
|
assert roles[spec.specialist].instructions.count(protocol.strip()) == 1
|
||||||
|
assert protocol.strip() not in roles["orchestrator"].instructions
|
||||||
|
|
||||||
|
|
||||||
|
def test_role_files_name_no_provider_specific_skills():
|
||||||
|
for path in AGENTS_DIR.rglob("*.md"):
|
||||||
|
text = path.read_text(encoding="utf-8").lower()
|
||||||
|
assert "google-cloud-" not in text, path.name
|
||||||
|
|
||||||
|
|
||||||
|
# --- validation -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def write_agents(tmp_path: Path, bundle, overrides: dict[str, str]) -> Path:
|
||||||
|
"""Copy the real role files, then overwrite some."""
|
||||||
|
target = tmp_path / "agents"
|
||||||
|
for path in AGENTS_DIR.rglob("*.md"):
|
||||||
|
destination = target / path.relative_to(AGENTS_DIR)
|
||||||
|
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
destination.write_text(path.read_text(encoding="utf-8"), encoding="utf-8")
|
||||||
|
for name, text in overrides.items():
|
||||||
|
(target / name).write_text(text, encoding="utf-8")
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("overrides", "message"),
|
||||||
|
[
|
||||||
|
({"decomposer.md": "---\nrole: other\n---\nBody"}, "match"),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"decomposer.md": "---\nrole: decomposer\ndeliverable: requirements\n---\nB"
|
||||||
|
},
|
||||||
|
"requirements",
|
||||||
|
),
|
||||||
|
({"decomposer.md": "---\nrole: decomposer\n---\nBody"}, "deliverable"),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"decomposer.md": "---\nrole: decomposer\n"
|
||||||
|
"deliverable: technical_decomposition\ntool_tiers: [write]\n---\nB"
|
||||||
|
},
|
||||||
|
"write",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"decomposer.md": "---\nrole: decomposer\n"
|
||||||
|
"deliverable: technical_decomposition\nincludes: [missing]\n---\nB"
|
||||||
|
},
|
||||||
|
"missing",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"decomposer.md": "---\nrole: decomposer\n"
|
||||||
|
"deliverable: technical_decomposition\nskills: [x]\n---\nB"
|
||||||
|
},
|
||||||
|
"skills",
|
||||||
|
),
|
||||||
|
(
|
||||||
|
{
|
||||||
|
"decomposer.md": "---\nrole: decomposer\n"
|
||||||
|
"deliverable: technical_decomposition\n---\n "
|
||||||
|
},
|
||||||
|
"instructions",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_invalid_role_files_are_rejected(tmp_path, bundle, overrides, message):
|
||||||
|
agents_dir = write_agents(tmp_path, bundle, overrides)
|
||||||
|
|
||||||
|
with pytest.raises(RoleConfigError, match=message):
|
||||||
|
load_roles(agents_dir, bundle)
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_role_file_is_rejected(tmp_path, bundle):
|
||||||
|
agents_dir = write_agents(tmp_path, bundle, {})
|
||||||
|
(agents_dir / "packager.md").unlink()
|
||||||
|
|
||||||
|
with pytest.raises(RoleConfigError, match="packager"):
|
||||||
|
load_roles(agents_dir, bundle)
|
||||||
@@ -9,6 +9,7 @@ from app.execution.deliverables import (
|
|||||||
PhaseId,
|
PhaseId,
|
||||||
check_submission,
|
check_submission,
|
||||||
deliverables_in,
|
deliverables_in,
|
||||||
|
inputs_of,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -78,6 +79,25 @@ def test_graph_matches_the_upstream_skill_dependencies():
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_inputs_are_earlier_phases_plus_indirect_dependencies_in_graph_order():
|
||||||
|
assert inputs_of(DeliverableKind.REQUIREMENTS) == ()
|
||||||
|
assert inputs_of(DeliverableKind.DEPLOYMENT_GUIDANCE) == (
|
||||||
|
DeliverableKind.REQUIREMENTS,
|
||||||
|
DeliverableKind.TECHNICAL_DECOMPOSITION,
|
||||||
|
DeliverableKind.PRODUCT_SELECTION,
|
||||||
|
DeliverableKind.DESIGN_RECOMMENDATIONS,
|
||||||
|
)
|
||||||
|
assert inputs_of(DeliverableKind.ARCHITECTURE_DESCRIPTION) == (
|
||||||
|
DeliverableKind.REQUIREMENTS,
|
||||||
|
DeliverableKind.TECHNICAL_DECOMPOSITION,
|
||||||
|
DeliverableKind.PRODUCT_SELECTION,
|
||||||
|
DeliverableKind.ARCHITECTURE_DIAGRAM,
|
||||||
|
)
|
||||||
|
assert set(inputs_of(DeliverableKind.SOLUTION_GUIDE)) == set(DeliverableKind) - {
|
||||||
|
DeliverableKind.SOLUTION_GUIDE
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def test_submission_meeting_the_contract_has_nothing_missing():
|
def test_submission_meeting_the_contract_has_nothing_missing():
|
||||||
content = (
|
content = (
|
||||||
"# Deployment guidance\n"
|
"# Deployment guidance\n"
|
||||||
|
|||||||
@@ -151,7 +151,12 @@ async def test_specialist_receives_its_dependencies_content(tmp_path):
|
|||||||
r for r in runtime.runs if r.role == specialist(K.ARCHITECTURE_DESCRIPTION)
|
r for r in runtime.runs if r.role == specialist(K.ARCHITECTURE_DESCRIPTION)
|
||||||
)
|
)
|
||||||
dependencies = describer_run.task.context["dependencies"]
|
dependencies = describer_run.task.context["dependencies"]
|
||||||
assert set(dependencies) == {"architecture_diagram", "product_selection"}
|
assert list(dependencies) == [
|
||||||
|
"requirements",
|
||||||
|
"technical_decomposition",
|
||||||
|
"product_selection",
|
||||||
|
"architecture_diagram",
|
||||||
|
]
|
||||||
assert "architecture.mmd" in dependencies["architecture_diagram"]["attachments"]
|
assert "architecture.mmd" in dependencies["architecture_diagram"]["attachments"]
|
||||||
assert describer_run.task.deliverable == "architecture_description"
|
assert describer_run.task.deliverable == "architecture_description"
|
||||||
|
|
||||||
|
|||||||
110
tests/execution/test_pillars.py
Normal file
110
tests/execution/test_pillars.py
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
"""The design advisor's fan-out to one advisor per architecture-framework pillar."""
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from app.agents.fake_runtime import Call, FakeAgentRuntime
|
||||||
|
from app.agents.roles import PILLAR_ADVISOR
|
||||||
|
from app.agents.runtime import RoleSpec, ToolSpec
|
||||||
|
from app.execution.pillars import CONSULT_TOOL, pillar_consultation_tool
|
||||||
|
from app.execution.toolkit import make_tool_factory
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.asyncio
|
||||||
|
|
||||||
|
PILLARS = {
|
||||||
|
"security": "waf-security",
|
||||||
|
"reliability": "waf-reliability",
|
||||||
|
"sustainability": "waf-sustainability",
|
||||||
|
}
|
||||||
|
ADVISOR = RoleSpec(PILLAR_ADVISOR, "Advise on one pillar.", budget=5)
|
||||||
|
|
||||||
|
|
||||||
|
def advice(pillar: str) -> list[Call]:
|
||||||
|
return [Call("submit_advice", {"content": f"Use strong defaults for {pillar}."})]
|
||||||
|
|
||||||
|
|
||||||
|
def turn_for_pillar(task):
|
||||||
|
return advice(task.context["pillar"])
|
||||||
|
|
||||||
|
|
||||||
|
async def consult(tool: ToolSpec, **args):
|
||||||
|
return await tool.handler(**args)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_consults_every_pillar_with_only_its_skill():
|
||||||
|
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [turn_for_pillar] * 3})
|
||||||
|
tool = pillar_consultation_tool(runtime, ADVISOR, PILLARS)
|
||||||
|
|
||||||
|
result = await consult(tool, brief="Event ingestion on serverless compute.")
|
||||||
|
|
||||||
|
assert result.outcome is None
|
||||||
|
for pillar in PILLARS:
|
||||||
|
assert f"## {pillar}" in result.message
|
||||||
|
assert f"Use strong defaults for {pillar}." in result.message
|
||||||
|
runs = {run.task.context["pillar"]: run for run in runtime.runs}
|
||||||
|
assert set(runs) == set(PILLARS)
|
||||||
|
assert (
|
||||||
|
runs["security"].task.context["brief"]
|
||||||
|
== "Event ingestion on serverless compute."
|
||||||
|
)
|
||||||
|
assert runtime.runs[0].tools == ("submit_advice",)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_advisors_are_scoped_to_their_pillar_skill():
|
||||||
|
seen: dict[str, tuple[str, ...]] = {}
|
||||||
|
|
||||||
|
class RecordingRuntime(FakeAgentRuntime):
|
||||||
|
async def run(self, role, task, tools):
|
||||||
|
seen[task.context["pillar"]] = role.skills
|
||||||
|
return await super().run(role, task, tools)
|
||||||
|
|
||||||
|
runtime = RecordingRuntime({PILLAR_ADVISOR: [turn_for_pillar] * 3})
|
||||||
|
|
||||||
|
await consult(pillar_consultation_tool(runtime, ADVISOR, PILLARS), brief="b")
|
||||||
|
|
||||||
|
assert seen == {pillar: (skill,) for pillar, skill in PILLARS.items()}
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_subset_of_pillars_can_be_consulted():
|
||||||
|
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [turn_for_pillar]})
|
||||||
|
tool = pillar_consultation_tool(runtime, ADVISOR, PILLARS)
|
||||||
|
|
||||||
|
result = await consult(tool, brief="b", pillars=["reliability"])
|
||||||
|
|
||||||
|
assert "## reliability" in result.message
|
||||||
|
assert "## security" not in result.message
|
||||||
|
|
||||||
|
|
||||||
|
async def test_unknown_pillars_are_refused():
|
||||||
|
tool = pillar_consultation_tool(FakeAgentRuntime({}), ADVISOR, PILLARS)
|
||||||
|
|
||||||
|
result = await consult(tool, brief="b", pillars=["speed"])
|
||||||
|
|
||||||
|
assert result.message.startswith("Refused")
|
||||||
|
assert "security" in result.message
|
||||||
|
|
||||||
|
|
||||||
|
async def test_an_advisor_that_fails_is_reported_without_losing_the_others():
|
||||||
|
def maybe_fail(task):
|
||||||
|
if task.context["pillar"] == "reliability":
|
||||||
|
return [Call("submit_advice", {"content": ""})]
|
||||||
|
return advice(task.context["pillar"])
|
||||||
|
|
||||||
|
runtime = FakeAgentRuntime({PILLAR_ADVISOR: [maybe_fail] * 3})
|
||||||
|
|
||||||
|
result = await consult(
|
||||||
|
pillar_consultation_tool(runtime, ADVISOR, PILLARS), brief="b"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "Use strong defaults for security." in result.message
|
||||||
|
assert "## reliability\n\nNo guidance:" in result.message
|
||||||
|
|
||||||
|
|
||||||
|
async def test_tool_factory_gives_the_consultation_only_to_the_design_advisor():
|
||||||
|
roles = {PILLAR_ADVISOR: ADVISOR}
|
||||||
|
factory = make_tool_factory(FakeAgentRuntime({}), roles, PILLARS)
|
||||||
|
|
||||||
|
design = factory(RoleSpec("design-advisor", "", budget=5), "design_recommendations")
|
||||||
|
other = factory(RoleSpec("decomposer", "", budget=5), "technical_decomposition")
|
||||||
|
|
||||||
|
assert [tool.name for tool in design] == [CONSULT_TOOL]
|
||||||
|
assert list(other) == []
|
||||||
@@ -60,6 +60,20 @@ def test_role_mcp_servers_can_exclude_preview_servers():
|
|||||||
assert bundle.mcp_servers_for_role("no-such-role") == ()
|
assert bundle.mcp_servers_for_role("no-such-role") == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_gcp_bundle_maps_roles_and_pillars_to_vendored_skills():
|
||||||
|
bundle = load_provider("gcp")
|
||||||
|
|
||||||
|
assert bundle.role_skills["orchestrator"] == ("google-cloud-solution-architecture",)
|
||||||
|
assert bundle.pillar_skills == {
|
||||||
|
"security": "google-cloud-waf-security",
|
||||||
|
"reliability": "google-cloud-waf-reliability",
|
||||||
|
"operational-excellence": "google-cloud-waf-operational-excellence",
|
||||||
|
"cost-optimization": "google-cloud-waf-cost-optimization",
|
||||||
|
"performance-optimization": "google-cloud-waf-performance-optimization",
|
||||||
|
"sustainability": "google-cloud-waf-sustainability",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def test_model_for_role_falls_back_to_default():
|
def test_model_for_role_falls_back_to_default():
|
||||||
bundle = load_provider("gcp")
|
bundle = load_provider("gcp")
|
||||||
|
|
||||||
@@ -154,6 +168,8 @@ def test_minimal_bundle_loads(tmp_path):
|
|||||||
"allow",
|
"allow",
|
||||||
),
|
),
|
||||||
({"role_mcp_servers": {"describer": ["missing"]}}, "missing"),
|
({"role_mcp_servers": {"describer": ["missing"]}}, "missing"),
|
||||||
|
({"role_skills": {"describer": ["no-such-skill"]}}, "no-such-skill"),
|
||||||
|
({"pillar_skills": {"security": "no-such-skill"}}, "no-such-skill"),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_invalid_bundles_are_rejected(tmp_path, override, message):
|
def test_invalid_bundles_are_rejected(tmp_path, override, message):
|
||||||
|
|||||||
Reference in New Issue
Block a user