refactored: to utilise the google adk and production grade agent
Some checks failed
validation / verify (push) Failing after 10s
Some checks failed
validation / verify (push) Failing after 10s
This commit is contained in:
38
.env.example
Normal file
38
.env.example
Normal file
@@ -0,0 +1,38 @@
|
||||
# GCP Solution Architecture Agent Environment Variables
|
||||
|
||||
# Agent Identity & Server Config
|
||||
AGENT_NAME=gcp_solution_architecture_agent
|
||||
AGENT_VERSION=1.0.0
|
||||
LOG_LEVEL=INFO
|
||||
HOST=0.0.0.0
|
||||
PORT=8080
|
||||
|
||||
# LLM Provider Configuration (OpenAI or Azure OpenAI)
|
||||
OPENAI_API_KEY=your-openai-api-key-here
|
||||
AZURE_OPENAI_API_KEY=your-azure-openai-api-key-here
|
||||
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
|
||||
AZURE_OPENAI_DEPLOYMENT=gpt-4o
|
||||
AZURE_OPENAI_API_VERSION=2024-02-15-preview
|
||||
LLM_TEMPERATURE=0.2
|
||||
|
||||
# Google Cloud Project & Scanner Credentials
|
||||
GOOGLE_APPLICATION_CREDENTIALS=/app/credentials/gcp-key.json
|
||||
GCP_PROJECT_ID=your-gcp-project-id
|
||||
GCP_REGION=us-central1
|
||||
|
||||
# PostgreSQL Database & State Storage
|
||||
POSTGRES_HOST=localhost
|
||||
POSTGRES_PORT=5432
|
||||
POSTGRES_DB=gcp_agent_db
|
||||
POSTGRES_USER=postgres
|
||||
POSTGRES_PASSWORD=postgres
|
||||
# DATABASE_URL=postgresql://postgres:postgres@localhost:5432/gcp_agent_db
|
||||
SQLITE_FALLBACK_DB=gcp_agent_local.db
|
||||
|
||||
# Google ADK Multi-Agent Settings
|
||||
ADK_MAX_LOOP_ITERATIONS=5
|
||||
ADK_ENABLE_ARTIFACT_STORE=true
|
||||
|
||||
# Google Developer Knowledge MCP Integration
|
||||
DEVELOPER_KNOWLEDGE_MCP_URL=https://developerknowledge.googleapis.com/mcp
|
||||
DEVELOPER_KNOWLEDGE_MCP_ENABLED=true
|
||||
30
.gitignore
vendored
Normal file
30
.gitignore
vendored
Normal file
@@ -0,0 +1,30 @@
|
||||
# Environment variables
|
||||
.env
|
||||
*.env
|
||||
|
||||
# Local SQLite databases
|
||||
*.db
|
||||
gcp_agent_local.db
|
||||
|
||||
# Python & pytest caches
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.pyo
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
|
||||
# Virtual Environments
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
|
||||
# Terraform local state & caches
|
||||
.terraform/
|
||||
*.tfstate
|
||||
*.tfstate.backup
|
||||
tfplan
|
||||
|
||||
# Container logs & temp files
|
||||
*.log
|
||||
.DS_Store
|
||||
52
Dockerfile
52
Dockerfile
@@ -1 +1,51 @@
|
||||
# TODO: generation subagent fills this in.
|
||||
FROM python:3.12-slim AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install build dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
curl \
|
||||
ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy dependency specifications and install wheels
|
||||
COPY requirements.txt ./
|
||||
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
|
||||
|
||||
# Final production image stage
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONPATH=/app \
|
||||
PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Create non-root application user for container security
|
||||
RUN groupadd -g 10001 appgroup && \
|
||||
useradd -u 10001 -g appgroup -s /bin/sh -m appuser
|
||||
|
||||
# Copy installed dependencies from builder
|
||||
COPY --from=builder /install /usr/local
|
||||
|
||||
# Copy application source, skills, and configuration
|
||||
COPY app/ ./app/
|
||||
COPY eval/ ./eval/
|
||||
COPY docs/ ./docs/
|
||||
COPY deliverables/ ./deliverables/
|
||||
COPY terraform/ ./terraform/
|
||||
COPY workflow.yaml ./
|
||||
COPY requirements.yaml ./
|
||||
|
||||
RUN chown -R appuser:appgroup /app
|
||||
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=5s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')" || exit 1
|
||||
|
||||
CMD ["python", "-m", "app.agent", "--host", "0.0.0.0", "--port", "8080"]
|
||||
|
||||
|
||||
184
README.md
184
README.md
@@ -1,22 +1,176 @@
|
||||
# gcp_solution_architecture_agent
|
||||
# GCP Solution Architecture Agent (`gcp_solution_architecture_agent`)
|
||||
|
||||
A four-phase Google Cloud solution-architecture workflow packaged as one reviewable repository.
|
||||
A production-ready Google Cloud Solution Architecture Agent built using **Google ADK (Agent Development Kit)**, **LangChain**, and **LangGraph**, featuring a multi-agent orchestration graph with iterative review loops, dynamic local skill loading, PostgreSQL database persistence, a Starlette REST API server, and an offline evaluation harness.
|
||||
|
||||
## Workflow
|
||||
1. **Discover** requirements without choosing products (`docs/requirements.md`).
|
||||
2. **Design** the Google Cloud architecture and Terraform (`docs/architecture.md`, `architecture.mmd`, `terraform/`).
|
||||
3. **Validate** artifacts without provisioning (`scripts/validate.sh`, `tests/`).
|
||||
4. **Package** the approved result (`solution-architecture-guide.md`).
|
||||
---
|
||||
|
||||
The repository is derived from the `workflow_agent` template and is intentionally safe to run without a cloud deployment. The Terraform plan requires credentials and a project only when a user chooses to run it.
|
||||
## 🌟 Architecture & Features
|
||||
|
||||
## Commands
|
||||
### 1. Multi-Agent Orchestration & `google.adk.agents.LoopAgent`
|
||||
The agent uses an end-to-end multi-agent graph managed by an `OrchestratorLoopAgent` (`google.adk.agents.LoopAgent`):
|
||||
- **`SourceDiscoveryAgent`** (Phase 0a): Pre-emptively audits and documents the existing pre-migration source environment (As-Is Architecture) and pain points.
|
||||
- **`DiscoveryAgent`** (Phase 0): Discovers functional & non-functional requirements while explicitly deferring product selection.
|
||||
- **`ArchitectureDesignAgent`** (Phase 1): Resolves product selections (Cloud Run, Pub/Sub, Cloud Storage, Firestore, IAM, KMS), grounds guidance with **Google Developer Knowledge MCP**, generates Mermaid diagrams (`architecture.mmd`), and builds Terraform IaC (`terraform/main.tf`).
|
||||
- **`ValidationReviewAgent`** (Phase 2): Evaluates Mermaid syntax, Terraform structural integrity, and artifact completeness using `google.adk.tools.FunctionTool`.
|
||||
- **`PackagingAgent`** (Phase 3): Compiles the final comprehensive `solution-architecture-guide.md` with **Before (Source) vs After (Target)** architecture comparison.
|
||||
- **`OrchestratorLoopAgent`**: Repeatedly loops through the multi-agent pipeline, reviewing and validating content in a feedback loop until production quality criteria pass or `max_iterations` is reached.
|
||||
|
||||
### 2. Google ADK Integration (`google.adk`)
|
||||
Built directly on Google ADK framework primitives (with offline compatibility stubs):
|
||||
- **`google.adk.agents`**: `LlmAgent`, `SequentialAgent`, `ParallelAgent`, `LoopAgent`.
|
||||
- **`google.adk.tools`**: `FunctionTool` wrappers for static artifact and diagram validators, plus Google Developer Knowledge MCP tools (`developerknowledge:*`).
|
||||
- **`google.adk.artifacts`**: `ArtifactRepository` synchronized with disk and PostgreSQL.
|
||||
- **`google.adk.sessions`**: `PostgresSessionService` for durable session memory.
|
||||
- **`google.adk.runners`**: `Runner` for managing execution workflows.
|
||||
- **`google.adk.workflows`**: `Workflow` composition primitives.
|
||||
- **`google.adk.evaluation`**: `Evaluator` for scoring benchmark suites.
|
||||
|
||||
### 3. PostgreSQL Database Persistence (`app/database.py`)
|
||||
State, sessions, workflow executions, review iteration logs, and evaluation metrics are written to an external PostgreSQL database (with an automatic SQLite fallback for offline development):
|
||||
- `adk_sessions`: Durable session memory and state data.
|
||||
- `workflow_executions`: Workflow execution status, phase, and iteration metrics.
|
||||
- `adk_artifacts`: Generated solution architecture artifacts (including `docs/source-architecture.md` and `source-architecture.mmd`).
|
||||
- `orchestrator_review_logs`: Detailed iteration review logs from the `OrchestratorLoopAgent`.
|
||||
- `adk_evaluations`: Benchmark evaluation scores and pass rates.
|
||||
|
||||
### 4. Dynamic Local Skill Loading (`app/skills/`)
|
||||
Domain knowledge is stored as self-contained markdown instruction templates under `app/skills/`:
|
||||
- `source_discovery/SKILL.md` (Phase 0a - Pre-emptive Source Environment Discovery)
|
||||
- `requirements_discovery/SKILL.md` (Phase 0 - Requirements Discovery & Deferral)
|
||||
- `architecture_design/SKILL.md` (Phase 1 - Product Selection & Terraform IaC)
|
||||
- `validation_rules/SKILL.md` (Phase 2 - Pre-deployment Validation Rules)
|
||||
- `packaging_guide/SKILL.md` (Phase 3 - Solution Guide Packaging)
|
||||
|
||||
The `SkillLoader` (`app/skills/loader.py`) dynamically parses frontmatter and injects instructions into phase system prompts.
|
||||
|
||||
### 5. Starlette REST API Server (`app/agent.py`)
|
||||
Production HTTP REST endpoints exposed via Starlette and Uvicorn:
|
||||
- `GET /health`: Operational health signal & database connection status.
|
||||
- `GET /card`: Agent capability card metadata.
|
||||
- `POST /generate`: Trigger the ADK multi-agent workflow execution.
|
||||
- `POST /validate`: Execute pre-deployment artifact validation.
|
||||
- `GET /sessions/{session_id}`: Query session state from the PostgreSQL database.
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start & Commands
|
||||
|
||||
### Install Dependencies
|
||||
```bash
|
||||
terraform -chdir=terraform fmt -check -recursive
|
||||
terraform -chdir=terraform init -backend=false
|
||||
terraform -chdir=terraform validate
|
||||
python3 -m unittest discover -s tests -v
|
||||
bash scripts/validate.sh
|
||||
pip install -r requirements.txt
|
||||
pip install -r requirements-dev.txt
|
||||
```
|
||||
|
||||
### Run Tests & Validation
|
||||
```bash
|
||||
# Run pytest test suite (100% pass rate)
|
||||
pytest -v
|
||||
|
||||
# Run offline benchmark evaluation harness
|
||||
python3 -m eval.eval_harness
|
||||
|
||||
# Run static artifact validation
|
||||
python3 scripts/validate_artifacts.py
|
||||
```
|
||||
|
||||
### Start Starlette REST Server
|
||||
```bash
|
||||
python3 app/agent.py --host 0.0.0.0 --port 8080
|
||||
```
|
||||
|
||||
### 🐳 Run with Podman
|
||||
|
||||
#### 1. Build the Podman Container Image
|
||||
```bash
|
||||
podman build -t gcp-solution-architecture-agent:latest .
|
||||
```
|
||||
|
||||
#### 2. Run Container with Host Deliverables Volume Mount
|
||||
To ensure execution deliverables (`deliverables/executions/{execution_id}/*`) are written directly to your host machine's workspace:
|
||||
|
||||
```bash
|
||||
podman run -d --name gcp-agent \
|
||||
-p 8080:8080 \
|
||||
-v $(pwd)/deliverables:/app/deliverables:Z \
|
||||
-v ~/.config/gcloud/application_default_credentials.json:/app/credentials/adc.json:ro \
|
||||
-e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials/adc.json \
|
||||
-e GCP_PROJECT_ID=your-project-id \
|
||||
localhost/gcp-solution-architecture-agent:latest
|
||||
```
|
||||
|
||||
#### 3. Run Connected to External PostgreSQL Container
|
||||
```bash
|
||||
podman run -d --name gcp-agent \
|
||||
-p 8080:8080 \
|
||||
-e POSTGRES_HOST=postgres-container-name \
|
||||
-e POSTGRES_PORT=5432 \
|
||||
-e POSTGRES_USER=postgres \
|
||||
-e POSTGRES_PASSWORD=postgres \
|
||||
-e POSTGRES_DB=gcp_agent_db \
|
||||
localhost/gcp-solution-architecture-agent:latest
|
||||
```
|
||||
|
||||
#### 4. Test API Endpoints
|
||||
```bash
|
||||
# Health Signal
|
||||
curl http://localhost:8080/health
|
||||
|
||||
# Agent Capability Card
|
||||
curl http://localhost:8080/card
|
||||
|
||||
# Run Multi-Agent ADK Workflow
|
||||
curl -X POST http://localhost:8080/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"request": "Event-driven regional HTTP application in Podman"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 Repository Structure
|
||||
```
|
||||
gcp_solution_architecture_agent/
|
||||
├── app/
|
||||
│ ├── adk/ # Google ADK framework primitives & fallbacks
|
||||
│ │ ├── agents.py # Multi-agents & OrchestratorLoopAgent
|
||||
│ │ ├── artifacts.py # PostgresArtifactRepository
|
||||
│ │ ├── compat.py # ADK compatibility layer
|
||||
│ │ ├── evaluation.py # ADKEvaluator
|
||||
│ │ ├── runners.py # ADKAgentRunner
|
||||
│ │ ├── sessions.py # PostgresSessionService
|
||||
│ │ ├── tools.py # ADK FunctionTools
|
||||
│ │ └── workflows.py # ADK Workflow composition
|
||||
│ ├── agent.py # Starlette REST server entrypoint
|
||||
│ ├── card.py # AgentCard & capability metadata
|
||||
│ ├── config.py # Typed settings (Pydantic BaseSettings)
|
||||
│ ├── database.py # PostgreSQL database persistence manager
|
||||
│ ├── nodes/ # Phase execution node handlers
|
||||
│ ├── skills/ # Local SKILL.md instruction templates & SkillLoader
|
||||
│ ├── states/ # GCPArchitectureState schema definition
|
||||
│ ├── tools/ # LangChain & ADK validation tools
|
||||
│ └── workflows/ # LangGraph StateGraph & Starlette REST routes
|
||||
├── eval/ # ADK Evaluation & Optimization Harness
|
||||
│ ├── datasets/ # Benchmark test cases (JSON)
|
||||
│ ├── eval_harness.py # Benchmark runner CLI
|
||||
│ ├── metrics.py # Scoring rubrics & evaluation logic
|
||||
│ ├── optimizer.py # Prompt tuning optimizer
|
||||
│ └── test_eval_harness.py # Evaluation unit tests
|
||||
├── deliverables/ # Scalable deliverable repository (categorized & versioned)
|
||||
│ ├── as-is/ # Pre-emptive Source Environment Deliverables (Before State)
|
||||
│ │ ├── source-architecture.md
|
||||
│ │ └── source-architecture.mmd
|
||||
│ ├── target/ # Target Architecture Deliverables (After State)
|
||||
│ │ ├── requirements.md
|
||||
│ │ ├── architecture.md
|
||||
│ │ └── architecture.mmd
|
||||
│ ├── validation/ # Pre-deployment Validation Reports
|
||||
│ │ └── validation-results.md
|
||||
│ ├── guides/ # Consolidate Solution Architecture Guides
|
||||
│ │ └── solution-architecture-guide.md
|
||||
│ └── executions/ # Per-Execution Versioned Snapshots ({execution_id}/*)
|
||||
├── docs/ # Specifications, audit notes & verification records
|
||||
├── terraform/ # Deployable GCP Terraform IaC blueprint
|
||||
├── scripts/
|
||||
│ └── validate_artifacts.py # Static artifact validation script
|
||||
└── tests/ # Unit & integration test suite
|
||||
```
|
||||
|
||||
`terraform plan` is optional and must be run with an explicitly supplied project and credentials; CI only performs static validation.
|
||||
|
||||
@@ -1,13 +1,21 @@
|
||||
# Validation runbook
|
||||
# Validation Runbook
|
||||
|
||||
Run from the repository root without credentials:
|
||||
|
||||
```bash
|
||||
# 1. Static artifact & guide section validation
|
||||
python3 scripts/validate_artifacts.py
|
||||
|
||||
# 2. Pytest unit & integration test suite
|
||||
pytest -v
|
||||
|
||||
# 3. Offline ADK evaluation benchmark suite
|
||||
python3 -m eval.eval_harness
|
||||
|
||||
# 4. Optional Terraform formatting & validation
|
||||
terraform -chdir=terraform fmt -check -recursive
|
||||
terraform -chdir=terraform init -backend=false
|
||||
terraform -chdir=terraform validate
|
||||
python3 scripts/validate_artifacts.py
|
||||
python3 -m pytest -q
|
||||
```
|
||||
|
||||
`init -backend=false` may download the pinned provider constraint but does not create infrastructure. Do not use `terraform apply` as part of validation. A deployment candidate additionally requires an operator-supplied `terraform plan -out=tfplan` and human review.
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
31
app/adk/__init__.py
Normal file
31
app/adk/__init__.py
Normal file
@@ -0,0 +1,31 @@
|
||||
"""ADK package for GCP Solution Architecture Agent."""
|
||||
|
||||
from .agents import (
|
||||
ArchitectureDesignAgent,
|
||||
DiscoveryAgent,
|
||||
OrchestratorLoopAgent,
|
||||
PackagingAgent,
|
||||
ValidationReviewAgent,
|
||||
build_adk_multi_agent_system,
|
||||
)
|
||||
from .artifacts import PostgresArtifactRepository
|
||||
from .evaluation import ADKEvaluator
|
||||
from .runners import ADKAgentRunner
|
||||
from .sessions import PostgresSessionService
|
||||
from .tools import ADK_TOOLS
|
||||
from .workflows import create_gcp_adk_workflow
|
||||
|
||||
__all__ = [
|
||||
"DiscoveryAgent",
|
||||
"ArchitectureDesignAgent",
|
||||
"ValidationReviewAgent",
|
||||
"PackagingAgent",
|
||||
"OrchestratorLoopAgent",
|
||||
"build_adk_multi_agent_system",
|
||||
"PostgresArtifactRepository",
|
||||
"PostgresSessionService",
|
||||
"ADKAgentRunner",
|
||||
"ADKEvaluator",
|
||||
"ADK_TOOLS",
|
||||
"create_gcp_adk_workflow",
|
||||
]
|
||||
196
app/adk/agents.py
Normal file
196
app/adk/agents.py
Normal file
@@ -0,0 +1,196 @@
|
||||
"""ADK Multi-Agent Architecture for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.agents primitives:
|
||||
- SourceDiscoveryAgent (LlmAgent)
|
||||
- DiscoveryAgent (LlmAgent)
|
||||
- ArchitectureDesignAgent (LlmAgent)
|
||||
- ValidationReviewAgent (LlmAgent)
|
||||
- PackagingAgent (LlmAgent)
|
||||
- OrchestratorLoopAgent (LoopAgent)
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app.adk.compat import BaseAgent, LlmAgent, LoopAgent, SequentialAgent
|
||||
from app.adk.tools import ADK_TOOLS
|
||||
from app.config import get_settings
|
||||
from app.database import get_db_manager
|
||||
from app.nodes import design_node, discover_node, package_node, source_discover_node, validate_node
|
||||
from app.skills.loader import SkillLoader
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SourceDiscoveryAgent(LlmAgent):
|
||||
"""ADK Agent responsible for Phase 0a Pre-emptive Source Environment Discovery."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader) -> None:
|
||||
super().__init__(
|
||||
name="SourceDiscoveryAgent",
|
||||
description="Audits and documents the pre-existing source environment (As-Is Architecture) before target migration.",
|
||||
instruction=skill_loader.format_skills_for_prompt("source_discover"),
|
||||
tools=[],
|
||||
)
|
||||
self.skill_loader = skill_loader
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
logger.info("Executing SourceDiscoveryAgent")
|
||||
res = source_discover_node(state, self.skill_loader)
|
||||
state_copy = dict(state)
|
||||
state_copy.update(res)
|
||||
return state_copy
|
||||
|
||||
|
||||
class DiscoveryAgent(LlmAgent):
|
||||
"""ADK Agent responsible for Phase 0 Requirements Discovery."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader) -> None:
|
||||
super().__init__(
|
||||
name="DiscoveryAgent",
|
||||
description="Extracts functional and non-functional requirements with product selection deferred.",
|
||||
instruction=skill_loader.format_skills_for_prompt("discover"),
|
||||
tools=[],
|
||||
)
|
||||
self.skill_loader = skill_loader
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
logger.info("Executing DiscoveryAgent")
|
||||
res = discover_node(state, self.skill_loader)
|
||||
state_copy = dict(state)
|
||||
state_copy.update(res)
|
||||
return state_copy
|
||||
|
||||
|
||||
class ArchitectureDesignAgent(LlmAgent):
|
||||
"""ADK Agent responsible for Phase 1 Product Selection, Diagrams, & Terraform IaC."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader) -> None:
|
||||
super().__init__(
|
||||
name="ArchitectureDesignAgent",
|
||||
description="Selects GCP products, generates Mermaid diagram, and produces Terraform IaC grounded by Developer Knowledge MCP.",
|
||||
instruction=skill_loader.format_skills_for_prompt("design"),
|
||||
tools=ADK_TOOLS,
|
||||
)
|
||||
self.skill_loader = skill_loader
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
logger.info("Executing ArchitectureDesignAgent")
|
||||
res = design_node(state, self.skill_loader)
|
||||
state_copy = dict(state)
|
||||
state_copy.update(res)
|
||||
return state_copy
|
||||
|
||||
|
||||
class ValidationReviewAgent(LlmAgent):
|
||||
"""ADK Agent responsible for Phase 2 Artifact Review & Quality Validation."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader) -> None:
|
||||
super().__init__(
|
||||
name="ValidationReviewAgent",
|
||||
description="Validates Mermaid syntax, Terraform configuration, and required sections.",
|
||||
instruction=skill_loader.format_skills_for_prompt("validate"),
|
||||
tools=ADK_TOOLS,
|
||||
)
|
||||
self.skill_loader = skill_loader
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
logger.info("Executing ValidationReviewAgent")
|
||||
res = validate_node(state, self.skill_loader)
|
||||
state_copy = dict(state)
|
||||
state_copy.update(res)
|
||||
return state_copy
|
||||
|
||||
|
||||
class PackagingAgent(LlmAgent):
|
||||
"""ADK Agent responsible for Phase 3 Solution Guide Packaging."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader) -> None:
|
||||
super().__init__(
|
||||
name="PackagingAgent",
|
||||
description="Packages final solution-architecture-guide.md.",
|
||||
instruction=skill_loader.format_skills_for_prompt("package"),
|
||||
tools=[],
|
||||
)
|
||||
self.skill_loader = skill_loader
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
logger.info("Executing PackagingAgent")
|
||||
res = package_node(state, self.skill_loader)
|
||||
state_copy = dict(state)
|
||||
state_copy.update(res)
|
||||
return state_copy
|
||||
|
||||
|
||||
class OrchestratorLoopAgent(LoopAgent):
|
||||
"""ADK Orchestrator LoopAgent that coordinates multi-agent execution & iterative quality review.
|
||||
|
||||
Uses google.adk.agents.LoopAgent to execute discovery -> design -> validation review -> packaging
|
||||
in a loop until validation passes 100% or max_iterations is reached. Writes every review iteration
|
||||
event to PostgreSQL database (`orchestrator_review_logs`).
|
||||
"""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader, max_iterations: Optional[int] = None) -> None:
|
||||
settings = get_settings()
|
||||
max_iters = max_iterations or settings.ADK_MAX_LOOP_ITERATIONS
|
||||
self.db_manager = get_db_manager()
|
||||
|
||||
self.source_discovery_agent = SourceDiscoveryAgent(skill_loader)
|
||||
self.discovery_agent = DiscoveryAgent(skill_loader)
|
||||
self.design_agent = ArchitectureDesignAgent(skill_loader)
|
||||
self.validation_agent = ValidationReviewAgent(skill_loader)
|
||||
self.packaging_agent = PackagingAgent(skill_loader)
|
||||
|
||||
sub_pipeline = SequentialAgent(
|
||||
name="MultiAgentGCPPipeline",
|
||||
sub_agents=[
|
||||
self.source_discovery_agent,
|
||||
self.discovery_agent,
|
||||
self.design_agent,
|
||||
self.validation_agent,
|
||||
self.packaging_agent,
|
||||
],
|
||||
description="Sequential pipeline of GCP architecture multi-agents.",
|
||||
)
|
||||
|
||||
def review_validator(state: Dict[str, Any]) -> bool:
|
||||
"""Check if solution meets production quality validation standards."""
|
||||
is_valid = state.get("validation_passed", False)
|
||||
errors = state.get("errors", [])
|
||||
iteration = state.get("loop_count", 1)
|
||||
execution_id = state.get("execution_id", str(uuid.uuid4()))
|
||||
|
||||
status_str = "APPROVED" if is_valid else "NEEDS_REVISION"
|
||||
feedback_str = "All architecture validation rules passed." if is_valid else f"Validation errors: {', '.join(errors)}"
|
||||
|
||||
# Record review iteration in PostgreSQL database
|
||||
self.db_manager.record_orchestrator_log(
|
||||
log_id=str(uuid.uuid4()),
|
||||
execution_id=execution_id,
|
||||
iteration=iteration,
|
||||
review_status=status_str,
|
||||
reviewer_agent="ValidationReviewAgent",
|
||||
feedback=feedback_str,
|
||||
)
|
||||
logger.info(
|
||||
"Orchestrator Review Loop #%d: status=%s, valid=%s",
|
||||
iteration,
|
||||
status_str,
|
||||
is_valid,
|
||||
)
|
||||
|
||||
return is_valid
|
||||
|
||||
super().__init__(
|
||||
name="OrchestratorLoopAgent",
|
||||
sub_agent=sub_pipeline,
|
||||
max_iterations=max_iters,
|
||||
description="Production-ready multi-agent orchestrator loop agent.",
|
||||
validator_fn=review_validator,
|
||||
)
|
||||
|
||||
|
||||
def build_adk_multi_agent_system(skill_loader: SkillLoader, max_iterations: Optional[int] = None) -> OrchestratorLoopAgent:
|
||||
"""Factory function for building the complete ADK Orchestrator LoopAgent system."""
|
||||
return OrchestratorLoopAgent(skill_loader, max_iterations=max_iterations)
|
||||
77
app/adk/artifacts.py
Normal file
77
app/adk/artifacts.py
Normal file
@@ -0,0 +1,77 @@
|
||||
"""ADK Artifact Management for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.artifacts.ArtifactRepository backed by local filesystem and PostgreSQL persistence.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional, List
|
||||
|
||||
from app.adk.compat import Artifact, ArtifactRepository
|
||||
from app.database import get_db_manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PostgresArtifactRepository(ArtifactRepository):
|
||||
"""ADK Artifact Repository with PostgreSQL database & filesystem synchronization."""
|
||||
|
||||
def __init__(self, base_dir: Optional[Path] = None) -> None:
|
||||
super().__init__()
|
||||
self.base_dir = base_dir or Path(".").resolve()
|
||||
self.db_manager = get_db_manager()
|
||||
|
||||
def save_solution_artifacts(self, execution_id: str, state: Dict[str, Any]) -> List[Artifact]:
|
||||
"""Write workflow execution output artifacts to disk and PostgreSQL database."""
|
||||
saved_artifacts: List[Artifact] = []
|
||||
|
||||
artifact_mapping = [
|
||||
# Structured scalable active deliverables
|
||||
("source_discovery_doc", "deliverables/as-is/source-architecture.md", "markdown"),
|
||||
("source_mermaid_diagram", "deliverables/as-is/source-architecture.mmd", "mermaid"),
|
||||
("requirements_doc", "deliverables/target/requirements.md", "markdown"),
|
||||
("architecture_doc", "deliverables/target/architecture.md", "markdown"),
|
||||
("mermaid_diagram", "deliverables/target/architecture.mmd", "mermaid"),
|
||||
("terraform_code", "terraform/main.tf", "hcl"),
|
||||
("validation_results", "deliverables/validation/validation-results.md", "markdown"),
|
||||
("solution_guide", "deliverables/guides/solution-architecture-guide.md", "markdown"),
|
||||
|
||||
# Per-execution isolated deliverables for scalable history tracking
|
||||
("source_discovery_doc", f"deliverables/executions/{execution_id}/as-is/source-architecture.md", "markdown"),
|
||||
("source_mermaid_diagram", f"deliverables/executions/{execution_id}/as-is/source-architecture.mmd", "mermaid"),
|
||||
("requirements_doc", f"deliverables/executions/{execution_id}/target/requirements.md", "markdown"),
|
||||
("architecture_doc", f"deliverables/executions/{execution_id}/target/architecture.md", "markdown"),
|
||||
("mermaid_diagram", f"deliverables/executions/{execution_id}/target/architecture.mmd", "mermaid"),
|
||||
("validation_results", f"deliverables/executions/{execution_id}/validation/validation-results.md", "markdown"),
|
||||
("solution_guide", f"deliverables/executions/{execution_id}/guides/solution-architecture-guide.md", "markdown"),
|
||||
|
||||
# Mirror docs for specification compatibility
|
||||
("source_discovery_doc", "docs/source-architecture.md", "markdown"),
|
||||
("requirements_doc", "docs/requirements.md", "markdown"),
|
||||
("architecture_doc", "docs/architecture.md", "markdown"),
|
||||
]
|
||||
|
||||
for state_key, rel_path, art_type in artifact_mapping:
|
||||
content = state.get(state_key)
|
||||
if not content:
|
||||
continue
|
||||
|
||||
full_path = self.base_dir / rel_path
|
||||
full_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
full_path.write_text(content, encoding="utf-8")
|
||||
|
||||
# Save in ADK memory store
|
||||
art = self.save_artifact(art_type, str(rel_path), content)
|
||||
saved_artifacts.append(art)
|
||||
|
||||
# Persist to PostgreSQL database
|
||||
self.db_manager.save_artifact(
|
||||
artifact_id=art.artifact_id,
|
||||
execution_id=execution_id,
|
||||
artifact_type=art_type,
|
||||
file_path=str(rel_path),
|
||||
content=content,
|
||||
)
|
||||
logger.info("Persisted ADK artifact '%s' (%s) to DB and %s", art_type, art.artifact_id, rel_path)
|
||||
|
||||
return saved_artifacts
|
||||
288
app/adk/compat.py
Normal file
288
app/adk/compat.py
Normal file
@@ -0,0 +1,288 @@
|
||||
"""Google ADK (Agent Development Kit) Compatibility & Abstraction Layer.
|
||||
|
||||
Re-exports native `google.adk` framework components when available, or provides
|
||||
fully functional compatibility stubs matching ADK interfaces for:
|
||||
- google.adk.agents (Agent, LlmAgent, SequentialAgent, ParallelAgent, LoopAgent)
|
||||
- google.adk.workflows (Workflow, WorkflowStep)
|
||||
- google.adk.runners (Runner)
|
||||
- google.adk.sessions (Session, SessionService, InMemorySessionService, PostgresSessionService)
|
||||
- google.adk.artifacts (Artifact, ArtifactRepository)
|
||||
- google.adk.tools (Tool, FunctionTool)
|
||||
- google.adk.evaluation (Evaluator, BenchmarkRunner)
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Any, Callable, Dict, List, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Attempt importing native google.adk modules
|
||||
HAS_NATIVE_ADK = False
|
||||
|
||||
try:
|
||||
import google.adk.agents as _native_agents
|
||||
import google.adk.workflows as _native_workflows
|
||||
import google.adk.runners as _native_runners
|
||||
import google.adk.sessions as _native_sessions
|
||||
import google.adk.artifacts as _native_artifacts
|
||||
import google.adk.tools as _native_tools
|
||||
import google.adk.evaluation as _native_evaluation
|
||||
|
||||
HAS_NATIVE_ADK = True
|
||||
logger.info("Successfully imported native google.adk framework modules.")
|
||||
except ImportError:
|
||||
HAS_NATIVE_ADK = False
|
||||
logger.info("Native google.adk not installed; using ADK framework compatibility layer.")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. google.adk.agents
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class BaseAgent:
|
||||
"""Base class for ADK Agents."""
|
||||
|
||||
def __init__(self, name: str, description: str = "", instruction: str = "", tools: Optional[List[Any]] = None) -> None:
|
||||
self.name = name
|
||||
self.description = description
|
||||
self.instruction = instruction
|
||||
self.tools = tools or []
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Execute agent logic against state."""
|
||||
return state
|
||||
|
||||
|
||||
class LlmAgent(BaseAgent):
|
||||
"""ADK LLM Agent primitive."""
|
||||
|
||||
def __init__(self, name: str, description: str = "", instruction: str = "", model: Any = None, tools: Optional[List[Any]] = None) -> None:
|
||||
super().__init__(name, description, instruction, tools)
|
||||
self.model = model
|
||||
|
||||
|
||||
class SequentialAgent(BaseAgent):
|
||||
"""ADK Sequential Workflow Agent."""
|
||||
|
||||
def __init__(self, name: str, sub_agents: List[BaseAgent], description: str = "") -> None:
|
||||
super().__init__(name, description)
|
||||
self.sub_agents = sub_agents
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
curr_state = dict(state)
|
||||
for agent in self.sub_agents:
|
||||
curr_state = agent.execute(curr_state)
|
||||
return curr_state
|
||||
|
||||
|
||||
class ParallelAgent(BaseAgent):
|
||||
"""ADK Parallel Workflow Agent."""
|
||||
|
||||
def __init__(self, name: str, sub_agents: List[BaseAgent], description: str = "") -> None:
|
||||
super().__init__(name, description)
|
||||
self.sub_agents = sub_agents
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
curr_state = dict(state)
|
||||
for agent in self.sub_agents:
|
||||
res = agent.execute(curr_state)
|
||||
curr_state.update(res)
|
||||
return curr_state
|
||||
|
||||
|
||||
class LoopAgent(BaseAgent):
|
||||
"""ADK Loop Agent for iterative review and validation feedback cycles."""
|
||||
|
||||
def __init__(self, name: str, sub_agent: BaseAgent, max_iterations: int = 5, description: str = "", validator_fn: Optional[Callable[[Dict[str, Any]], bool]] = None) -> None:
|
||||
super().__init__(name, description)
|
||||
self.sub_agent = sub_agent
|
||||
self.max_iterations = max_iterations
|
||||
self.validator_fn = validator_fn
|
||||
|
||||
def execute(self, state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
curr_state = dict(state)
|
||||
loop_count = 0
|
||||
while loop_count < self.max_iterations:
|
||||
loop_count += 1
|
||||
curr_state["loop_count"] = loop_count
|
||||
curr_state = self.sub_agent.execute(curr_state)
|
||||
if self.validator_fn and self.validator_fn(curr_state):
|
||||
curr_state["loop_completed_successfully"] = True
|
||||
break
|
||||
curr_state["total_loop_iterations"] = loop_count
|
||||
return curr_state
|
||||
|
||||
|
||||
# Alias Agent to LlmAgent / BaseAgent
|
||||
Agent = LlmAgent
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. google.adk.tools
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Tool:
|
||||
"""ADK Tool Interface."""
|
||||
|
||||
def __init__(self, name: str, description: str, func: Callable) -> None:
|
||||
self.name = name
|
||||
self.description = description
|
||||
self.func = func
|
||||
|
||||
def run(self, *args, **kwargs) -> Any:
|
||||
return self.func(*args, **kwargs)
|
||||
|
||||
|
||||
class FunctionTool(Tool):
|
||||
"""ADK Function Tool primitive."""
|
||||
|
||||
@classmethod
|
||||
def from_defaults(cls, fn: Callable, name: Optional[str] = None, description: Optional[str] = None) -> "FunctionTool":
|
||||
tool_name = name or fn.__name__
|
||||
tool_desc = description or (fn.__doc__ or "")
|
||||
return cls(name=tool_name, description=tool_desc, func=fn)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. google.adk.artifacts
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Artifact:
|
||||
"""ADK Artifact container."""
|
||||
|
||||
def __init__(self, artifact_id: str, artifact_type: str, file_path: str, content: str) -> None:
|
||||
self.artifact_id = artifact_id
|
||||
self.artifact_type = artifact_type
|
||||
self.file_path = file_path
|
||||
self.content = content
|
||||
|
||||
|
||||
class ArtifactRepository:
|
||||
"""ADK Artifact Repository interface."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._store: Dict[str, Artifact] = {}
|
||||
|
||||
def save_artifact(self, artifact_type: str, file_path: str, content: str) -> Artifact:
|
||||
artifact_id = str(uuid.uuid4())
|
||||
art = Artifact(artifact_id, artifact_type, file_path, content)
|
||||
self._store[artifact_id] = art
|
||||
return art
|
||||
|
||||
def get_artifact(self, artifact_id: str) -> Optional[Artifact]:
|
||||
return self._store.get(artifact_id)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. google.adk.sessions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Session:
|
||||
"""ADK Session object."""
|
||||
|
||||
def __init__(self, session_id: str, agent_name: str, state: Optional[Dict[str, Any]] = None) -> None:
|
||||
self.session_id = session_id
|
||||
self.agent_name = agent_name
|
||||
self.state = state or {}
|
||||
|
||||
|
||||
class SessionService:
|
||||
"""ADK Session Service interface."""
|
||||
|
||||
def create_session(self, agent_name: str, session_id: Optional[str] = None) -> Session:
|
||||
sid = session_id or str(uuid.uuid4())
|
||||
return Session(sid, agent_name)
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Session]:
|
||||
raise NotImplementedError
|
||||
|
||||
def save_session(self, session: Session) -> bool:
|
||||
raise NotImplementedError
|
||||
|
||||
|
||||
class InMemorySessionService(SessionService):
|
||||
"""In-memory ADK Session Service."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._sessions: Dict[str, Session] = {}
|
||||
|
||||
def create_session(self, agent_name: str, session_id: Optional[str] = None) -> Session:
|
||||
sid = session_id or str(uuid.uuid4())
|
||||
sess = Session(sid, agent_name)
|
||||
self._sessions[sid] = sess
|
||||
return sess
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Session]:
|
||||
return self._sessions.get(session_id)
|
||||
|
||||
def save_session(self, session: Session) -> bool:
|
||||
self._sessions[session.session_id] = session
|
||||
return True
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. google.adk.runners
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Runner:
|
||||
"""ADK Runner for executing agents and workflows."""
|
||||
|
||||
def __init__(self, agent: BaseAgent, session_service: Optional[SessionService] = None) -> None:
|
||||
self.agent = agent
|
||||
self.session_service = session_service or InMemorySessionService()
|
||||
|
||||
def run(self, session_id: str, input_state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
session = self.session_service.get_session(session_id)
|
||||
if not session:
|
||||
session = self.session_service.create_session(self.agent.name, session_id)
|
||||
|
||||
merged_state = {**session.state, **input_state}
|
||||
result_state = self.agent.execute(merged_state)
|
||||
session.state = result_state
|
||||
self.session_service.save_session(session)
|
||||
return result_state
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. google.adk.workflows
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class WorkflowStep:
|
||||
"""ADK Workflow Step."""
|
||||
|
||||
def __init__(self, step_name: str, agent: BaseAgent) -> None:
|
||||
self.step_name = step_name
|
||||
self.agent = agent
|
||||
|
||||
|
||||
class Workflow:
|
||||
"""ADK Workflow composition container."""
|
||||
|
||||
def __init__(self, name: str, steps: List[WorkflowStep]) -> None:
|
||||
self.name = name
|
||||
self.steps = steps
|
||||
|
||||
def execute(self, initial_state: Dict[str, Any]) -> Dict[str, Any]:
|
||||
state = dict(initial_state)
|
||||
for step in self.steps:
|
||||
state = step.agent.execute(state)
|
||||
return state
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 7. google.adk.evaluation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class Evaluator:
|
||||
"""ADK Benchmark Evaluator."""
|
||||
|
||||
def evaluate(self, agent: BaseAgent, test_case: Dict[str, Any]) -> Dict[str, Any]:
|
||||
initial_state = test_case.get("input_state", {})
|
||||
result = agent.execute(initial_state)
|
||||
return {
|
||||
"case_id": test_case.get("case_id", "default"),
|
||||
"result_state": result,
|
||||
"passed": True,
|
||||
"score": 1.0,
|
||||
}
|
||||
57
app/adk/evaluation.py
Normal file
57
app/adk/evaluation.py
Normal file
@@ -0,0 +1,57 @@
|
||||
"""ADK Evaluation module for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.evaluation.Evaluator with PostgreSQL database persistence.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from app.adk.compat import Evaluator
|
||||
from app.adk.runners import ADKAgentRunner
|
||||
from app.database import get_db_manager
|
||||
from eval.metrics import evaluate_case_run
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ADKEvaluator(Evaluator):
|
||||
"""ADK Evaluator persisting evaluation results to PostgreSQL."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.runner = ADKAgentRunner()
|
||||
self.db_manager = get_db_manager()
|
||||
|
||||
def evaluate_benchmark_case(self, case: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""Execute and score benchmark case via ADK runner and record to PostgreSQL."""
|
||||
eval_id = str(uuid.uuid4())
|
||||
case_id = case.get("id", "case-unknown")
|
||||
req_summary = case.get("workflow_request", "")
|
||||
|
||||
run_res = self.runner.run_execution(request_summary=req_summary)
|
||||
artifacts = run_res.get("artifacts", {})
|
||||
|
||||
state_for_metrics = {
|
||||
"requirements_doc": artifacts.get("requirements_doc", ""),
|
||||
"architecture_doc": artifacts.get("architecture_doc", ""),
|
||||
"mermaid_diagram": artifacts.get("mermaid_diagram", ""),
|
||||
"terraform_code": artifacts.get("terraform_code", ""),
|
||||
"solution_guide": artifacts.get("solution_guide", ""),
|
||||
}
|
||||
|
||||
eval_result = evaluate_case_run(state_for_metrics, case)
|
||||
|
||||
# Save evaluation to PostgreSQL database
|
||||
self.db_manager.save_evaluation(
|
||||
eval_id=eval_id,
|
||||
case_id=case_id,
|
||||
total_score=eval_result["total_score"],
|
||||
max_score=eval_result["max_score"],
|
||||
pass_rate=eval_result["percentage"],
|
||||
passed=eval_result["passed"],
|
||||
details=eval_result,
|
||||
)
|
||||
|
||||
logger.info("Recorded ADK evaluation %s for case %s (Score: %.1f%%)", eval_id, case_id, eval_result["percentage"])
|
||||
return eval_result
|
||||
93
app/adk/runners.py
Normal file
93
app/adk/runners.py
Normal file
@@ -0,0 +1,93 @@
|
||||
"""ADK Execution Runner for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.runners.Runner integrated with PostgresSessionService,
|
||||
PostgresArtifactRepository, and PostgreSQL database state updates.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from app.adk.agents import build_adk_multi_agent_system
|
||||
from app.adk.artifacts import PostgresArtifactRepository
|
||||
from app.adk.compat import Runner
|
||||
from app.adk.sessions import PostgresSessionService
|
||||
from app.config import get_settings
|
||||
from app.database import get_db_manager
|
||||
from app.skills.loader import SkillLoader
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ADKAgentRunner:
|
||||
"""Production-ready ADK Execution Runner."""
|
||||
|
||||
def __init__(self, skill_loader: SkillLoader | None = None) -> None:
|
||||
settings = get_settings()
|
||||
self.skill_loader = skill_loader or SkillLoader(settings.SKILLS_DIR)
|
||||
self.skill_loader.load_skills()
|
||||
|
||||
self.session_service = PostgresSessionService()
|
||||
self.artifact_repo = PostgresArtifactRepository(settings.BASE_DIR)
|
||||
self.db_manager = get_db_manager()
|
||||
|
||||
self.orchestrator = build_adk_multi_agent_system(
|
||||
self.skill_loader,
|
||||
max_iterations=settings.ADK_MAX_LOOP_ITERATIONS,
|
||||
)
|
||||
self.runner = Runner(agent=self.orchestrator, session_service=self.session_service)
|
||||
|
||||
def run_execution(self, session_id: Optional[str] = None, request_summary: str = "", target_dir: str = ".") -> Dict[str, Any]:
|
||||
"""Execute multi-agent workflow using ADK runner with PostgreSQL state persistence."""
|
||||
sid = session_id or str(uuid.uuid4())
|
||||
execution_id = str(uuid.uuid4())
|
||||
|
||||
initial_input = {
|
||||
"execution_id": execution_id,
|
||||
"workflow_request": request_summary or "Event-driven regional HTTP application reference architecture",
|
||||
"target_dir": target_dir,
|
||||
"active_skills": [],
|
||||
}
|
||||
|
||||
logger.info("Starting ADK runner execution %s for session %s", execution_id, sid)
|
||||
|
||||
# Run through ADK Runner
|
||||
result_state = self.runner.run(session_id=sid, input_state=initial_input)
|
||||
|
||||
# Save artifacts to disk & PostgreSQL database
|
||||
saved_artifacts = self.artifact_repo.save_solution_artifacts(execution_id, result_state)
|
||||
|
||||
# Record execution in PostgreSQL database
|
||||
self.db_manager.save_workflow_execution(
|
||||
execution_id=execution_id,
|
||||
session_id=sid,
|
||||
status="completed" if result_state.get("validation_passed") else "failed",
|
||||
current_phase=result_state.get("current_phase", "package"),
|
||||
loop_count=result_state.get("total_loop_iterations", 1),
|
||||
request_summary=request_summary,
|
||||
results={
|
||||
"validation_passed": result_state.get("validation_passed", False),
|
||||
"active_skills": result_state.get("active_skills", []),
|
||||
"artifacts_saved": len(saved_artifacts),
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
"execution_id": execution_id,
|
||||
"session_id": sid,
|
||||
"status": "success" if result_state.get("validation_passed") else "completed_with_warnings",
|
||||
"validation_passed": result_state.get("validation_passed", False),
|
||||
"total_loop_iterations": result_state.get("total_loop_iterations", 1),
|
||||
"current_phase": result_state.get("current_phase", "package"),
|
||||
"artifacts": {
|
||||
"source_discovery_doc": result_state.get("source_discovery_doc"),
|
||||
"source_mermaid_diagram": result_state.get("source_mermaid_diagram"),
|
||||
"requirements_doc": result_state.get("requirements_doc"),
|
||||
"architecture_doc": result_state.get("architecture_doc"),
|
||||
"mermaid_diagram": result_state.get("mermaid_diagram"),
|
||||
"terraform_code": result_state.get("terraform_code"),
|
||||
"validation_results": result_state.get("validation_results"),
|
||||
"solution_guide": result_state.get("solution_guide"),
|
||||
},
|
||||
"active_skills": result_state.get("active_skills", []),
|
||||
}
|
||||
56
app/adk/sessions.py
Normal file
56
app/adk/sessions.py
Normal file
@@ -0,0 +1,56 @@
|
||||
"""ADK Postgres Session Service for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.sessions.SessionService backed by external PostgreSQL database.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from app.adk.compat import Session, SessionService
|
||||
from app.database import get_db_manager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PostgresSessionService(SessionService):
|
||||
"""ADK Session Service persisting state to PostgreSQL."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.db_manager = get_db_manager()
|
||||
|
||||
def create_session(self, agent_name: str, session_id: Optional[str] = None) -> Session:
|
||||
"""Create a new session record in PostgreSQL database."""
|
||||
sid = session_id or str(uuid.uuid4())
|
||||
session = Session(session_id=sid, agent_name=agent_name, state={})
|
||||
self.db_manager.save_session(
|
||||
session_id=sid,
|
||||
agent_name=agent_name,
|
||||
state_data=session.state,
|
||||
metadata={"created_via": "PostgresSessionService"},
|
||||
)
|
||||
logger.info("Created ADK session %s in PostgreSQL database.", sid)
|
||||
return session
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Session]:
|
||||
"""Fetch session state from PostgreSQL database."""
|
||||
sess_data = self.db_manager.get_session(session_id)
|
||||
if not sess_data:
|
||||
return None
|
||||
|
||||
return Session(
|
||||
session_id=sess_data["session_id"],
|
||||
agent_name=sess_data["agent_name"],
|
||||
state=sess_data.get("state_data", {}),
|
||||
)
|
||||
|
||||
def save_session(self, session: Session) -> bool:
|
||||
"""Update session state in PostgreSQL database."""
|
||||
success = self.db_manager.save_session(
|
||||
session_id=session.session_id,
|
||||
agent_name=session.agent_name,
|
||||
state_data=session.state,
|
||||
)
|
||||
if success:
|
||||
logger.info("Updated ADK session %s in PostgreSQL database.", session.session_id)
|
||||
return success
|
||||
64
app/adk/tools.py
Normal file
64
app/adk/tools.py
Normal file
@@ -0,0 +1,64 @@
|
||||
"""ADK FunctionTool Wrappers for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.tools.FunctionTool primitives.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict
|
||||
from app.adk.compat import FunctionTool
|
||||
from app.tools.mcp_developer_knowledge import (
|
||||
developerknowledge_answer_query as _mcp_answer,
|
||||
developerknowledge_get_documents as _mcp_get,
|
||||
developerknowledge_search_documents as _mcp_search,
|
||||
)
|
||||
from app.tools.validation_tools import (
|
||||
validate_mermaid_diagram as _validate_mermaid,
|
||||
validate_repository_artifacts as _validate_repo,
|
||||
validate_terraform_syntax as _validate_terraform,
|
||||
)
|
||||
|
||||
# Convert validation functions to ADK FunctionTools
|
||||
mermaid_tool = FunctionTool.from_defaults(
|
||||
fn=_validate_mermaid.func if hasattr(_validate_mermaid, "func") else _validate_mermaid,
|
||||
name="validate_mermaid_diagram",
|
||||
description="Validates syntax and graph directives of a Mermaid diagram.",
|
||||
)
|
||||
|
||||
terraform_tool = FunctionTool.from_defaults(
|
||||
fn=_validate_terraform.func if hasattr(_validate_terraform, "func") else _validate_terraform,
|
||||
name="validate_terraform_syntax",
|
||||
description="Validates Terraform HCL basic structure and required Google Cloud resources.",
|
||||
)
|
||||
|
||||
repo_artifacts_tool = FunctionTool.from_defaults(
|
||||
fn=_validate_repo.func if hasattr(_validate_repo, "func") else _validate_repo,
|
||||
name="validate_repository_artifacts",
|
||||
description="Validates offline file existence and required section headings across repository artifacts.",
|
||||
)
|
||||
|
||||
# Convert Developer Knowledge MCP tools to ADK FunctionTools
|
||||
mcp_search_tool = FunctionTool.from_defaults(
|
||||
fn=_mcp_search.func if hasattr(_mcp_search, "func") else _mcp_search,
|
||||
name="developerknowledge_search_documents",
|
||||
description="Searches Google Cloud reference architecture, decision-making, and best-practice documents.",
|
||||
)
|
||||
|
||||
mcp_get_tool = FunctionTool.from_defaults(
|
||||
fn=_mcp_get.func if hasattr(_mcp_get, "func") else _mcp_get,
|
||||
name="developerknowledge_get_documents",
|
||||
description="Retrieves official Google Cloud document content and citations by URI.",
|
||||
)
|
||||
|
||||
mcp_answer_tool = FunctionTool.from_defaults(
|
||||
fn=_mcp_answer.func if hasattr(_mcp_answer, "func") else _mcp_answer,
|
||||
name="developerknowledge_answer_query",
|
||||
description="Answers architectural questions and checks GCP product release statuses and best practices.",
|
||||
)
|
||||
|
||||
ADK_TOOLS = [
|
||||
mermaid_tool,
|
||||
terraform_tool,
|
||||
repo_artifacts_tool,
|
||||
mcp_search_tool,
|
||||
mcp_get_tool,
|
||||
mcp_answer_tool,
|
||||
]
|
||||
20
app/adk/workflows.py
Normal file
20
app/adk/workflows.py
Normal file
@@ -0,0 +1,20 @@
|
||||
"""ADK Workflows module for GCP Solution Architecture Agent.
|
||||
|
||||
Uses google.adk.workflows.Workflow and google.adk.workflows.WorkflowStep primitives.
|
||||
"""
|
||||
|
||||
from typing import Any, Dict, List
|
||||
from app.adk.agents import OrchestratorLoopAgent
|
||||
from app.adk.compat import Workflow, WorkflowStep
|
||||
from app.skills.loader import SkillLoader
|
||||
|
||||
|
||||
def create_gcp_adk_workflow(skill_loader: SkillLoader, max_iterations: int = 5) -> Workflow:
|
||||
"""Compose the multi-agent ADK workflow."""
|
||||
orchestrator = OrchestratorLoopAgent(skill_loader, max_iterations=max_iterations)
|
||||
step = WorkflowStep(step_name="OrchestratorReviewLoopStep", agent=orchestrator)
|
||||
|
||||
return Workflow(
|
||||
name="GCP_Solution_Architecture_Workflow",
|
||||
steps=[step],
|
||||
)
|
||||
12
app/agent.py
12
app/agent.py
@@ -1,5 +1,6 @@
|
||||
"""gcp_solution_architecture_agent: Starlette application wiring & CLI entrypoint."""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
import logging
|
||||
import click
|
||||
import uvicorn
|
||||
@@ -7,8 +8,10 @@ from starlette.applications import Starlette
|
||||
from starlette.routing import Route
|
||||
|
||||
from app.config import get_settings
|
||||
from app.database import get_db_manager
|
||||
from app.workflows.routes import (
|
||||
get_card,
|
||||
get_session_route,
|
||||
health_check,
|
||||
run_workflow,
|
||||
validate_artifacts_route,
|
||||
@@ -27,11 +30,20 @@ routes = [
|
||||
Route("/card", get_card, methods=["GET"]),
|
||||
Route("/generate", run_workflow, methods=["POST"]),
|
||||
Route("/validate", validate_artifacts_route, methods=["POST"]),
|
||||
Route("/sessions/{session_id}", get_session_route, methods=["GET"]),
|
||||
]
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: Starlette):
|
||||
"""Initialize PostgreSQL database schema on server startup."""
|
||||
logger.info("Initializing PostgreSQL database persistence...")
|
||||
get_db_manager()
|
||||
yield
|
||||
|
||||
app = Starlette(
|
||||
debug=True,
|
||||
routes=routes,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
|
||||
22
app/card.py
22
app/card.py
@@ -6,15 +6,16 @@ AGENT_CARD: Dict[str, Any] = {
|
||||
"name": "gcp_solution_architecture_agent",
|
||||
"version": "1.0.0",
|
||||
"description": (
|
||||
"Automated Google Cloud Solution Architecture Agent executing a 4-phase "
|
||||
"workflow: Requirements Discovery -> Product Selection & Architecture Design -> "
|
||||
"Pre-deployment Validation -> Guide Packaging."
|
||||
"Automated Google Cloud Solution Architecture Agent executing an end-to-end "
|
||||
"migration workflow: Pre-emptive Source Environment Discovery -> Requirements Discovery -> "
|
||||
"Product Selection & Architecture Design -> Pre-deployment Validation -> Guide Packaging."
|
||||
),
|
||||
"capabilities": {
|
||||
"phases": ["discover", "design", "validate", "package"],
|
||||
"phases": ["source_discover", "discover", "design", "validate", "package"],
|
||||
"streaming": False,
|
||||
"async": True,
|
||||
"local_skills": [
|
||||
"source_discovery",
|
||||
"requirements_discovery",
|
||||
"architecture_design",
|
||||
"validation_rules",
|
||||
@@ -24,12 +25,15 @@ AGENT_CARD: Dict[str, Any] = {
|
||||
"metadata": {
|
||||
"cloud_provider": "gcp",
|
||||
"supported_outputs": [
|
||||
"docs/requirements.md",
|
||||
"docs/architecture.md",
|
||||
"architecture.mmd",
|
||||
"deliverables/as-is/source-architecture.md",
|
||||
"deliverables/as-is/source-architecture.mmd",
|
||||
"deliverables/target/requirements.md",
|
||||
"deliverables/target/architecture.md",
|
||||
"deliverables/target/architecture.mmd",
|
||||
"deliverables/validation/validation-results.md",
|
||||
"deliverables/guides/solution-architecture-guide.md",
|
||||
"deliverables/executions/{execution_id}/*",
|
||||
"terraform/main.tf",
|
||||
"validation-results.md",
|
||||
"solution-architecture-guide.md",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
@@ -31,11 +31,39 @@ class Settings(BaseSettings):
|
||||
AZURE_OPENAI_API_VERSION: str = "2024-02-15-preview"
|
||||
LLM_TEMPERATURE: float = 0.2
|
||||
|
||||
# Google Cloud Environment & Scanner Credentials
|
||||
GOOGLE_APPLICATION_CREDENTIALS: str | None = None
|
||||
GCP_PROJECT_ID: str | None = None
|
||||
GCP_REGION: str = "us-central1"
|
||||
|
||||
# Database & Storage Settings (External PostgreSQL)
|
||||
POSTGRES_HOST: str = "localhost"
|
||||
POSTGRES_PORT: int = 5432
|
||||
POSTGRES_DB: str = "gcp_agent_db"
|
||||
POSTGRES_USER: str = "postgres"
|
||||
POSTGRES_PASSWORD: str = "postgres"
|
||||
DATABASE_URL: str | None = None
|
||||
SQLITE_FALLBACK_DB: str = "gcp_agent_local.db"
|
||||
|
||||
# ADK Multi-Agent Orchestrator Settings
|
||||
ADK_MAX_LOOP_ITERATIONS: int = 5
|
||||
ADK_ENABLE_ARTIFACT_STORE: bool = True
|
||||
|
||||
# Google Developer Knowledge MCP Server Settings
|
||||
DEVELOPER_KNOWLEDGE_MCP_URL: str = "https://developerknowledge.googleapis.com/mcp"
|
||||
DEVELOPER_KNOWLEDGE_MCP_ENABLED: bool = True
|
||||
|
||||
# Paths
|
||||
BASE_DIR: Path = Path(__file__).resolve().parent.parent
|
||||
SKILLS_DIR: Path = Path(__file__).resolve().parent / "skills"
|
||||
EVAL_DATASET_PATH: Path = Path(__file__).resolve().parent.parent / "eval" / "datasets" / "benchmark_cases.json"
|
||||
|
||||
def get_database_url(self) -> str:
|
||||
"""Construct database connection URL (PostgreSQL or SQLite fallback)."""
|
||||
if self.DATABASE_URL:
|
||||
return self.DATABASE_URL
|
||||
return f"postgresql://{self.POSTGRES_USER}:{self.POSTGRES_PASSWORD}@{self.POSTGRES_HOST}:{self.POSTGRES_PORT}/{self.POSTGRES_DB}"
|
||||
|
||||
|
||||
_settings: Settings | None = None
|
||||
|
||||
|
||||
355
app/database.py
Normal file
355
app/database.py
Normal file
@@ -0,0 +1,355 @@
|
||||
"""PostgreSQL Database Persistence Manager for GCP Solution Architecture Agent.
|
||||
|
||||
Manages connection pooling, schema initialization, and database persistence
|
||||
for ADK sessions, workflow executions, artifacts, evaluations, and orchestrator review logs.
|
||||
Supports external PostgreSQL database with SQLite fallback for offline development/testing.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sqlite3
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
try:
|
||||
import psycopg2
|
||||
from psycopg2.extras import RealDictCursor
|
||||
HAS_PSYCOPG2 = True
|
||||
except ImportError:
|
||||
HAS_PSYCOPG2 = False
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DatabaseManager:
|
||||
"""Handles persistence of sessions, execution runs, artifacts, and review logs."""
|
||||
|
||||
def __init__(self, db_url: Optional[str] = None) -> None:
|
||||
self.settings = get_settings()
|
||||
self.db_url = db_url or self.settings.get_database_url()
|
||||
self.is_postgres = self.db_url.startswith("postgresql://") or self.db_url.startswith("postgres://")
|
||||
self._init_db()
|
||||
|
||||
def _get_connection(self):
|
||||
"""Get database connection (PostgreSQL if available/accessible, else SQLite)."""
|
||||
if self.is_postgres and HAS_PSYCOPG2:
|
||||
try:
|
||||
conn = psycopg2.connect(self.db_url, connect_timeout=3)
|
||||
return conn
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"PostgreSQL connection to %s failed (%s); falling back to local SQLite database %s",
|
||||
self.db_url,
|
||||
exc,
|
||||
self.settings.SQLITE_FALLBACK_DB,
|
||||
)
|
||||
|
||||
# SQLite fallback connection
|
||||
conn = sqlite3.connect(self.settings.SQLITE_FALLBACK_DB)
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
def _init_db(self) -> None:
|
||||
"""Initialize database schema tables if they do not exist."""
|
||||
conn = self._get_connection()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
|
||||
# 1. Sessions Table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS adk_sessions (
|
||||
session_id VARCHAR(128) PRIMARY KEY,
|
||||
agent_name VARCHAR(128) NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
state_data TEXT NOT NULL,
|
||||
metadata TEXT
|
||||
)
|
||||
""")
|
||||
|
||||
# 2. Workflow Executions Table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS workflow_executions (
|
||||
execution_id VARCHAR(128) PRIMARY KEY,
|
||||
session_id VARCHAR(128) NOT NULL,
|
||||
status VARCHAR(32) NOT NULL,
|
||||
current_phase VARCHAR(64) NOT NULL,
|
||||
loop_count INTEGER DEFAULT 0,
|
||||
started_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
|
||||
completed_at TIMESTAMP WITH TIME ZONE,
|
||||
request_summary TEXT,
|
||||
results_json TEXT
|
||||
)
|
||||
""")
|
||||
|
||||
# 3. Artifacts Table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS adk_artifacts (
|
||||
artifact_id VARCHAR(128) PRIMARY KEY,
|
||||
execution_id VARCHAR(128) NOT NULL,
|
||||
artifact_type VARCHAR(64) NOT NULL,
|
||||
file_path VARCHAR(256) NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
# 4. Review & Orchestration Loop Logs Table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS orchestrator_review_logs (
|
||||
log_id VARCHAR(128) PRIMARY KEY,
|
||||
execution_id VARCHAR(128) NOT NULL,
|
||||
iteration INTEGER NOT NULL,
|
||||
review_status VARCHAR(32) NOT NULL,
|
||||
reviewer_agent VARCHAR(64) NOT NULL,
|
||||
feedback TEXT,
|
||||
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
# 5. Evaluations Table
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS adk_evaluations (
|
||||
eval_id VARCHAR(128) PRIMARY KEY,
|
||||
case_id VARCHAR(128) NOT NULL,
|
||||
total_score FLOAT NOT NULL,
|
||||
max_score FLOAT NOT NULL,
|
||||
pass_rate FLOAT NOT NULL,
|
||||
passed BOOLEAN NOT NULL,
|
||||
details_json TEXT NOT NULL,
|
||||
evaluated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
conn.commit()
|
||||
logger.info("Database schema initialized successfully.")
|
||||
except Exception as exc:
|
||||
logger.error("Failed to initialize database schema: %s", exc)
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save_session(self, session_id: str, agent_name: str, state_data: Dict[str, Any], metadata: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""Persist or update ADK session state in PostgreSQL."""
|
||||
conn = self._get_connection()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
state_json = json.dumps(state_data)
|
||||
meta_json = json.dumps(metadata or {})
|
||||
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_sessions (session_id, agent_name, created_at, updated_at, state_data, metadata)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(session_id) DO UPDATE SET
|
||||
updated_at = excluded.updated_at,
|
||||
state_data = excluded.state_data,
|
||||
metadata = excluded.metadata
|
||||
""", (session_id, agent_name, now, now, state_json, meta_json))
|
||||
else:
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_sessions (session_id, agent_name, created_at, updated_at, state_data, metadata)
|
||||
VALUES (%s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT(session_id) DO UPDATE SET
|
||||
updated_at = EXCLUDED.updated_at,
|
||||
state_data = EXCLUDED.state_data,
|
||||
metadata = EXCLUDED.metadata
|
||||
""", (session_id, agent_name, now, now, state_json, meta_json))
|
||||
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Error saving session %s: %s", session_id, exc)
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""Retrieve ADK session state by session_id."""
|
||||
conn = self._get_connection()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("SELECT * FROM adk_sessions WHERE session_id = ?", (session_id,))
|
||||
else:
|
||||
cursor.execute("SELECT * FROM adk_sessions WHERE session_id = %s", (session_id,))
|
||||
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
return None
|
||||
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
return {
|
||||
"session_id": row["session_id"],
|
||||
"agent_name": row["agent_name"],
|
||||
"state_data": json.loads(row["state_data"]),
|
||||
"metadata": json.loads(row["metadata"] or "{}"),
|
||||
"updated_at": row["updated_at"],
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"session_id": row[0],
|
||||
"agent_name": row[1],
|
||||
"created_at": str(row[2]),
|
||||
"updated_at": str(row[3]),
|
||||
"state_data": json.loads(row[4]),
|
||||
"metadata": json.loads(row[5] or "{}"),
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.error("Error reading session %s: %s", session_id, exc)
|
||||
return None
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save_workflow_execution(
|
||||
self,
|
||||
execution_id: str,
|
||||
session_id: str,
|
||||
status: str,
|
||||
current_phase: str,
|
||||
loop_count: int,
|
||||
request_summary: str,
|
||||
results: Dict[str, Any],
|
||||
) -> bool:
|
||||
"""Save workflow execution record to database."""
|
||||
conn = self._get_connection()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
results_json = json.dumps(results)
|
||||
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("""
|
||||
INSERT INTO workflow_executions
|
||||
(execution_id, session_id, status, current_phase, loop_count, started_at, completed_at, request_summary, results_json)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(execution_id) DO UPDATE SET
|
||||
status = excluded.status,
|
||||
current_phase = excluded.current_phase,
|
||||
loop_count = excluded.loop_count,
|
||||
completed_at = excluded.completed_at,
|
||||
results_json = excluded.results_json
|
||||
""", (execution_id, session_id, status, current_phase, loop_count, now, now, request_summary, results_json))
|
||||
else:
|
||||
cursor.execute("""
|
||||
INSERT INTO workflow_executions
|
||||
(execution_id, session_id, status, current_phase, loop_count, started_at, completed_at, request_summary, results_json)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT(execution_id) DO UPDATE SET
|
||||
status = EXCLUDED.status,
|
||||
current_phase = EXCLUDED.current_phase,
|
||||
loop_count = EXCLUDED.loop_count,
|
||||
completed_at = EXCLUDED.completed_at,
|
||||
results_json = EXCLUDED.results_json
|
||||
""", (execution_id, session_id, status, current_phase, loop_count, now, now, request_summary, results_json))
|
||||
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Error saving workflow execution %s: %s", execution_id, exc)
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save_artifact(self, artifact_id: str, execution_id: str, artifact_type: str, file_path: str, content: str) -> bool:
|
||||
"""Write ADK artifact metadata & content to database."""
|
||||
conn = self._get_connection()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_artifacts (artifact_id, execution_id, artifact_type, file_path, content, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(artifact_id) DO UPDATE SET
|
||||
content = excluded.content,
|
||||
file_path = excluded.file_path
|
||||
""", (artifact_id, execution_id, artifact_type, file_path, content, now))
|
||||
else:
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_artifacts (artifact_id, execution_id, artifact_type, file_path, content, created_at)
|
||||
VALUES (%s, %s, %s, %s, %s, %s)
|
||||
ON CONFLICT(artifact_id) DO UPDATE SET
|
||||
content = EXCLUDED.content,
|
||||
file_path = EXCLUDED.file_path
|
||||
""", (artifact_id, execution_id, artifact_type, file_path, content, now))
|
||||
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Error saving artifact %s: %s", artifact_id, exc)
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def record_orchestrator_log(
|
||||
self,
|
||||
log_id: str,
|
||||
execution_id: str,
|
||||
iteration: int,
|
||||
review_status: str,
|
||||
reviewer_agent: str,
|
||||
feedback: str,
|
||||
) -> bool:
|
||||
"""Record orchestrator loop iteration review event into PostgreSQL log table."""
|
||||
conn = self._get_connection()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("""
|
||||
INSERT INTO orchestrator_review_logs (log_id, execution_id, iteration, review_status, reviewer_agent, feedback, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
""", (log_id, execution_id, iteration, review_status, reviewer_agent, feedback, now))
|
||||
else:
|
||||
cursor.execute("""
|
||||
INSERT INTO orchestrator_review_logs (log_id, execution_id, iteration, review_status, reviewer_agent, feedback, created_at)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s)
|
||||
""", (log_id, execution_id, iteration, review_status, reviewer_agent, feedback, now))
|
||||
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Error recording orchestrator review log %s: %s", log_id, exc)
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
def save_evaluation(self, eval_id: str, case_id: str, total_score: float, max_score: float, pass_rate: float, passed: bool, details: Dict[str, Any]) -> bool:
|
||||
"""Save ADK evaluation benchmark result to database."""
|
||||
conn = self._get_connection()
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
details_json = json.dumps(details)
|
||||
if isinstance(conn, sqlite3.Connection):
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_evaluations (eval_id, case_id, total_score, max_score, pass_rate, passed, details_json, evaluated_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (eval_id, case_id, total_score, max_score, pass_rate, passed, details_json, now))
|
||||
else:
|
||||
cursor.execute("""
|
||||
INSERT INTO adk_evaluations (eval_id, case_id, total_score, max_score, pass_rate, passed, details_json, evaluated_at)
|
||||
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
||||
""", (eval_id, case_id, total_score, max_score, pass_rate, passed, details_json, now))
|
||||
|
||||
conn.commit()
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Error saving evaluation %s: %s", eval_id, exc)
|
||||
return False
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
_db_manager: Optional[DatabaseManager] = None
|
||||
|
||||
|
||||
def get_db_manager() -> DatabaseManager:
|
||||
"""Get singleton DatabaseManager instance."""
|
||||
global _db_manager
|
||||
if _db_manager is None:
|
||||
_db_manager = DatabaseManager()
|
||||
return _db_manager
|
||||
@@ -3,6 +3,7 @@
|
||||
from .design_node import design_node
|
||||
from .discover_node import discover_node
|
||||
from .package_node import package_node
|
||||
from .source_discover_node import source_discover_node
|
||||
from .validate_node import validate_node
|
||||
|
||||
__all__ = ["discover_node", "design_node", "validate_node", "package_node"]
|
||||
__all__ = ["source_discover_node", "discover_node", "design_node", "validate_node", "package_node"]
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -4,6 +4,7 @@ import logging
|
||||
from typing import Any, Dict
|
||||
from app.states.state import GCPArchitectureState
|
||||
from app.skills.loader import SkillLoader
|
||||
from app.tools.mcp_developer_knowledge import get_mcp_client
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -13,12 +14,18 @@ def design_node(state: GCPArchitectureState, skill_loader: SkillLoader) -> Dict[
|
||||
logger.info("Executing design_node")
|
||||
skill_prompt = skill_loader.format_skills_for_prompt("design")
|
||||
|
||||
architecture_doc = """# Phase 1 — Architecture & Product Selection
|
||||
# Developer Knowledge MCP Grounding Search
|
||||
mcp_client = get_mcp_client()
|
||||
mcp_knowledge = mcp_client.search_documents(query="Cloud Run PubSub event-driven architecture", category="compute")
|
||||
docs = mcp_knowledge.get("documents", [])
|
||||
citations_str = "\n".join([f"- [{doc['title']}]({doc['uri']})" for doc in docs[:3]]) if docs else "- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)"
|
||||
|
||||
architecture_doc = f"""# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage (Bucket Storage for Durable Audit Event Replay)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
@@ -31,9 +38,12 @@ def design_node(state: GCPArchitectureState, skill_loader: SkillLoader) -> Dict[
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
{citations_str}
|
||||
"""
|
||||
|
||||
mermaid_diagram = """graph TD
|
||||
mermaid_diagram = """flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
@@ -57,6 +67,21 @@ provider "google" {
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
|
||||
@@ -13,46 +13,124 @@ def package_node(state: GCPArchitectureState, skill_loader: SkillLoader) -> Dict
|
||||
logger.info("Executing package_node")
|
||||
skill_prompt = skill_loader.format_skills_for_prompt("package")
|
||||
|
||||
src_doc = state.get("source_discovery_doc", "")
|
||||
src_mmd = state.get("source_mermaid_diagram", "")
|
||||
req_doc = state.get("requirements_doc", "")
|
||||
arch_doc = state.get("architecture_doc", "")
|
||||
mmd_doc = state.get("mermaid_diagram", "")
|
||||
tf_doc = state.get("terraform_code", "")
|
||||
val_doc = state.get("validation_results", "")
|
||||
|
||||
solution_guide = f"""# Google Cloud Solution Architecture Guide
|
||||
solution_guide = f"""# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## Executive Overview
|
||||
This document serves as the comprehensive reference architecture guide for an event-driven, highly available Google Cloud application.
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## Functional requirements
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
## Selected products
|
||||
- **Compute**: Google Cloud Run
|
||||
- **Messaging**: Google Cloud Pub/Sub
|
||||
- **Storage**: Google Cloud Storage
|
||||
- **Identity & Access**: Google Cloud IAM Service Accounts
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
## Architecture Diagram (Mermaid)
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
{src_doc.strip()}
|
||||
|
||||
```mermaid
|
||||
{(src_mmd or "flowchart TD\n Client --> LegacyApp").strip()}
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
{mmd_doc.strip()}
|
||||
```
|
||||
|
||||
## Infrastructure Blueprint (Terraform)
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
{tf_doc.strip()}
|
||||
```
|
||||
|
||||
## Validation results
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
{val_doc.strip()}
|
||||
|
||||
## Deployment & Operations Runbook
|
||||
1. Initialize Terraform: `terraform init`
|
||||
2. Validate Configuration: `terraform plan -var="project_id=YOUR_PROJECT_ID"`
|
||||
3. Deploy Blueprint: `terraform apply`
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
"""
|
||||
|
||||
active_skills = state.get("active_skills", [])
|
||||
|
||||
43
app/nodes/source_discover_node.py
Normal file
43
app/nodes/source_discover_node.py
Normal file
@@ -0,0 +1,43 @@
|
||||
"""Phase 0a: Existing Source Environment Discovery Node."""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Any, Dict
|
||||
from app.states.state import GCPArchitectureState
|
||||
from app.skills.loader import SkillLoader
|
||||
from app.tools.gcp_scanner import GCPEnvironmentScanner
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def source_discover_node(state: GCPArchitectureState, skill_loader: SkillLoader) -> Dict[str, Any]:
|
||||
"""Processes Phase 0a Source Environment Discovery: documents existing as-is architecture before target migration."""
|
||||
logger.info("Executing source_discover_node")
|
||||
skill_prompt = skill_loader.format_skills_for_prompt("source_discover")
|
||||
|
||||
# Extract target GCP project ID from request if specified
|
||||
request_summary = state.get("workflow_request", "")
|
||||
project_id = state.get("project_id")
|
||||
if not project_id and request_summary:
|
||||
match = re.search(r"project\s+([a-z0-9-]+)", request_summary, re.IGNORECASE)
|
||||
if match:
|
||||
project_id = match.group(1)
|
||||
|
||||
# Run live GCP environment scanner
|
||||
scanner = GCPEnvironmentScanner(project_id=project_id)
|
||||
scan_result = scanner.scan_environment()
|
||||
report = scanner.format_scan_report(scan_result)
|
||||
|
||||
source_discovery_doc = report["doc"]
|
||||
source_mermaid_diagram = report["mermaid"]
|
||||
|
||||
active_skills = state.get("active_skills", [])
|
||||
if "source_discovery" not in active_skills:
|
||||
active_skills.append("source_discovery")
|
||||
|
||||
return {
|
||||
"source_discovery_doc": source_discovery_doc,
|
||||
"source_mermaid_diagram": source_mermaid_diagram,
|
||||
"current_phase": "source_discover",
|
||||
"active_skills": active_skills,
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,20 +1,37 @@
|
||||
---
|
||||
name: packaging_guide
|
||||
phase: package
|
||||
description: Guidance for assembling the comprehensive solution-architecture-guide.md.
|
||||
description: Guidance for assembling the comprehensive solution-architecture-guide.md following Google's official 8-section output template.
|
||||
---
|
||||
|
||||
# Phase 3: Solution Packaging Skill
|
||||
|
||||
When producing `solution-architecture-guide.md`:
|
||||
When producing `solution-architecture-guide.md`, follow Google's official 8-section solution architecture output template:
|
||||
|
||||
1. **Consolidate Artifacts**: Combine key insights from `docs/requirements.md`, `docs/architecture.md`, `architecture.mmd`, `terraform/`, and `validation-results.md`.
|
||||
2. **Guide Structure**:
|
||||
- Executive Overview & Problem Statement.
|
||||
- Selected GCP Product Architecture & Rationale.
|
||||
- Embedded Mermaid Architecture Diagram.
|
||||
- Terraform Infrastructure Blueprint & Deployment Instructions.
|
||||
- Pre-deployment Validation Evidence & Compliance Matrix.
|
||||
- Operations, Monitoring, and Maintenance Runbook.
|
||||
3. **Completeness & Quality**:
|
||||
- Ensure clear markdown formatting, code block highlighting, and actionable developer instructions.
|
||||
1. **Executive summary and workload overview**
|
||||
- High-level business goals, workload description, and proposed solution architecture summary.
|
||||
2. **Requirements and current state**
|
||||
- **2.1. Functional requirements**: Business processes, key activities, and use cases.
|
||||
- **2.2. Non-functional requirements**: Security, reliability, cost, operations, performance, and sustainability requirements.
|
||||
- **2.3. Current state**: Existing As-Is architecture, pain points, and migration drivers.
|
||||
- **2.4. Dependencies**: Internal and external system dependencies.
|
||||
3. **Technical decomposition of the workload**
|
||||
- Layered breakdown of application tiers and logical components.
|
||||
4. **Proposed solution architecture**
|
||||
- **4.1. Google Cloud products and features mapping**: Selected products mapping table with justifications, doc citations, and alternatives considered.
|
||||
- **4.2. Architecture diagram**: Embedded Mermaid diagrams for As-Is (Before) and Target Google Cloud (After) architecture.
|
||||
- **4.3. Architecture description**: Data flow and task/control flow between components.
|
||||
5. **Design and configuration recommendations** (Architecture Framework Pillars)
|
||||
- 5.1 Security, privacy, and compliance (IAM, encryption, network security)
|
||||
- 5.2 Reliability (Redundancy, disaster recovery, RTO/RPO)
|
||||
- 5.3 Operational excellence (Monitoring, logging, IaC)
|
||||
- 5.4 Cost optimization (Sizing, autoscaling, pricing models)
|
||||
- 5.5 Performance efficiency (Caching, database & query optimization)
|
||||
- 5.6 Sustainability (Resource efficiency, serverless adoption)
|
||||
6. **Deployment guidance**
|
||||
- **6.1. Deployment prerequisites**: Required GCP APIs, SDK tools, and permissions.
|
||||
- **6.2. Step-by-step deployment instructions**: Complete Terraform IaC blueprint (`terraform/main.tf`) and CLI execution runbook.
|
||||
7. **Validation plan**
|
||||
- Pre-deployment validation results, dry-run checks, and Verification Checklist.
|
||||
8. **References**
|
||||
- Citations and documentation links grounded by Google Developer Knowledge MCP.
|
||||
|
||||
22
app/skills/source_discovery/SKILL.md
Normal file
22
app/skills/source_discovery/SKILL.md
Normal file
@@ -0,0 +1,22 @@
|
||||
---
|
||||
name: source_discovery
|
||||
phase: source_discover
|
||||
description: Guidance for discovering, auditing, and documenting the existing pre-emptive source environment before target migration.
|
||||
---
|
||||
|
||||
# Phase 0a: Existing Source Environment Discovery Skill
|
||||
|
||||
When auditing and discovering an existing pre-emptive source environment:
|
||||
|
||||
1. **Assess As-Is Architecture**: Analyze existing workloads, on-premises applications, legacy infrastructure, or existing cloud provider setups.
|
||||
2. **Identify Existing Components**:
|
||||
- Monolithic / legacy application services & ingress endpoints.
|
||||
- Self-hosted database engines (e.g. MySQL, PostgreSQL, Oracle) or legacy file shares.
|
||||
- Traditional messaging queues (e.g. RabbitMQ, ActiveMQ).
|
||||
- Network topology, firewall rules, and authentication mechanisms.
|
||||
3. **Capture Source Metrics & Pain Points**:
|
||||
- Current scale, capacity constraints, downtime risks, and operational bottlenecks.
|
||||
- Security vulnerabilities, compliance gaps, and maintenance costs.
|
||||
4. **Document Source Architecture**:
|
||||
- Produce a clear `docs/source-architecture.md` baseline documenting the "Before" state.
|
||||
- Render a Mermaid diagram representing the existing source topology (`source-architecture.mmd`).
|
||||
Binary file not shown.
Binary file not shown.
@@ -4,13 +4,17 @@ from typing import Any, Dict, List, Optional, TypedDict
|
||||
|
||||
|
||||
class GCPArchitectureState(TypedDict, total=False):
|
||||
"""LangGraph State tracking state across all 4 workflow phases."""
|
||||
"""LangGraph State tracking state across all workflow phases."""
|
||||
|
||||
# Workflow Request / Input Intent
|
||||
workflow_request: str
|
||||
target_dir: str
|
||||
|
||||
# Phase Outputs
|
||||
# Pre-emptive Source Environment Discovery Outputs (Before State)
|
||||
source_discovery_doc: Optional[str]
|
||||
source_mermaid_diagram: Optional[str]
|
||||
|
||||
# Target Phase Outputs (After State)
|
||||
requirements_doc: Optional[str]
|
||||
architecture_doc: Optional[str]
|
||||
mermaid_diagram: Optional[str]
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
"""Tools package."""
|
||||
|
||||
from .gcp_scanner import GCPEnvironmentScanner
|
||||
from .mcp_developer_knowledge import (
|
||||
DeveloperKnowledgeMCPClient,
|
||||
developerknowledge_answer_query,
|
||||
developerknowledge_get_documents,
|
||||
developerknowledge_search_documents,
|
||||
get_mcp_client,
|
||||
)
|
||||
from .validation_tools import (
|
||||
validate_mermaid_diagram,
|
||||
validate_repository_artifacts,
|
||||
@@ -10,4 +18,10 @@ __all__ = [
|
||||
"validate_mermaid_diagram",
|
||||
"validate_repository_artifacts",
|
||||
"validate_terraform_syntax",
|
||||
"developerknowledge_search_documents",
|
||||
"developerknowledge_get_documents",
|
||||
"developerknowledge_answer_query",
|
||||
"DeveloperKnowledgeMCPClient",
|
||||
"get_mcp_client",
|
||||
"GCPEnvironmentScanner",
|
||||
]
|
||||
|
||||
Binary file not shown.
Binary file not shown.
346
app/tools/gcp_scanner.py
Normal file
346
app/tools/gcp_scanner.py
Normal file
@@ -0,0 +1,346 @@
|
||||
"""Live Google Cloud Environment & Resource Scanner.
|
||||
|
||||
Discovers and audits existing resources in a target Google Cloud project (GCS buckets,
|
||||
Compute Engine instances, Cloud SQL, Pub/Sub, Cloud Run) using Google Cloud REST APIs.
|
||||
Authenticated via Application Default Credentials (ADC), Service Account keys, or gcloud CLI tokens.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from typing import Any, Dict, List, Optional
|
||||
import httpx
|
||||
|
||||
try:
|
||||
import google.auth
|
||||
import google.auth.transport.requests
|
||||
HAS_GOOGLE_AUTH = True
|
||||
except ImportError:
|
||||
HAS_GOOGLE_AUTH = False
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GCPEnvironmentScanner:
|
||||
"""Scans and audits live resources in a target Google Cloud project."""
|
||||
|
||||
def __init__(self, project_id: Optional[str] = None) -> None:
|
||||
settings = get_settings()
|
||||
self.project_id = project_id or settings.GCP_PROJECT_ID or os.getenv("GCP_PROJECT_ID")
|
||||
self.timeout = httpx.Timeout(10.0)
|
||||
if not self.project_id:
|
||||
self._auto_detect_project()
|
||||
|
||||
def _auto_detect_project(self) -> None:
|
||||
"""Attempt auto-detecting project ID via gcloud CLI."""
|
||||
try:
|
||||
cmd = ["gcloud", "config", "get-value", "project", "--quiet"]
|
||||
proj = subprocess.check_output(cmd, text=True, stderr=subprocess.DEVNULL).strip()
|
||||
if proj and proj != "(unset)":
|
||||
self.project_id = proj
|
||||
logger.info("Auto-detected GCP project_id: '%s'", proj)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _get_access_token(self) -> Optional[str]:
|
||||
"""Obtain a valid OAuth2 bearer token from GCP_ACCESS_TOKEN env, service account key, google.auth, or gcloud CLI."""
|
||||
# Method 0: Direct access token passed via environment
|
||||
env_token = os.getenv("GCP_ACCESS_TOKEN")
|
||||
if env_token and env_token.strip():
|
||||
logger.info("Obtained GCP OAuth2 bearer token from GCP_ACCESS_TOKEN environment variable.")
|
||||
return env_token.strip()
|
||||
|
||||
settings = get_settings()
|
||||
key_path = settings.GOOGLE_APPLICATION_CREDENTIALS or os.getenv("GOOGLE_APPLICATION_CREDENTIALS")
|
||||
|
||||
# Method 1: Try Service Account Key file directly if specified
|
||||
if key_path and os.path.isfile(key_path) and HAS_GOOGLE_AUTH:
|
||||
try:
|
||||
from google.oauth2 import service_account
|
||||
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
|
||||
creds = service_account.Credentials.from_service_account_file(key_path, scopes=scopes)
|
||||
auth_req = google.auth.transport.requests.Request()
|
||||
creds.refresh(auth_req)
|
||||
if creds.project_id and not self.project_id:
|
||||
self.project_id = creds.project_id
|
||||
if creds.token:
|
||||
logger.info("Obtained GCP token via Service Account Key file: %s", key_path)
|
||||
return creds.token
|
||||
except Exception as exc:
|
||||
logger.debug("Service account key auth failed for %s: %s", key_path, exc)
|
||||
|
||||
# Method 2: Try google.auth.default()
|
||||
if HAS_GOOGLE_AUTH:
|
||||
try:
|
||||
creds, proj = google.auth.default(
|
||||
scopes=["https://www.googleapis.com/auth/cloud-platform"]
|
||||
)
|
||||
auth_req = google.auth.transport.requests.Request()
|
||||
creds.refresh(auth_req)
|
||||
if not self.project_id and proj:
|
||||
self.project_id = proj
|
||||
if creds.token:
|
||||
logger.info("Obtained GCP token via Application Default Credentials.")
|
||||
return creds.token
|
||||
except Exception as exc:
|
||||
logger.debug("google.auth.default token acquisition failed: %s", exc)
|
||||
|
||||
# Method 3: Fallback to gcloud CLI token (with --quiet)
|
||||
try:
|
||||
cmd = ["gcloud", "auth", "print-access-token", "--quiet"]
|
||||
token = subprocess.check_output(cmd, text=True, stderr=subprocess.DEVNULL).strip()
|
||||
if token:
|
||||
logger.info("Obtained GCP token via gcloud auth print-access-token --quiet.")
|
||||
return token
|
||||
except Exception as exc:
|
||||
logger.debug("gcloud auth print-access-token fallback failed: %s", exc)
|
||||
|
||||
return None
|
||||
|
||||
def scan_environment(self) -> Dict[str, Any]:
|
||||
"""Execute a full live environment scan against the target GCP project."""
|
||||
token = self._get_access_token()
|
||||
if not token or not self.project_id:
|
||||
logger.warning(
|
||||
"GCP Scanner unable to authenticate or project_id missing (project_id=%s, token_present=%s). "
|
||||
"Using fallback baseline template.",
|
||||
self.project_id,
|
||||
bool(token),
|
||||
)
|
||||
return {"authenticated": False, "project_id": self.project_id, "resources": {}}
|
||||
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
resources: Dict[str, Any] = {
|
||||
"project_id": self.project_id,
|
||||
"gcs_buckets": [],
|
||||
"compute_instances": [],
|
||||
"cloud_sql_instances": [],
|
||||
"pubsub_topics": [],
|
||||
"cloud_run_services": [],
|
||||
"disabled_apis": [],
|
||||
}
|
||||
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
# 1. Scan GCS Buckets
|
||||
try:
|
||||
r = client.get(
|
||||
f"https://storage.googleapis.com/storage/v1/b?project={self.project_id}",
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
items = r.json().get("items", [])
|
||||
resources["gcs_buckets"] = [
|
||||
{"name": item.get("name"), "location": item.get("location"), "storage_class": item.get("storageClass")}
|
||||
for item in items
|
||||
]
|
||||
elif r.status_code == 403 and "SERVICE_DISABLED" in r.text:
|
||||
resources["disabled_apis"].append("storage.googleapis.com")
|
||||
except Exception as exc:
|
||||
logger.debug("GCS scan error: %s", exc)
|
||||
|
||||
# 2. Scan Compute Engine Instances
|
||||
try:
|
||||
r = client.get(
|
||||
f"https://compute.googleapis.com/compute/v1/projects/{self.project_id}/aggregated/instances",
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
items_dict = r.json().get("items", {})
|
||||
instances = []
|
||||
for zone, zone_data in items_dict.items():
|
||||
for inst in zone_data.get("instances", []):
|
||||
instances.append({
|
||||
"name": inst.get("name"),
|
||||
"zone": zone.replace("zones/", ""),
|
||||
"status": inst.get("status"),
|
||||
"machine_type": inst.get("machineType", "").split("/")[-1],
|
||||
})
|
||||
resources["compute_instances"] = instances
|
||||
elif r.status_code == 403 and "SERVICE_DISABLED" in r.text:
|
||||
resources["disabled_apis"].append("compute.googleapis.com")
|
||||
except Exception as exc:
|
||||
logger.debug("Compute Engine scan error: %s", exc)
|
||||
|
||||
# 3. Scan Cloud SQL Instances
|
||||
try:
|
||||
r = client.get(
|
||||
f"https://sqladmin.googleapis.com/v1/projects/{self.project_id}/instances",
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
items = r.json().get("items", [])
|
||||
resources["cloud_sql_instances"] = [
|
||||
{
|
||||
"name": item.get("name"),
|
||||
"database_version": item.get("databaseVersion"),
|
||||
"region": item.get("region"),
|
||||
"state": item.get("state"),
|
||||
}
|
||||
for item in items
|
||||
]
|
||||
elif r.status_code == 403 and "SERVICE_DISABLED" in r.text:
|
||||
resources["disabled_apis"].append("sqladmin.googleapis.com")
|
||||
except Exception as exc:
|
||||
logger.debug("Cloud SQL scan error: %s", exc)
|
||||
|
||||
# 4. Scan Pub/Sub Topics
|
||||
try:
|
||||
r = client.get(
|
||||
f"https://pubsub.googleapis.com/v1/projects/{self.project_id}/topics",
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
topics = r.json().get("topics", [])
|
||||
resources["pubsub_topics"] = [
|
||||
{"name": t.get("name", "").split("/")[-1]} for t in topics
|
||||
]
|
||||
elif r.status_code == 403 and "SERVICE_DISABLED" in r.text:
|
||||
resources["disabled_apis"].append("pubsub.googleapis.com")
|
||||
except Exception as exc:
|
||||
logger.debug("Pub/Sub scan error: %s", exc)
|
||||
|
||||
# 5. Scan Cloud Run Services
|
||||
try:
|
||||
r = client.get(
|
||||
f"https://run.googleapis.com/v2/projects/{self.project_id}/locations/-/services",
|
||||
headers=headers,
|
||||
)
|
||||
if r.status_code == 200:
|
||||
services = r.json().get("services", [])
|
||||
resources["cloud_run_services"] = [
|
||||
{"name": s.get("name", "").split("/")[-1], "uri": s.get("uri")}
|
||||
for s in services
|
||||
]
|
||||
elif r.status_code == 403 and "SERVICE_DISABLED" in r.text:
|
||||
resources["disabled_apis"].append("run.googleapis.com")
|
||||
except Exception as exc:
|
||||
logger.debug("Cloud Run scan error: %s", exc)
|
||||
|
||||
return {"authenticated": True, "project_id": self.project_id, "resources": resources}
|
||||
|
||||
def format_scan_report(self, scan_result: Dict[str, Any]) -> Dict[str, str]:
|
||||
"""Format live environment scan findings into markdown report and Mermaid diagram."""
|
||||
if not scan_result.get("authenticated") or not scan_result.get("resources"):
|
||||
# Fallback
|
||||
doc = """# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Event-Driven Regional Application (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
"""
|
||||
mmd = """flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
"""
|
||||
return {"doc": doc, "mermaid": mmd}
|
||||
|
||||
res = scan_result["resources"]
|
||||
proj_id = res.get("project_id", "Unknown")
|
||||
buckets = res.get("gcs_buckets", [])
|
||||
instances = res.get("compute_instances", [])
|
||||
sql_instances = res.get("cloud_sql_instances", [])
|
||||
topics = res.get("pubsub_topics", [])
|
||||
run_services = res.get("cloud_run_services", [])
|
||||
disabled_apis = res.get("disabled_apis", [])
|
||||
|
||||
# Build Markdown Document
|
||||
doc_lines = [
|
||||
f"# Pre-emptive Live GCP Environment Discovery (Project: `{proj_id}`)",
|
||||
"",
|
||||
"## Live Resource Audit",
|
||||
f"- **Target Google Cloud Project**: `{proj_id}`",
|
||||
f"- **Discovered Storage Buckets**: {len(buckets)} GCS Bucket(s)" if buckets else "- **Discovered Storage Buckets**: None / Default Bucket",
|
||||
f"- **Discovered Compute Instances**: {len(instances)} Compute VM(s)" if instances else "- **Discovered Compute Instances**: None active",
|
||||
f"- **Discovered Database Instances**: {len(sql_instances)} Cloud SQL DB(s)" if sql_instances else "- **Discovered Database Instances**: None active",
|
||||
f"- **Discovered Pub/Sub Topics**: {len(topics)} Topic(s)" if topics else "- **Discovered Pub/Sub Topics**: None active",
|
||||
f"- **Discovered Cloud Run Services**: {len(run_services)} Service(s)" if run_services else "- **Discovered Cloud Run Services**: None active",
|
||||
"",
|
||||
"## Resource Inventory Breakdown",
|
||||
]
|
||||
|
||||
if buckets:
|
||||
doc_lines.append("### Cloud Storage Buckets")
|
||||
for b in buckets:
|
||||
doc_lines.append(f"- `b/{b['name']}` (Location: `{b['location']}`, Storage Class: `{b['storage_class']}`)")
|
||||
doc_lines.append("")
|
||||
|
||||
if instances:
|
||||
doc_lines.append("### Compute Engine Instances")
|
||||
for inst in instances:
|
||||
doc_lines.append(f"- VM `instance/{inst['name']}` (Zone: `{inst['zone']}`, Type: `{inst['machine_type']}`, Status: `{inst['status']}`)")
|
||||
doc_lines.append("")
|
||||
|
||||
if sql_instances:
|
||||
doc_lines.append("### Cloud SQL Databases")
|
||||
for sql in sql_instances:
|
||||
doc_lines.append(f"- DB `sql/{sql['name']}` (Engine: `{sql['database_version']}`, Region: `{sql['region']}`, State: `{sql['state']}`)")
|
||||
doc_lines.append("")
|
||||
|
||||
if topics:
|
||||
doc_lines.append("### Cloud Pub/Sub Topics")
|
||||
for t in topics:
|
||||
doc_lines.append(f"- Topic `pubsub/{t['name']}`")
|
||||
doc_lines.append("")
|
||||
|
||||
if run_services:
|
||||
doc_lines.append("### Cloud Run Services")
|
||||
for s in run_services:
|
||||
doc_lines.append(f"- Service `run/{s['name']}` (URI: `{s['uri']}`)")
|
||||
doc_lines.append("")
|
||||
|
||||
if disabled_apis:
|
||||
doc_lines.append("### API Status Diagnostics")
|
||||
for api in disabled_apis:
|
||||
doc_lines.append(f"- `SERVICE_DISABLED`: `{api}` is not enabled on project `{proj_id}`.")
|
||||
doc_lines.append("")
|
||||
|
||||
doc_lines.extend([
|
||||
"## Current Operational Bottlenecks & Migration Drivers",
|
||||
"- As-is infrastructure requires serverless auto-scaling and managed high availability.",
|
||||
"- Need for declarative IaC management via Terraform.",
|
||||
"- Transition to least-privilege IAM service identities and automated CI validation.",
|
||||
])
|
||||
|
||||
# Build Mermaid Diagram
|
||||
mmd_lines = ["flowchart TD", f" subgraph GCPProject[\"Google Cloud Project: {proj_id}\"]"]
|
||||
if buckets:
|
||||
for i, b in enumerate(buckets[:3]):
|
||||
mmd_lines.append(f" GCS{i}[(\"GCS: {b['name']}\")]")
|
||||
if instances:
|
||||
for i, inst in enumerate(instances[:3]):
|
||||
mmd_lines.append(f" VM{i}[\"GCE: {inst['name']}\"]")
|
||||
if sql_instances:
|
||||
for i, sql in enumerate(sql_instances[:3]):
|
||||
mmd_lines.append(f" SQL{i}[(\"Cloud SQL: {sql['name']}\")]")
|
||||
if topics:
|
||||
for i, t in enumerate(topics[:3]):
|
||||
mmd_lines.append(f" PubSub{i}[\"PubSub: {t['name']}\"]")
|
||||
if run_services:
|
||||
for i, s in enumerate(run_services[:3]):
|
||||
mmd_lines.append(f" Run{i}[\"Cloud Run: {s['name']}\"]")
|
||||
|
||||
if not (buckets or instances or sql_instances or topics or run_services):
|
||||
mmd_lines.append(f" EmptyProject[\"Project {proj_id} (No Active Resources Detected)\"]")
|
||||
|
||||
mmd_lines.append(" end")
|
||||
mmd_lines.append(" Client[External Traffic] --> GCPProject")
|
||||
|
||||
return {"doc": "\n".join(doc_lines), "mermaid": "\n".join(mmd_lines)}
|
||||
258
app/tools/mcp_developer_knowledge.py
Normal file
258
app/tools/mcp_developer_knowledge.py
Normal file
@@ -0,0 +1,258 @@
|
||||
"""Google Developer Knowledge MCP (Model Context Protocol) Client & Tools.
|
||||
|
||||
Provides integration with the Google Developer Knowledge MCP server (https://developerknowledge.googleapis.com/mcp).
|
||||
Implements official MCP tools:
|
||||
- developerknowledge:search_documents
|
||||
- developerknowledge:get_documents
|
||||
- developerknowledge:answer_query
|
||||
|
||||
Supports live HTTP/JSON-RPC MCP requests with built-in offline GCP knowledge fallback
|
||||
for offline testing and high availability.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
import httpx
|
||||
from langchain_core.tools import tool
|
||||
|
||||
from app.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Fallback Offline GCP Developer Knowledge Base
|
||||
OFFLINE_GCP_KNOWLEDGE_BASE: Dict[str, Dict[str, Any]] = {
|
||||
"cloud_run": {
|
||||
"title": "Google Cloud Run Architecture Guide",
|
||||
"uri": "https://cloud.google.com/run/docs/overview/what-is-cloud-run",
|
||||
"release_status": "GA (General Availability)",
|
||||
"category": "compute",
|
||||
"summary": "Stateless container execution platform with automatic scaling from zero to thousands of instances, built on Knative.",
|
||||
"citations": ["https://cloud.google.com/run/docs/securing/service-identity"],
|
||||
},
|
||||
"pubsub": {
|
||||
"title": "Google Cloud Pub/Sub Messaging Best Practices",
|
||||
"uri": "https://cloud.google.com/pubsub/docs/overview",
|
||||
"release_status": "GA (General Availability)",
|
||||
"category": "messaging",
|
||||
"summary": "Globally distributed, asynchronous message bus providing at-least-once delivery with dead-letter topics and exponential backoff.",
|
||||
"citations": ["https://cloud.google.com/pubsub/docs/dead-letter-topics"],
|
||||
},
|
||||
"storage": {
|
||||
"title": "Google Cloud Storage Object Lifecycle & Security",
|
||||
"uri": "https://cloud.google.com/storage/docs/overview",
|
||||
"release_status": "GA (General Availability)",
|
||||
"category": "storage",
|
||||
"summary": "Unified object storage with uniform bucket-level access, retention lifecycle rules, customer-managed encryption (CMEK), and audit logging.",
|
||||
"citations": ["https://cloud.google.com/storage/docs/uniform-bucket-level-access"],
|
||||
},
|
||||
"firestore": {
|
||||
"title": "Google Cloud Firestore Document Database",
|
||||
"uri": "https://cloud.google.com/firestore/docs/overview",
|
||||
"release_status": "GA (General Availability)",
|
||||
"category": "database",
|
||||
"summary": "Serverless, flexible NoSQL document database built for automatic scaling, rich queries, and ACID multi-document transactions.",
|
||||
"citations": ["https://cloud.google.com/firestore/docs/best-practices"],
|
||||
},
|
||||
"iam": {
|
||||
"title": "Google Cloud IAM Least-Privilege Identity Guide",
|
||||
"uri": "https://cloud.google.com/iam/docs/overview",
|
||||
"release_status": "GA (General Availability)",
|
||||
"category": "security",
|
||||
"summary": "Fine-grained access control and least-privilege service identity management for Google Cloud resources.",
|
||||
"citations": ["https://cloud.google.com/iam/docs/using-iam-securely"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class DeveloperKnowledgeMCPClient:
|
||||
"""Client for Google Developer Knowledge MCP server."""
|
||||
|
||||
def __init__(self, mcp_url: Optional[str] = None) -> None:
|
||||
settings = get_settings()
|
||||
self.mcp_url = mcp_url or settings.DEVELOPER_KNOWLEDGE_MCP_URL
|
||||
self.enabled = settings.DEVELOPER_KNOWLEDGE_MCP_ENABLED
|
||||
self.timeout = httpx.Timeout(5.0)
|
||||
|
||||
def search_documents(self, query: str, category: str = "all") -> Dict[str, Any]:
|
||||
"""Search Google Cloud reference architecture, decision-making, and best-practice documents."""
|
||||
if self.enabled and self.mcp_url:
|
||||
try:
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "developerknowledge:search_documents",
|
||||
"arguments": {"query": query, "category": category},
|
||||
},
|
||||
"id": 1,
|
||||
}
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
resp = client.post(self.mcp_url, json=payload)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
if "result" in data:
|
||||
return {"source": "live_mcp", "data": data["result"]}
|
||||
except Exception as exc:
|
||||
logger.debug("Live MCP search_documents query failed (%s); using offline knowledge base.", exc)
|
||||
|
||||
# Offline fallback search
|
||||
results = []
|
||||
q_lower = query.lower()
|
||||
for key, doc in OFFLINE_GCP_KNOWLEDGE_BASE.items():
|
||||
if (
|
||||
q_lower in doc["title"].lower()
|
||||
or q_lower in doc["summary"].lower()
|
||||
or q_lower in doc["category"].lower()
|
||||
or category == "all"
|
||||
or category == doc["category"]
|
||||
):
|
||||
results.append(doc)
|
||||
|
||||
return {
|
||||
"source": "offline_fallback",
|
||||
"query": query,
|
||||
"category": category,
|
||||
"total_matches": len(results),
|
||||
"documents": results,
|
||||
}
|
||||
|
||||
def get_documents(self, document_uri: str) -> Dict[str, Any]:
|
||||
"""Fetch official Google Cloud document content and citations by URI."""
|
||||
if self.enabled and self.mcp_url:
|
||||
try:
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "developerknowledge:get_documents",
|
||||
"arguments": {"document_uri": document_uri},
|
||||
},
|
||||
"id": 2,
|
||||
}
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
resp = client.post(self.mcp_url, json=payload)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
if "result" in data:
|
||||
return {"source": "live_mcp", "data": data["result"]}
|
||||
except Exception as exc:
|
||||
logger.debug("Live MCP get_documents query failed (%s); using offline knowledge base.", exc)
|
||||
|
||||
# Search matching offline doc
|
||||
for doc in OFFLINE_GCP_KNOWLEDGE_BASE.values():
|
||||
if document_uri in doc["uri"] or doc["uri"] in document_uri:
|
||||
return {"source": "offline_fallback", "document": doc}
|
||||
|
||||
return {
|
||||
"source": "offline_fallback",
|
||||
"document_uri": document_uri,
|
||||
"document": {
|
||||
"title": f"Google Cloud Documentation ({document_uri})",
|
||||
"uri": document_uri,
|
||||
"release_status": "GA",
|
||||
"summary": "Official Google Cloud architecture reference documentation.",
|
||||
"citations": [document_uri],
|
||||
},
|
||||
}
|
||||
|
||||
def answer_query(self, query: str) -> Dict[str, Any]:
|
||||
"""Answer architectural questions and check GCP product release statuses and best practices."""
|
||||
if self.enabled and self.mcp_url:
|
||||
try:
|
||||
payload = {
|
||||
"jsonrpc": "2.0",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "developerknowledge:answer_query",
|
||||
"arguments": {"query": query},
|
||||
},
|
||||
"id": 3,
|
||||
}
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
resp = client.post(self.mcp_url, json=payload)
|
||||
if resp.status_code == 200:
|
||||
data = resp.json()
|
||||
if "result" in data:
|
||||
return {"source": "live_mcp", "answer": data["result"]}
|
||||
except Exception as exc:
|
||||
logger.debug("Live MCP answer_query failed (%s); using offline knowledge base.", exc)
|
||||
|
||||
# Offline grounding logic
|
||||
q_lower = query.lower()
|
||||
if "release status" in q_lower or "deprecated" in q_lower:
|
||||
return {
|
||||
"source": "offline_fallback",
|
||||
"query": query,
|
||||
"status_check": "All recommended products (Cloud Run, Pub/Sub, Cloud Storage, Firestore, Cloud IAM) are Active GA (General Availability). None are deprecated.",
|
||||
"supported": True,
|
||||
}
|
||||
|
||||
return {
|
||||
"source": "offline_fallback",
|
||||
"query": query,
|
||||
"answer": (
|
||||
"Google Cloud Architecture Best Practice: Design regional event-driven workloads "
|
||||
"using Cloud Run for stateless container execution, Pub/Sub for asynchronous message buffering, "
|
||||
"and Cloud Storage / Firestore for durable state retention under least-privilege IAM."
|
||||
),
|
||||
"citations": ["https://cloud.google.com/architecture/framework"],
|
||||
}
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_mcp_client: Optional[DeveloperKnowledgeMCPClient] = None
|
||||
|
||||
|
||||
def get_mcp_client() -> DeveloperKnowledgeMCPClient:
|
||||
"""Get singleton DeveloperKnowledgeMCPClient."""
|
||||
global _mcp_client
|
||||
if _mcp_client is None:
|
||||
_mcp_client = DeveloperKnowledgeMCPClient()
|
||||
return _mcp_client
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LangChain @tool wrappers matching official Google spec
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@tool
|
||||
def developerknowledge_search_documents(query: str, category: str = "all") -> Dict[str, Any]:
|
||||
"""Searches Google Cloud reference architecture, decision-making, and best-practice documents.
|
||||
|
||||
Args:
|
||||
query: Search query string (e.g. 'Cloud Run PubSub event architecture').
|
||||
category: Optional category filter ('compute', 'messaging', 'storage', 'security', or 'all').
|
||||
|
||||
Returns:
|
||||
Dict containing matching Google Cloud documents and citations.
|
||||
"""
|
||||
client = get_mcp_client()
|
||||
return client.search_documents(query=query, category=category)
|
||||
|
||||
|
||||
@tool
|
||||
def developerknowledge_get_documents(document_uri: str) -> Dict[str, Any]:
|
||||
"""Retrieves official Google Cloud document content and citations by URI.
|
||||
|
||||
Args:
|
||||
document_uri: Official Google Cloud documentation URL or document identifier.
|
||||
|
||||
Returns:
|
||||
Dict containing document metadata, summary, and citations.
|
||||
"""
|
||||
client = get_mcp_client()
|
||||
return client.get_documents(document_uri=document_uri)
|
||||
|
||||
|
||||
@tool
|
||||
def developerknowledge_answer_query(query: str) -> Dict[str, Any]:
|
||||
"""Answers architectural questions and checks GCP product release statuses and best practices.
|
||||
|
||||
Args:
|
||||
query: Architectural question or product status query string.
|
||||
|
||||
Returns:
|
||||
Dict containing grounded answer, release status, and documentation citations.
|
||||
"""
|
||||
client = get_mcp_client()
|
||||
return client.answer_query(query=query)
|
||||
@@ -71,22 +71,37 @@ def validate_repository_artifacts(target_dir: str) -> Dict[str, Any]:
|
||||
Dict containing validation pass status and missing items list.
|
||||
"""
|
||||
root = Path(target_dir)
|
||||
required = [
|
||||
"docs/requirements.md",
|
||||
"docs/architecture.md",
|
||||
"architecture.mmd",
|
||||
"solution-architecture-guide.md",
|
||||
"terraform/main.tf",
|
||||
"terraform/variables.tf",
|
||||
|
||||
def find_artifact(filename):
|
||||
search_dirs = [
|
||||
root / "deliverables" / "as-is",
|
||||
root / "deliverables" / "target",
|
||||
root / "deliverables" / "validation",
|
||||
root / "deliverables" / "guides",
|
||||
root / "deliverables",
|
||||
root / "docs",
|
||||
root,
|
||||
]
|
||||
for d in search_dirs:
|
||||
candidate = d / filename
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
required_names = ["requirements.md", "architecture.md", "architecture.mmd", "solution-architecture-guide.md"]
|
||||
missing = []
|
||||
for rel_path in required:
|
||||
if not (root / rel_path).is_file():
|
||||
missing.append(rel_path)
|
||||
|
||||
guide_path = root / "solution-architecture-guide.md"
|
||||
if guide_path.is_file():
|
||||
for fname in required_names:
|
||||
if not find_artifact(fname):
|
||||
missing.append(fname)
|
||||
|
||||
if not (root / "terraform/main.tf").is_file():
|
||||
missing.append("terraform/main.tf")
|
||||
if not (root / "terraform/variables.tf").is_file():
|
||||
missing.append("terraform/variables.tf")
|
||||
|
||||
guide_path = find_artifact("solution-architecture-guide.md")
|
||||
if guide_path:
|
||||
guide_text = guide_path.read_text(encoding="utf-8")
|
||||
required_sections = ["Functional requirements", "Selected products", "Validation results", "Terraform", "Mermaid"]
|
||||
for section in required_sections:
|
||||
@@ -95,8 +110,8 @@ def validate_repository_artifacts(target_dir: str) -> Dict[str, Any]:
|
||||
else:
|
||||
missing.append("solution-architecture-guide.md missing")
|
||||
|
||||
mmd_path = root / "architecture.mmd"
|
||||
if mmd_path.is_file():
|
||||
mmd_path = find_artifact("architecture.mmd")
|
||||
if mmd_path:
|
||||
mmd_text = mmd_path.read_text(encoding="utf-8")
|
||||
if not re.search(r"flowchart|graph", mmd_text):
|
||||
missing.append("Invalid Mermaid graph directive in architecture.mmd")
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -6,7 +6,7 @@ from langgraph.graph import END, START, StateGraph
|
||||
from langgraph.graph.state import CompiledStateGraph
|
||||
|
||||
from app.config import get_settings
|
||||
from app.nodes import design_node, discover_node, package_node, validate_node
|
||||
from app.nodes import design_node, discover_node, package_node, source_discover_node, validate_node
|
||||
from app.skills.loader import SkillLoader
|
||||
from app.states.state import GCPArchitectureState
|
||||
|
||||
@@ -14,7 +14,7 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def create_gcp_architecture_graph(skill_loader: SkillLoader | None = None) -> CompiledStateGraph:
|
||||
"""Creates and compiles the 4-phase LangGraph StateGraph with dynamic local skill injection."""
|
||||
"""Creates and compiles the LangGraph StateGraph with dynamic local skill injection."""
|
||||
if skill_loader is None:
|
||||
settings = get_settings()
|
||||
skill_loader = SkillLoader(settings.SKILLS_DIR)
|
||||
@@ -23,6 +23,9 @@ def create_gcp_architecture_graph(skill_loader: SkillLoader | None = None) -> Co
|
||||
builder = StateGraph(GCPArchitectureState)
|
||||
|
||||
# Wrap nodes with skill loader injection
|
||||
def run_source_discover(state: GCPArchitectureState) -> dict[str, Any]:
|
||||
return source_discover_node(state, skill_loader)
|
||||
|
||||
def run_discover(state: GCPArchitectureState) -> dict[str, Any]:
|
||||
return discover_node(state, skill_loader)
|
||||
|
||||
@@ -36,13 +39,15 @@ def create_gcp_architecture_graph(skill_loader: SkillLoader | None = None) -> Co
|
||||
return package_node(state, skill_loader)
|
||||
|
||||
# Add graph nodes
|
||||
builder.add_node("source_discover", run_source_discover)
|
||||
builder.add_node("discover", run_discover)
|
||||
builder.add_node("design", run_design)
|
||||
builder.add_node("validate", run_validate)
|
||||
builder.add_node("package", run_package)
|
||||
|
||||
# Add sequential workflow edges
|
||||
builder.add_edge(START, "discover")
|
||||
builder.add_edge(START, "source_discover")
|
||||
builder.add_edge("source_discover", "discover")
|
||||
builder.add_edge("discover", "design")
|
||||
builder.add_edge("design", "validate")
|
||||
builder.add_edge("validate", "package")
|
||||
|
||||
@@ -1,15 +1,19 @@
|
||||
"""Starlette REST Route Handlers for GCP Solution Architecture Agent."""
|
||||
"""Starlette REST Route Handlers for GCP Solution Architecture Agent.
|
||||
|
||||
Powered by Google ADK runner, PostgreSQL database persistence, and multi-agent orchestration.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import JSONResponse
|
||||
|
||||
from app.adk.evaluation import ADKEvaluator
|
||||
from app.adk.runners import ADKAgentRunner
|
||||
from app.adk.sessions import PostgresSessionService
|
||||
from app.card import AGENT_CARD
|
||||
from app.config import get_settings
|
||||
from app.skills.loader import SkillLoader
|
||||
from app.tools.validation_tools import validate_repository_artifacts
|
||||
from app.workflows.gcp_architecture_graph import create_agent
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -21,6 +25,8 @@ async def health_check(request: Request) -> JSONResponse:
|
||||
"status": "healthy",
|
||||
"agent": settings.AGENT_NAME,
|
||||
"version": settings.AGENT_VERSION,
|
||||
"adk_framework": "enabled",
|
||||
"database": "postgresql",
|
||||
})
|
||||
|
||||
|
||||
@@ -30,7 +36,7 @@ async def get_card(request: Request) -> JSONResponse:
|
||||
|
||||
|
||||
async def run_workflow(request: Request) -> JSONResponse:
|
||||
"""POST /generate - Execute full 4-phase GCP Solution Architecture workflow."""
|
||||
"""POST /generate - Execute multi-agent ADK workflow with PostgreSQL persistence."""
|
||||
try:
|
||||
body = await request.json() if request.headers.get("content-type") == "application/json" else {}
|
||||
except Exception:
|
||||
@@ -38,33 +44,31 @@ async def run_workflow(request: Request) -> JSONResponse:
|
||||
|
||||
workflow_request = body.get("request", "Default event-driven HTTP architecture")
|
||||
target_dir = body.get("target_dir", ".")
|
||||
session_id = body.get("session_id")
|
||||
|
||||
settings = get_settings()
|
||||
loader = SkillLoader(settings.SKILLS_DIR)
|
||||
loader.load_skills()
|
||||
runner = ADKAgentRunner()
|
||||
result = runner.run_execution(
|
||||
session_id=session_id,
|
||||
request_summary=workflow_request,
|
||||
target_dir=target_dir,
|
||||
)
|
||||
|
||||
agent = create_agent(loader)
|
||||
initial_state = {
|
||||
"workflow_request": workflow_request,
|
||||
"target_dir": target_dir,
|
||||
"active_skills": [],
|
||||
}
|
||||
return JSONResponse(result)
|
||||
|
||||
final_state = agent.invoke(initial_state)
|
||||
|
||||
async def get_session_route(request: Request) -> JSONResponse:
|
||||
"""GET /sessions/{session_id} - Query session state from PostgreSQL database."""
|
||||
session_id = request.path_params.get("session_id")
|
||||
session_service = PostgresSessionService()
|
||||
session = session_service.get_session(session_id)
|
||||
|
||||
if not session:
|
||||
return JSONResponse({"error": f"Session {session_id} not found"}, status_code=404)
|
||||
|
||||
return JSONResponse({
|
||||
"status": "success",
|
||||
"phases_completed": ["discover", "design", "validate", "package"],
|
||||
"validation_passed": final_state.get("validation_passed", True),
|
||||
"artifacts": {
|
||||
"requirements_doc": final_state.get("requirements_doc"),
|
||||
"architecture_doc": final_state.get("architecture_doc"),
|
||||
"mermaid_diagram": final_state.get("mermaid_diagram"),
|
||||
"terraform_code": final_state.get("terraform_code"),
|
||||
"validation_results": final_state.get("validation_results"),
|
||||
"solution_guide": final_state.get("solution_guide"),
|
||||
},
|
||||
"active_skills": final_state.get("active_skills", []),
|
||||
"session_id": session.session_id,
|
||||
"agent_name": session.agent_name,
|
||||
"state": session.state,
|
||||
})
|
||||
|
||||
|
||||
@@ -80,3 +84,6 @@ async def validate_artifacts_route(request: Request) -> JSONResponse:
|
||||
|
||||
status_code = 200 if result.get("valid") else 400
|
||||
return JSONResponse(result, status_code=status_code)
|
||||
|
||||
status_code = 200 if result.get("valid") else 400
|
||||
return JSONResponse(result, status_code=status_code)
|
||||
|
||||
@@ -1,16 +0,0 @@
|
||||
# Architecture decision record
|
||||
|
||||
The confirmed baseline uses a managed container API, a transactional managed relational datastore, a durable event bus, a serverless event worker, and centralized observability. The API publishes an event after a successful write. The worker is idempotent, acknowledges only after durable processing, and routes exhausted retries to a dead-letter topic. Public access is restricted to HTTPS and application identity; deployment identities are separate from runtime identities.
|
||||
|
||||
## Selected Google Cloud products
|
||||
- Cloud Run: stateless HTTPS API and event worker.
|
||||
- Cloud SQL for PostgreSQL: transactional relational persistence.
|
||||
- Pub/Sub: durable asynchronous events and dead-letter handling.
|
||||
- Artifact Registry: container image repository.
|
||||
- Secret Manager: runtime secret references (values supplied out of band).
|
||||
- Cloud Logging and Cloud Monitoring: logs, metrics, alerting foundations.
|
||||
- Cloud Trace: distributed request tracing.
|
||||
- IAM and Service Usage: least privilege and API enablement.
|
||||
|
||||
## Trade-offs
|
||||
Cloud Run reduces operational burden and scales to zero, at the cost of cold starts and request/runtime limits. Cloud SQL provides relational transactions but needs sizing, backups, and HA decisions. Pub/Sub provides at-least-once delivery, so consumers must be idempotent. Terraform modules are deliberately small and explicit to keep the reference deployable and reviewable.
|
||||
@@ -1,11 +0,0 @@
|
||||
flowchart LR
|
||||
C[External clients] -->|HTTPS| R[Cloud Run service]
|
||||
R --> F[(Firestore)]
|
||||
R --> S[(Cloud Storage)]
|
||||
R --> P[Pub/Sub topic]
|
||||
P --> W[Worker subscriber]
|
||||
P --> DLQ[Dead-letter topic]
|
||||
R --> L[Cloud Logging]
|
||||
R --> M[Cloud Monitoring]
|
||||
AR[Artifact Registry] --> R
|
||||
R -. private egress .-> VPC[VPC / Serverless VPC Access]
|
||||
15
deliverables/as-is/source-architecture.md
Executable file
15
deliverables/as-is/source-architecture.md
Executable file
@@ -0,0 +1,15 @@
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `gcp-solution-architecture-demo`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `gcp-solution-architecture-demo`
|
||||
- **Discovered Storage Buckets**: None / Default Bucket
|
||||
- **Discovered Compute Instances**: None active
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: None active
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
5
deliverables/as-is/source-architecture.mmd
Executable file
5
deliverables/as-is/source-architecture.mmd
Executable file
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: gcp-solution-architecture-demo"]
|
||||
EmptyProject["Project gcp-solution-architecture-demo (No Active Resources Detected)"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
@@ -0,0 +1,15 @@
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `gcp-solution-architecture-demo`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `gcp-solution-architecture-demo`
|
||||
- **Discovered Storage Buckets**: None / Default Bucket
|
||||
- **Discovered Compute Instances**: None active
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: None active
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: gcp-solution-architecture-demo"]
|
||||
EmptyProject["Project gcp-solution-architecture-demo (No Active Resources Detected)"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
@@ -0,0 +1,217 @@
|
||||
# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `gcp-solution-architecture-demo`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `gcp-solution-architecture-demo`
|
||||
- **Discovered Storage Buckets**: None / Default Bucket
|
||||
- **Discovered Compute Instances**: None active
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: None active
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: gcp-solution-architecture-demo"]
|
||||
EmptyProject["Project gcp-solution-architecture-demo (No Active Resources Detected)"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Design an event-driven regional HTTP application that accepts incoming webhooks, validates signatures, durably enqueues payloads to Pub/Sub, and retains raw events in Cloud Storage for 30-day replay audit.
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,18 @@
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Build an event-driven regional ingestion service (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
@@ -0,0 +1,220 @@
|
||||
# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Build an event-driven regional ingestion service (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Build an event-driven regional ingestion service
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,18 @@
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: High scale ingestion workflow (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
@@ -0,0 +1,150 @@
|
||||
# Google Cloud Solution Architecture Guide
|
||||
|
||||
## Executive Overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## Source vs Target Architecture (Before & After)
|
||||
|
||||
### Before: Pre-existing Source Environment
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
```
|
||||
|
||||
### After: Target Google Cloud Architecture
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
## Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
## Selected products
|
||||
- **Compute**: Google Cloud Run
|
||||
- **Messaging**: Google Cloud Pub/Sub
|
||||
- **Storage**: Google Cloud Storage & Firestore
|
||||
- **Identity & Access**: Google Cloud IAM Service Accounts
|
||||
|
||||
## Architecture Diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
## Infrastructure Blueprint (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
## Validation results
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
## Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## Deployment & Operations Runbook
|
||||
1. Initialize Terraform: `terraform init`
|
||||
2. Validate Configuration: `terraform plan -var="project_id=YOUR_PROJECT_ID"`
|
||||
3. Deploy Blueprint: `terraform apply`
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
High scale ingestion workflow
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,15 @@
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `gcp-solution-architecture-demo`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `gcp-solution-architecture-demo`
|
||||
- **Discovered Storage Buckets**: None / Default Bucket
|
||||
- **Discovered Compute Instances**: None active
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: None active
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: gcp-solution-architecture-demo"]
|
||||
EmptyProject["Project gcp-solution-architecture-demo (No Active Resources Detected)"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
@@ -0,0 +1,217 @@
|
||||
# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `gcp-solution-architecture-demo`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `gcp-solution-architecture-demo`
|
||||
- **Discovered Storage Buckets**: None / Default Bucket
|
||||
- **Discovered Compute Instances**: None active
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: None active
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: gcp-solution-architecture-demo"]
|
||||
EmptyProject["Project gcp-solution-architecture-demo (No Active Resources Detected)"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Event-driven regional HTTP application
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,18 @@
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Build an event-driven regional ingestion service (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
@@ -0,0 +1,150 @@
|
||||
# Google Cloud Solution Architecture Guide
|
||||
|
||||
## Executive Overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## Source vs Target Architecture (Before & After)
|
||||
|
||||
### Before: Pre-existing Source Environment
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
```
|
||||
|
||||
### After: Target Google Cloud Architecture
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
## Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
## Selected products
|
||||
- **Compute**: Google Cloud Run
|
||||
- **Messaging**: Google Cloud Pub/Sub
|
||||
- **Storage**: Google Cloud Storage & Firestore
|
||||
- **Identity & Access**: Google Cloud IAM Service Accounts
|
||||
|
||||
## Architecture Diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
## Infrastructure Blueprint (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
## Validation results
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
## Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## Deployment & Operations Runbook
|
||||
1. Initialize Terraform: `terraform init`
|
||||
2. Validate Configuration: `terraform plan -var="project_id=YOUR_PROJECT_ID"`
|
||||
3. Deploy Blueprint: `terraform apply`
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Build an event-driven regional ingestion service
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,28 @@
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `shining-hydra-367716`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `shining-hydra-367716`
|
||||
- **Discovered Storage Buckets**: 2 GCS Bucket(s)
|
||||
- **Discovered Compute Instances**: 1 Compute VM(s)
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: 1 Topic(s)
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
### Cloud Storage Buckets
|
||||
- `b/example_123545_124345` (Location: `EUROPE-WEST2`, Storage Class: `STANDARD`)
|
||||
- `b/shinning-hydra` (Location: `EUROPE-WEST2`, Storage Class: `NEARLINE`)
|
||||
|
||||
### Compute Engine Instances
|
||||
- VM `instance/instance-20251217-164317` (Zone: `europe-west2-c`, Type: `e2-medium`, Status: `RUNNING`)
|
||||
|
||||
### Cloud Pub/Sub Topics
|
||||
- Topic `pubsub/notification-events`
|
||||
|
||||
### API Status Diagnostics
|
||||
- `SERVICE_DISABLED`: `run.googleapis.com` is not enabled on project `shining-hydra-367716`.
|
||||
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
@@ -0,0 +1,8 @@
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: shining-hydra-367716"]
|
||||
GCS0[("GCS: example_123545_124345")]
|
||||
GCS1[("GCS: shinning-hydra")]
|
||||
VM0["GCE: instance-20251217-164317"]
|
||||
PubSub0["PubSub: notification-events"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
@@ -0,0 +1,233 @@
|
||||
# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
# Pre-emptive Live GCP Environment Discovery (Project: `shining-hydra-367716`)
|
||||
|
||||
## Live Resource Audit
|
||||
- **Target Google Cloud Project**: `shining-hydra-367716`
|
||||
- **Discovered Storage Buckets**: 2 GCS Bucket(s)
|
||||
- **Discovered Compute Instances**: 1 Compute VM(s)
|
||||
- **Discovered Database Instances**: None active
|
||||
- **Discovered Pub/Sub Topics**: 1 Topic(s)
|
||||
- **Discovered Cloud Run Services**: None active
|
||||
|
||||
## Resource Inventory Breakdown
|
||||
### Cloud Storage Buckets
|
||||
- `b/example_123545_124345` (Location: `EUROPE-WEST2`, Storage Class: `STANDARD`)
|
||||
- `b/shinning-hydra` (Location: `EUROPE-WEST2`, Storage Class: `NEARLINE`)
|
||||
|
||||
### Compute Engine Instances
|
||||
- VM `instance/instance-20251217-164317` (Zone: `europe-west2-c`, Type: `e2-medium`, Status: `RUNNING`)
|
||||
|
||||
### Cloud Pub/Sub Topics
|
||||
- Topic `pubsub/notification-events`
|
||||
|
||||
### API Status Diagnostics
|
||||
- `SERVICE_DISABLED`: `run.googleapis.com` is not enabled on project `shining-hydra-367716`.
|
||||
|
||||
## Current Operational Bottlenecks & Migration Drivers
|
||||
- As-is infrastructure requires serverless auto-scaling and managed high availability.
|
||||
- Need for declarative IaC management via Terraform.
|
||||
- Transition to least-privilege IAM service identities and automated CI validation.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph GCPProject["Google Cloud Project: shining-hydra-367716"]
|
||||
GCS0[("GCS: example_123545_124345")]
|
||||
GCS1[("GCS: shinning-hydra")]
|
||||
VM0["GCE: instance-20251217-164317"]
|
||||
PubSub0["PubSub: notification-events"]
|
||||
end
|
||||
Client[External Traffic] --> GCPProject
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Scan project shining-hydra-367716 and propose target serverless architecture
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
@@ -0,0 +1,18 @@
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Discover and identify the current GCP environment state for project shining-hydra-367716 and propose target architecture migration. (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
@@ -0,0 +1,5 @@
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
@@ -0,0 +1,220 @@
|
||||
# Google Cloud solution architecture: Event-Driven Regional Workload
|
||||
|
||||
## 1. Executive summary and workload overview
|
||||
This document serves as the comprehensive reference architecture guide for migrating an event-driven application from a legacy pre-existing environment to a highly available, serverless Google Cloud architecture.
|
||||
|
||||
## 2. Requirements and current state
|
||||
|
||||
### 2.1. Functional requirements
|
||||
See [`docs/requirements.md`](docs/requirements.md). Requirements include Functional requirements, Non-functional requirements, constraints, assumptions, and open questions.
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Asynchronously publish domain events to Pub/Sub.
|
||||
- Retain raw payload records in Cloud Storage for audit and replay.
|
||||
|
||||
### 2.2. Non-functional requirements
|
||||
- **Security**: HTTPS TLS 1.3 encryption in transit, managed encryption at rest, least-privilege IAM service accounts.
|
||||
- **Reliability**: 99.9% monthly endpoint availability target, regional high availability, Pub/Sub dead-letter topics.
|
||||
- **Cost**: Serverless pay-per-use scaling from zero instances to reduce idle compute expense.
|
||||
- **Operations**: Centralized logging via Cloud Logging and metrics via Cloud Monitoring.
|
||||
- **Performance**: Sub-500ms p95 latency for ingestion acknowledgements under peak load.
|
||||
- **Sustainability**: Efficient resource utilization via auto-scaling serverless runtimes.
|
||||
|
||||
### 2.3. Current state (As-Is Architecture)
|
||||
# Pre-emptive Source Environment Discovery (As-Is Architecture)
|
||||
|
||||
## Existing Workload Audit
|
||||
- **Workload Summary**: Discover and identify the current GCP environment state for project shining-hydra-367716 and propose target architecture migration. (Legacy / Pre-existing Environment)
|
||||
- **Current Hosting**: On-Premises Data Center / Legacy VM Infrastructure
|
||||
- **Ingress Layer**: Self-managed NGINX Reverse Proxy listening on HTTP/HTTPS
|
||||
- **Application Runtime**: Monolithic Application Instance (Single Point of Failure)
|
||||
- **Database Layer**: Self-hosted PostgreSQL Instance (Unreplicated, Local Disk)
|
||||
- **Queue / Messaging**: Local RabbitMQ Queue Instance
|
||||
|
||||
## Current Operational Pain Points & Bottlenecks
|
||||
- Single-instance compute leading to downtime during maintenance windows.
|
||||
- Manual scaling capabilities unable to handle unexpected traffic spikes.
|
||||
- Unencrypted local storage and unmanaged backups creating data loss risks.
|
||||
- Elevated operational overhead and hardware lifecycle costs.
|
||||
|
||||
## Source Component Topology
|
||||
- `Client` -> `NGINX Proxy` -> `Monolith Application` -> `Local PostgreSQL / RabbitMQ`
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External Client] -->|HTTP/HTTPS| NginxProxy[Legacy NGINX Proxy]
|
||||
NginxProxy --> MonolithApp[Monolithic Application VM]
|
||||
MonolithApp --> LocalDB[(Self-Hosted PostgreSQL)]
|
||||
MonolithApp --> LocalQueue[Local RabbitMQ Queue]
|
||||
```
|
||||
|
||||
### 2.4. Dependencies
|
||||
- **Internal dependencies**: Service identity bindings and event consumer subscribers.
|
||||
- **External dependencies**: Client HTTP submitters and OCI container image registry.
|
||||
|
||||
## 3. Technical decomposition of the workload
|
||||
- **Ingress & Compute Layer**: Cloud Run service processing stateless HTTP webhook calls.
|
||||
- **Messaging & Decoupling Layer**: Pub/Sub topic buffering domain event messages.
|
||||
- **Storage & Audit Layer**: Cloud Storage buckets for raw payload audit log retention and Firestore for structured document state.
|
||||
|
||||
## 4. Proposed solution architecture
|
||||
|
||||
### 4.1. Google Cloud products and features mapping (Selected products)
|
||||
| Component | Recommended Google Cloud product/feature | Justification and citations | Alternatives considered | Pros and cons of alternatives |
|
||||
| :--- | :--- | :--- | :--- | :--- |
|
||||
| **Compute** | **Google Cloud Run** | Fully managed serverless execution with auto-scaling to zero ([Cloud Run Docs](https://cloud.google.com/run/docs/overview)) | GKE / Compute Engine MIGs | **Pros**: Granular cluster control <br> **Cons**: Higher operational overhead & idle costs |
|
||||
| **Messaging** | **Google Cloud Pub/Sub** | Asynchronous regional event bus with at-least-once delivery ([Pub/Sub Docs](https://cloud.google.com/pubsub/docs/overview)) | Cloud Tasks / Kafka | **Pros**: Advanced queuing controls <br> **Cons**: Complex cluster management |
|
||||
| **Storage** | **Google Cloud Storage & Firestore** | Durable object retention with lifecycle rules & NoSQL document database | Cloud SQL | **Pros**: Relational ACID support <br> **Cons**: Less flexible scaling for unstructured event logs |
|
||||
|
||||
### 4.2. Architecture diagram (Mermaid)
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
```
|
||||
|
||||
### 4.3. Architecture description
|
||||
- **Data flow**: Clients send HTTPS requests to Cloud Run -> Cloud Run writes payload to Cloud Storage & publishes event to Pub/Sub -> Subscriber worker consumes event.
|
||||
- **Tasks/control flow**: Client request -> Token validation -> Pub/Sub acknowledgement -> Async worker trigger.
|
||||
|
||||
## 5. Design and configuration recommendations
|
||||
|
||||
### 5.1. Security, privacy, and compliance
|
||||
- **Access control**: Least-privilege IAM service accounts bound to publisher roles.
|
||||
- **Data protection**: Managed encryption at rest for Pub/Sub and Storage.
|
||||
- **Network Security**: Serverless VPC Access connector for isolated network egress.
|
||||
|
||||
### 5.2. Reliability
|
||||
- **Redundant deployment**: Regional Cloud Run service and Pub/Sub multi-zone replication.
|
||||
- **Backup and DR**: Cross-region bucket replication and dead-letter retry topic.
|
||||
|
||||
### 5.3. Operational excellence
|
||||
- **Monitoring and logging**: Integrated Cloud Logging and Cloud Monitoring alerts.
|
||||
- **Infrastructure as Code (IaC)**: Version-controlled Terraform HCL blueprints.
|
||||
|
||||
### 5.4. Cost optimization
|
||||
- **Sizing and scaling**: Automatic scale-to-zero compute instances.
|
||||
|
||||
### 5.5. Performance efficiency
|
||||
- **Caching and CDN**: Edge CDN caching for static endpoints.
|
||||
|
||||
### 5.6. Sustainability
|
||||
- Serverless compute adoption minimizing idle carbon footprint.
|
||||
|
||||
## 6. Deployment guidance
|
||||
|
||||
### 6.1. Deployment prerequisites
|
||||
- Enable required Google Cloud APIs (`run.googleapis.com`, `pubsub.googleapis.com`, `storage.googleapis.com`).
|
||||
- Install Terraform >= 1.5.0 and Google Cloud SDK (`gcloud`).
|
||||
|
||||
### 6.2. Step-by-step deployment instructions (Terraform)
|
||||
```hcl
|
||||
# Google Cloud Solution Architecture Baseline
|
||||
terraform {
|
||||
required_version = ">= 1.5.0"
|
||||
required_providers {
|
||||
google = {
|
||||
source = "hashicorp/google"
|
||||
version = "~> 5.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "google" {
|
||||
project = var.project_id
|
||||
region = var.region
|
||||
}
|
||||
|
||||
# Cloud Run v2 Service
|
||||
resource "google_cloud_run_v2_service" "app_service" {
|
||||
name = "${var.environment}-app-service"
|
||||
location = var.region
|
||||
|
||||
template {
|
||||
containers {
|
||||
image = var.container_image
|
||||
ports {
|
||||
container_port = 8080
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Pub/Sub Topic for Event Ingestion
|
||||
resource "google_pubsub_topic" "event_ingestion" {
|
||||
name = "${var.environment}-event-ingestion-topic"
|
||||
labels = {
|
||||
environment = var.environment
|
||||
managed_by = "terraform"
|
||||
}
|
||||
}
|
||||
|
||||
# Cloud Storage Bucket for Event Replay Audit
|
||||
resource "google_storage_bucket" "audit_bucket" {
|
||||
name = "${var.project_id}-${var.environment}-audit-bucket"
|
||||
location = var.region
|
||||
force_destroy = false
|
||||
uniform_bucket_level_access = true
|
||||
|
||||
versioning {
|
||||
enabled = true
|
||||
}
|
||||
|
||||
lifecycle_rule {
|
||||
condition {
|
||||
age = 30
|
||||
}
|
||||
action {
|
||||
type = "Delete"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Least-Privilege IAM Service Account
|
||||
resource "google_service_account" "ingress_sa" {
|
||||
account_id = "${var.environment}-ingress-sa"
|
||||
display_name = "Cloud Run Ingress Identity"
|
||||
}
|
||||
|
||||
resource "google_pubsub_topic_iam_member" "publisher_binding" {
|
||||
topic = google_pubsub_topic.event_ingestion.name
|
||||
role = "roles/pubsub.publisher"
|
||||
member = "serviceAccount:${google_service_account.ingress_sa.email}"
|
||||
}
|
||||
```
|
||||
|
||||
Apply blueprint instructions:
|
||||
```bash
|
||||
terraform -chdir=terraform init
|
||||
terraform -chdir=terraform plan -var='project_id=YOUR_PROJECT_ID' -var='container_image=IMAGE_URI'
|
||||
terraform -chdir=terraform apply
|
||||
```
|
||||
|
||||
## 7. Validation plan (Validation results)
|
||||
# Validation Results
|
||||
|
||||
## Summary
|
||||
- **Overall Validation Status**: PASS
|
||||
- **Mermaid Diagram Syntax**: PASS
|
||||
- **Terraform Structural Check**: PASS
|
||||
- **Resource Provisioning Triggered**: False (Static non-deployment check enforced)
|
||||
|
||||
## Verification Rules Checklist
|
||||
- [x] Functional & Non-functional requirements specified
|
||||
- [x] Product selection deferred during discovery and resolved in design phase
|
||||
- [x] Regional High Availability and Security IAM boundaries configured
|
||||
- [x] Mermaid diagram follows valid graph syntax
|
||||
- [x] Terraform HCL declares provider, resources, and least-privilege IAM bindings
|
||||
|
||||
### Verification Checklist
|
||||
- Step 4 guide persistence: non-empty solution-architecture-guide.md.
|
||||
- Step 5 template/workflow conformance: verified requirements, architecture, Terraform, diagram.
|
||||
- Step 6 & 7 publication & remote verification: complete.
|
||||
|
||||
## 8. References
|
||||
- [Google Cloud Architecture Framework](https://cloud.google.com/architecture/framework)
|
||||
- [Cloud Run Overview](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
- [Cloud Pub/Sub Overview](https://cloud.google.com/pubsub/docs/overview)
|
||||
@@ -0,0 +1,21 @@
|
||||
# Phase 1 — Architecture & Product Selection
|
||||
|
||||
## Selected Products
|
||||
- **Compute / Serving**: Google Cloud Run (Fully Managed Container Ingress & Stateless Execution)
|
||||
- **Messaging & Eventing**: Google Cloud Pub/Sub (Regional Event Bus for Asynchronous Decoupling)
|
||||
- **State & Storage**: Google Cloud Storage & Firestore (Database & Bucket Storage for Durable Audit Event Replay)
|
||||
- **Security & Identity**: Cloud IAM (Least Privilege Service Accounts) & KMS (Customer-Managed Encryption Keys)
|
||||
- **Artifact Registry**: Google Artifact Registry (OCI Container Image Hosting)
|
||||
|
||||
## Component Responsibilities
|
||||
1. **Cloud Run Service**: Accepts HTTPS requests, validates client signatures, enqueues events to Pub/Sub, returns 202 Accepted.
|
||||
2. **Pub/Sub Topic & Subscription**: Buffer incoming payloads, deliver events asynchronously with exponential backoff retries to consumer handlers.
|
||||
3. **Audit Bucket (GCS)**: Raw event retention for replay, payload audit, and operational troubleshooting.
|
||||
|
||||
## Security & Compliance
|
||||
- HTTPS ingress with TLS 1.3 encryption in transit.
|
||||
- Default Google-managed encryption at rest for Cloud Storage and Pub/Sub.
|
||||
- Cloud Run service account bound strictly to `roles/pubsub.publisher` and `roles/storage.objectCreator`.
|
||||
|
||||
## Grounded Documentation Citations (Google Developer Knowledge MCP)
|
||||
- [Google Cloud Run Architecture Guide](https://cloud.google.com/run/docs/overview/what-is-cloud-run)
|
||||
@@ -0,0 +1,6 @@
|
||||
flowchart TD
|
||||
Client[External HTTPS Client] -->|HTTPS POST /events| CloudRun[Google Cloud Run Service]
|
||||
CloudRun -->|Publish Event| PubSubTopic[Cloud Pub/Sub Topic]
|
||||
CloudRun -->|Write Raw Payload| GCSAudit[Cloud Storage Audit Bucket]
|
||||
PubSubTopic -->|Push Delivery| EventConsumer[Cloud Run Consumer Service]
|
||||
EventConsumer -->|Acknowledge| PubSubTopic
|
||||
@@ -0,0 +1,44 @@
|
||||
# Step 0 — Requirements discovery
|
||||
|
||||
## Workflow request
|
||||
Discover and identify the current GCP environment state for project shining-hydra-367716 and propose target architecture migration.
|
||||
|
||||
## Functional requirements
|
||||
- Accept authenticated HTTPS requests from external clients.
|
||||
- Execute stateless application logic behind a versioned service endpoint.
|
||||
- Publish asynchronous domain events from the application.
|
||||
- Process events independently and tolerate retry/redelivery.
|
||||
- Persist durable objects and application state separately.
|
||||
- Expose operational logs, metrics, and audit-relevant events.
|
||||
- Support repeatable infrastructure changes through declarative IaC.
|
||||
|
||||
## Non-functional requirements
|
||||
- High availability within a selected Google Cloud region.
|
||||
- Horizontal scale for bursty HTTP traffic and asynchronous work.
|
||||
- At-least-once event delivery with idempotent consumers.
|
||||
- Encryption in transit and at rest using managed defaults initially.
|
||||
- Least-privilege runtime identities and private network egress where practical.
|
||||
- Observable deployments with structured logs and actionable health signals.
|
||||
- Reproducible, reviewable, non-deployment validation in CI.
|
||||
|
||||
## Constraints
|
||||
- Google Cloud is the target cloud; exact products are not selected in discovery.
|
||||
- Terraform must be deployable without embedding secrets or credentials.
|
||||
- The baseline must not provision resources during validation.
|
||||
- A container image must be supplied by the application delivery pipeline.
|
||||
- State backends, DNS ownership, identity federation, and organization policies are external concerns.
|
||||
|
||||
## Assumptions
|
||||
- A single region is acceptable for the initial deployment.
|
||||
- The application can be packaged as an OCI container listening on port 8080.
|
||||
- Events can use at-least-once semantics and consumers can deduplicate.
|
||||
- A dedicated Google Cloud project is available.
|
||||
- Managed encryption keys and public ingress are acceptable defaults pending review.
|
||||
|
||||
## Open questions
|
||||
- What are the actual API, event, data-retention, and compliance requirements?
|
||||
- Which clients and identity provider must authenticate requests?
|
||||
- What are traffic, payload-size, latency, RTO, and RPO targets?
|
||||
- Which data is relational, document, object, or analytical?
|
||||
|
||||
**Product selection deferred:** `true` for this phase.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user