feat: all four evaluation cases pass; intake, checklist review, thresholds

Case 04 (interactive)
- A scripted answer may name who it is for, so facts written for
  requirements discovery are not given to a Specialist confirming a draft.
- A reply that restates what the agreed requirements already say is read
  as an answer, not a request for changes.
- With a user present, a first submission is refused until the user has
  been shown a draft of that Deliverable (CONFIRM_DRAFTS).

Routing and reviews
- Intake: a message that starts something is read first. A question is
  answered and starts nothing; a change to delivered work starts as a new
  request with a note, until Change requests are built.
- Requirements and recommendations list what infrastructure code can show
  (conditions) apart from what it cannot (outside_code); the latter needs
  no Check and validation proposes nothing for it.
- A requirements review, "names no products", behind its own switch (off).
- Each pillar advisor reviews its pillar's checklist; the engine writes the
  reviews into the design recommendations. The judged checklist check went
  from 9-45% a pillar to 100% in one run of case 04.

Evaluation
- Per-case thresholds for Revisions, seconds and tokens, set from every
  completed run on this build.
- A progress file after every agent run, so an interrupted run keeps its
  time and tokens.
- eval/calibrate_judge.py with a 32-question set awaiting a person's
  labels. Unlabelled: Jev stable at 1.00, Gemini 0.92, agreeing on 0.91.

Smaller
- An Execution that ends with warnings writes its Deliverables, with the
  warnings and each status in the manifest.
- CI installs the application's dependencies before linting and testing.
- A load-balancer pattern in the IaC patterns Skill.

Plans record nine runs of 2026-10-08. Iteration 6 is complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-09 09:32:57 +01:00
parent d41607f505
commit 654370bb46
53 changed files with 2122 additions and 106 deletions

View File

@@ -4,7 +4,7 @@ description: >-
Terraform patterns for Google Cloud products that validate against one pinned
version of the hashicorp/google provider: Cloud Run services and jobs,
Pub/Sub, Cloud Storage, IAM, Secret Manager, Cloud KMS, Cloud SQL,
networking, Cloud Scheduler, Artifact Registry, Monitoring and API
networking, load balancing, Cloud Scheduler, Artifact Registry, Monitoring and API
enablement. Use when writing Terraform for a Google Cloud product, before
writing the resource, to start from a pattern whose arguments are known to be
in the right place.
@@ -35,6 +35,7 @@ Every pattern here passes `terraform validate` against `hashicorp/google` 8.6.0.
| Cloud KMS | `references/kms.md` |
| Cloud SQL | `references/cloud-sql.md` |
| VPC networking | `references/networking.md` |
| External Application Load Balancer, TLS policy, firewall policy, Identity-Aware Proxy | `references/load-balancer.md` |
| Cloud Scheduler | `references/cloud-scheduler.md` |
| Artifact Registry | `references/artifact-registry.md` |
| Cloud Monitoring and Logging | `references/monitoring.md` |

View File

@@ -0,0 +1,184 @@
# External Application Load Balancer
A global HTTPS load balancer in front of a Cloud Run service, with a TLS policy, a web application firewall policy and, optionally, Identity-Aware Proxy.
The chain is: forwarding rule → target HTTPS proxy → URL map → backend service → serverless network endpoint group → Cloud Run service. Each setting lives on one link of it, and putting one on the wrong link is the usual mistake:
- **The minimum TLS version is on an SSL policy, attached to the target HTTPS proxy** (`ssl_policy`). A backend service has no `ssl_policy` setting.
- **The certificate is on the target HTTPS proxy** (`ssl_certificates`).
- **The firewall policy is attached to the backend service** (`security_policy`).
- **Identity-Aware Proxy is the `iap` block of the backend service.** It decides who reaches the application; what a signed-in user may do inside it is the application's own code and cannot be shown by infrastructure.
- **Request logging is the `log_config` block of the backend service.**
- A backend service for Cloud Run uses a serverless NEG and takes no `health_checks`, no `port_name` and no `timeout_sec`.
- `load_balancing_scheme = "EXTERNAL_MANAGED"` on both the backend service and the forwarding rule.
- Set the Cloud Run service's `ingress` to `INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER`, or it stays reachable around the load balancer.
- Redirecting HTTP to HTTPS needs a second URL map, target HTTP proxy and forwarding rule on port 80.
```hcl
variable "project_id" {
type = string
description = "Project to deploy into."
}
variable "region" {
type = string
description = "Region for regional resources."
}
variable "domain" {
type = string
description = "Domain name the load balancer serves."
}
resource "google_cloud_run_v2_service" "app" {
project = var.project_id
name = "booking-app"
location = var.region
ingress = "INGRESS_TRAFFIC_INTERNAL_LOAD_BALANCER"
template {
containers {
image = "us-docker.pkg.dev/cloudrun/container/hello"
}
}
}
resource "google_compute_region_network_endpoint_group" "app" {
project = var.project_id
name = "booking-app"
region = var.region
network_endpoint_type = "SERVERLESS"
cloud_run {
service = google_cloud_run_v2_service.app.name
}
}
resource "google_compute_security_policy" "edge" {
project = var.project_id
name = "booking-edge"
rule {
action = "deny(403)"
priority = 1000
match {
expr {
expression = "evaluatePreconfiguredWaf('sqli-v33-stable', {'sensitivity': 1})"
}
}
description = "Block SQL injection."
}
rule {
action = "allow"
priority = 2147483647
match {
versioned_expr = "SRC_IPS_V1"
config {
src_ip_ranges = ["*"]
}
}
description = "Default rule."
}
}
resource "google_compute_backend_service" "app" {
project = var.project_id
name = "booking-app"
protocol = "HTTPS"
load_balancing_scheme = "EXTERNAL_MANAGED"
security_policy = google_compute_security_policy.edge.id
backend {
group = google_compute_region_network_endpoint_group.app.id
}
# Only signed-in identities granted roles/iap.httpsResourceAccessor get through.
iap {
enabled = true
}
log_config {
enable = true
sample_rate = 1.0
}
}
resource "google_compute_url_map" "app" {
project = var.project_id
name = "booking-app"
default_service = google_compute_backend_service.app.id
}
resource "google_compute_ssl_policy" "modern" {
project = var.project_id
name = "tls-1-2-modern"
profile = "MODERN"
min_tls_version = "TLS_1_2"
}
resource "google_compute_managed_ssl_certificate" "app" {
project = var.project_id
name = "booking-app"
managed {
domains = [var.domain]
}
}
resource "google_compute_target_https_proxy" "app" {
project = var.project_id
name = "booking-app"
url_map = google_compute_url_map.app.id
ssl_certificates = [google_compute_managed_ssl_certificate.app.id]
ssl_policy = google_compute_ssl_policy.modern.id
}
resource "google_compute_global_address" "app" {
project = var.project_id
name = "booking-app"
}
resource "google_compute_global_forwarding_rule" "https" {
project = var.project_id
name = "booking-app-https"
load_balancing_scheme = "EXTERNAL_MANAGED"
ip_address = google_compute_global_address.app.id
port_range = "443"
target = google_compute_target_https_proxy.app.id
}
# HTTP to HTTPS.
resource "google_compute_url_map" "redirect" {
project = var.project_id
name = "booking-app-redirect"
default_url_redirect {
https_redirect = true
strip_query = false
}
}
resource "google_compute_target_http_proxy" "redirect" {
project = var.project_id
name = "booking-app-redirect"
url_map = google_compute_url_map.redirect.id
}
resource "google_compute_global_forwarding_rule" "http" {
project = var.project_id
name = "booking-app-http"
load_balancing_scheme = "EXTERNAL_MANAGED"
ip_address = google_compute_global_address.app.id
port_range = "80"
target = google_compute_target_http_proxy.redirect.id
}
# Who may pass Identity-Aware Proxy.
resource "google_iap_web_backend_service_iam_member" "staff" {
project = var.project_id
web_backend_service = google_compute_backend_service.app.name
role = "roles/iap.httpsResourceAccessor"
member = "domain:${var.domain}"
}
```

View File

@@ -26,11 +26,14 @@ A list. Number functional requirements `FR-1`, `FR-2` and non-functional ones `N
[
{"id": "FR-1", "kind": "functional", "text": "Accept signed webhooks over HTTPS."},
{"id": "NFR-1", "kind": "non_functional", "text": "Raw events are encrypted at rest and retained for 30 days.",
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]}
"conditions": ["Raw events are encrypted at rest.", "Raw events are kept for 30 days and then deleted."]},
{"id": "NFR-2", "kind": "non_functional", "text": "The service is available 99.9% of each month, across two zones.",
"conditions": ["The service runs in at least two zones."],
"outside_code": ["Measured availability is at least 99.9% a month."]}
]
```
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
Every non-functional requirement lists its `conditions`: one entry for each separate thing that must be true for it to be met and that infrastructure configuration can show. What it asks for that configuration cannot show goes under `outside_code`: an outcome measured in operation, behaviour of the application's own code, or a process people follow. A requirement has at least one entry between the two; one that is entirely outside code has `"conditions": []`. Write each so it can be checked on its own, and so that together they say everything the text says. The Target architecture needs a Check for each one.
`kind` is `functional` or `non_functional`. Write one requirement per entry, so each can be checked on its own.
@@ -59,7 +62,7 @@ A list. Each recommendation names its pillar and the components it applies to; u
"conditions": ["The topic is encrypted with a customer-managed key.", "The messaging service's own identity may use that key."]}]
```
Every recommendation lists its `conditions`, as a non-functional requirement does. When it applies to several components, give a condition for each.
Every recommendation lists its `conditions`, and `outside_code` where it applies, as a non-functional requirement does. When it applies to several components, give a condition for each.
Write one configurable decision per recommendation. "Harden the service" cannot become a Check; "disable public ingress on the worker service" can.
@@ -87,7 +90,7 @@ Write one configurable decision per recommendation. "Harden the service" cannot
- **Elements.** One per resource to be created. `component` is the component it realises; leave it out for a supporting Element such as an identity, a key or a network. `product` for a component's Element is exactly the product selected for that component.
- **Relationships.** `from` and `to` are Element ids. `kind` says what the connection is: calls, publishes to, reads from, runs as, is granted access to.
- **Checks.** `applies_to` is an Element or Relationship id. `source` is the requirement or recommendation id the Check comes from. Every Element has at least one Check.
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
- **Coverage.** Every entry in the `conditions` of a non-functional requirement or a recommendation needs a Check that verifies it, with that requirement or recommendation as its `source`. An entry under `outside_code` gets no Check, and a requirement or recommendation with nothing but `outside_code` needs none. A condition that applies to several Elements needs a Check on each: "processing scales to zero when idle" applies to every processing Element, not one of them.
- **Completeness.** Every component has an Element. Every non-functional requirement and every recommendation is the `source` of at least one Check, unless the recommendation is listed under `not_applicable` with a reason. Every selected product appears on its component's Element.
#### Writing a Check
@@ -135,5 +138,5 @@ Implement every Element, Relationship and Check, and nothing the Target architec
- Give every Check exactly one verdict: `pass`, `fail` or `not_verifiable`, with the evidence you read in the code.
- `dry_run.verdict` is `pass`, `fail`, or `not_run` when the tool could not run.
- Find every gap in one pass: go through each requirement and recommendation in turn and compare what its text asks with the Checks that name it as `source`.
- Find every gap in one pass: go through each requirement and recommendation in turn and compare its `conditions` with the Checks that name it as `source`. Propose nothing for its `outside_code`.
- Judge the code against the Checks and nothing else. Something the Target architecture lacks is a `proposed_additions` entry, with the requirement or recommendation id it serves in `reason`. It is never a failed Check.