Files
gcp_solution_architecture_a…/skills/google-cloud-iac-patterns/references/bigquery.md
Jonathan Boniface 71da0ed504
Some checks failed
validation / verify (push) Failing after 12s
chore: housekeeping from the runs of 2026-10-09
- An Execution's files are written as each Deliverable is submitted, marked
  in_progress, so a killed process or one waiting for its user leaves them on
  disk. A write removes files an earlier write left that it no longer has.
- A question about delivered work is answered from that Execution's solution
  guide by the reply writer; a question about the service is answered as
  before. Nothing is started for either.
- Lessons: prove writes only for lessons the Skill holds; the Skill records
  the vendored Skill versions the lessons were read against, status names any
  that changed, and a test fails until they are reviewed.
- Four more code patterns: Firestore, BigQuery, Eventarc and GKE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-10 11:44:38 +01:00

3.2 KiB

BigQuery

A dataset, a partitioned and clustered table, and the grants for a writer and a reader.

  • location is a region or EU/US, and cannot be changed. Queries cannot join datasets in different locations.
  • delete_contents_on_destroy = false makes Terraform refuse to remove a dataset that still has tables.
  • A table's deletion_protection is true unless set; take it from a variable so a test deployment can be removed.
  • schema is a JSON string. Changing a column's type or mode replaces the table.
  • Partition on a DATE or TIMESTAMP column with time_partitioning, and set require_partition_filter so a query cannot scan every partition. expiration_ms there removes old partitions.
  • Grant on the dataset with google_bigquery_dataset_iam_member. A principal that runs queries also needs roles/bigquery.jobUser on the project.
  • Do not mix google_bigquery_dataset_iam_member with access blocks on the dataset: each removes what the other sets.
variable "project_id" {
  type        = string
  description = "Project to deploy into."
}

variable "region" {
  type        = string
  description = "Region for regional resources."
}

variable "writer_service_account_email" {
  type        = string
  description = "Service account that loads rows."
}

variable "reader_service_account_email" {
  type        = string
  description = "Service account that queries the dataset."
}

variable "deletion_protection" {
  type        = bool
  description = "Whether the table is protected from deletion."
  default     = true
}

resource "google_bigquery_dataset" "analytics" {
  project                    = var.project_id
  dataset_id                 = "analytics"
  location                   = var.region
  delete_contents_on_destroy = false

  default_table_expiration_ms = null
  max_time_travel_hours       = 168
}

resource "google_bigquery_table" "events" {
  project             = var.project_id
  dataset_id          = google_bigquery_dataset.analytics.dataset_id
  table_id            = "events"
  deletion_protection = var.deletion_protection

  require_partition_filter = true
  clustering               = ["event_type"]

  time_partitioning {
    type          = "DAY"
    field         = "event_time"
    expiration_ms = 34560000000
  }

  schema = jsonencode([
    { name = "event_id", type = "STRING", mode = "REQUIRED" },
    { name = "event_type", type = "STRING", mode = "REQUIRED" },
    { name = "event_time", type = "TIMESTAMP", mode = "REQUIRED" },
    { name = "payload", type = "JSON", mode = "NULLABLE" }
  ])
}

resource "google_bigquery_dataset_iam_member" "writer" {
  project    = var.project_id
  dataset_id = google_bigquery_dataset.analytics.dataset_id
  role       = "roles/bigquery.dataEditor"
  member     = "serviceAccount:${var.writer_service_account_email}"
}

resource "google_bigquery_dataset_iam_member" "reader" {
  project    = var.project_id
  dataset_id = google_bigquery_dataset.analytics.dataset_id
  role       = "roles/bigquery.dataViewer"
  member     = "serviceAccount:${var.reader_service_account_email}"
}

resource "google_project_iam_member" "reader_runs_queries" {
  project = var.project_id
  role    = "roles/bigquery.jobUser"
  member  = "serviceAccount:${var.reader_service_account_email}"
}