dataform-engineering-fundamentals

这段内容是审核时保存的快照。外部来源才是完整且最新的版本。

---
name: dataform-engineering-fundamentals
description: Use when developing BigQuery Dataform transformations, SQLX files, source declarations, or troubleshooting pipelines - enforces TDD workflow (tests first), ALWAYS use ${ref()} never hardcoded table paths, comprehensive columns:{} documentation, safety practices (--schema-suffix dev, --dry-run), proper ref() syntax, .sqlx for new declarations, no schema config in operations/tests, and architecture patterns that prevent technical debt under time pressure
---

# Dataform Engineering Fundamentals

## Overview

**Core principle**: Safety practices and proper architecture are NEVER optional in Dataform development, regardless of time pressure or business urgency. The time "saved" by shortcuts gets multiplied into hours of debugging, broken dependencies, and production issues.

**REQUIRED FOUNDATION:** This skill builds upon superpowers:test-driven-development. All TDD principles from that skill apply here; this skill adapts them for BigQuery Dataform SQLX files.

**For PostgreSQL / Supabase:** use **sqlanvil-engineering-fundamentals** instead (or alongside) — the architecture, `${ref()}`, TDD, and `columns:{}` rules below carry over unchanged; that skill covers the Postgres/Supabase deltas (config blocks, credentials, DDL, named connections).

**Official docs:** syntax/config/API at https://cloud.google.com/dataform/docs · repository structure & naming at https://cloud.google.com/dataform/docs/best-practices-repositories

## When to Use

Use this skill for ANY Dataform work — new SQLX transformations, modifying tables, adding sources, troubleshooting pipelines, and "quick" reports or ad-hoc analysis.

**Especially** when under time pressure, stakeholders are waiting, you're working late and exhausted, or you're tempted to "just make it work" — discipline matters most exactly when it's tempting to skip.

**Related skills:**
- **superpowers:brainstorming** — refine rough requirements into clear designs *before* writing code
- **superpowers:systematic-debugging** — structured problem-solving for pipeline failures
- **superpowers:root-cause-tracing** — trace cascading errors back to their source
- **elements-of-style:writing-clearly-and-concisely** — for column descriptions, commit messages, any prose

## Non-Negotiable Safety Practices

ALWAYS required. No exceptions for deadlines, urgency, or "simple" tasks.

### 1. Always Use `--schema-suffix dev` for Testing

```bash
# WRONG: testing in production
dataform run --actions my_table

# CORRECT: test in dev first
dataform run --schema-suffix dev --actions my_table
```

Writes to `schema_dev.my_table` instead of production, so you can test without touching real data or dashboards.

### 2. Always Use `--dry-run` Before Execution

```bash
dataform compile                                              # check compilation
dataform run --schema-suffix dev --dry-run --actions my_table # validate SQL, estimate cost
dataform run --schema-suffix dev --actions my_table           # only then execute
```

Catches SQL errors, missing dependencies, and cost surprises before using BigQuery slots.

### 3. Source Declarations Before `ref()`

Declare external tables before querying them, so dependency tracking works.

```sql
-- definitions/sources/external_system/table_name.sqlx
config {
  type: "declaration",
  database: "project_id",
  schema: "external_schema",
  name: "table_name"
}
```

Then reference with `FROM ${ref("table_name")}` — never `FROM \`project.external_schema.table_name\``.

### 4. ALWAYS Use `${ref()}` — NEVER Hardcoded Table Paths

```sql
-- WRONG
FROM `project.reporting_schema.customer_metrics`
SELECT * FROM project.source_schema.customers

-- CORRECT (after declaring the source)
FROM ${ref("customer_metrics")}
SELECT * FROM ${ref("customers")}
```

`${ref()}` builds the dependency graph automatically, makes `--schema-suffix` work, and keeps refactoring safe. Hardcoded paths break all three. **Exception: none.** There is no valid