dataform

Dataform Core (@dataform/core)

@dataform/core is the open-source SQLX compiler, JavaScript workflow SDK, and Protocol Buffer schema package for Dataform. It compiles SQLX (.sqlx), JavaScript (.js), action configuration (actions.yaml), BigQuery Property Graph (graph.yaml), and Jupyter Notebook (.ipynb) definitions into a dependency-ordered execution graph (dataform.CompiledGraph) for BigQuery.

For command-line execution (dataform init, compile, test, run, format), see the Dataform CLI Reference (@dataform/cli) and the root repository overview.


Contents


Installation and Version Pinning

In both Dataform in Google Cloud and @dataform/cli, specify dataformCoreVersion in your project’s workflow_settings.yaml:

defaultProject: my-gcp-project
defaultLocation: US
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: <version>

When dataformCoreVersion is set in workflow_settings.yaml, @dataform/cli automatically installs that version of @dataform/core in an isolated temporary directory at compile time (see Stateless installs).

Via package.json / npm

If you manage @dataform/core and external JavaScript packages with package.json (omitting dataformCoreVersion from workflow_settings.yaml), or if you consume @dataform/core programmatically:

npm i @dataform/core

Project Structure

A standard Dataform project has the following layout:

my-dataform-project/
├── workflow_settings.yaml     # Project-level settings and defaults
├── definitions/               # Action definitions (.sqlx, .js, actions.yaml, graph.yaml, .sql, .ipynb)
│   ├── sources.sqlx
│   ├── staging/
│   │   └── stg_orders.sqlx
│   └── marts/
│       └── fct_orders.sqlx
└── includes/                  # Shared JavaScript modules and constants
    └── constants.js

Note on Project Execution & Security: Dataform compiles SQLX and JavaScript project code using Node.js’s native node:vm context. While module resolution is scoped to the project directory, node:vm is not a security sandbox against untrusted code. Only compile and run Dataform projects from trusted sources.


Project Configuration (workflow_settings.yaml)

workflow_settings.yaml is validated against the dataform.WorkflowSettings schema.

| Field | Type | Default | Description | |—|—|—|—| | dataformCoreVersion | string | — | @dataform/core version to compile against (for example, 3.1.5). | | defaultProject | string | — | Required. Default Google Cloud project ID (database) for compiled actions. | | defaultDataset | string | — | Required. Default BigQuery dataset (schema) for tables, views, and operations. | | defaultLocation | string | — | Required. Default BigQuery dataset location (for example, US, EU, us-central1). | | defaultAssertionDataset | string | — | Required. Default BigQuery dataset (schema) for assertion views. | | vars | map<string, string> | {} | Custom string key-value variables accessible via dataform.projectConfig.vars.<key>. | | projectSuffix | string | — | Suffix appended to target Google Cloud project IDs (<project>_<projectSuffix>). | | datasetSuffix | string | — | Suffix appended to target dataset names (<dataset>_<datasetSuffix>). | | namePrefix | string | — | Prefix prepended to target action names (<namePrefix>_<name>). | | builtinAssertionNamePrefix | string | — | Prefix prepended to auto-generated built-in assertion names. | | disableAssertions | boolean | false | Marks all manual and built-in assertions as disabled: true in the compiled graph. | | defaultReservation | string | — | Default BigQuery reservation for query execution (projects/<p>/locations/<l>/reservations/<r> or none). | | includeTestsInCompiledGraph | boolean | false | Includes unit test actions (type: "test") in CompiledGraph.tests. | | preserveGovernanceControls | boolean | false | Preserves existing BigQuery column governance controls (such as Data Policies) when recreating tables. | | defaultJobLabels | map<string, string> | {} | Default labels applied to BigQuery jobs executed for actions in the project. | | defaultNotebookRuntimeOptions | object | — | Default runtime options for Notebook actions (outputBucket, runtimeTemplateName, repositorySnapshotDestination). | | defaultIcebergConfig | object | — | Project-level defaults for BigQuery Apache Iceberg tables (bucketName, tableFolderRoot, tableFolderSubpath, connection). | | extension | object | — | External compilation extension package configuration (name, compilationMode, options). | | lineage | object | — | Knowledge Catalog Data Lineage settings (lineage: { enabled: true }). See Data Lineage guide. | | defaultLakehouseConfig | object | — | Project-level defaults for 4-part Lakehouse targets (catalog, namespace). |


SQLX Syntax Reference

A .sqlx file combines a config {} metadata block, optional JavaScript (js {}) and hook blocks, and a SQL body with ${...} JavaScript template interpolation.

SQLX Code Blocks

Block Supported Action Types Description
config { ... } All .sqlx types Declares the action type (defaults to "operations" if omitted), target overrides, dependencies, assertions, and BigQuery options.
js { ... } All .sqlx types Local JavaScript variables and helper functions scoped to the current .sqlx file.
pre_operations { ... } table, incremental, view SQL statements executed before the main table/view query runs (multiple statements can be terminated with ; or split into separate compiled array entries with ---).
post_operations { ... } table, incremental, view SQL statements executed after the main table/view query completes (multiple statements can be terminated with ; or split into separate compiled array entries with ---).
incremental_where { ... } incremental Optional SQL predicate that sets Table.where to filter rows during incremental updates (SELECT * FROM (<query>) AS subquery WHERE <predicate>). Not supported with compilation_mode: "jit".
input "<target>" { ... } test Mock SQL query replacing a referenced upstream dependency (ref("<target>")) during a unit test.
--- operations, pre_operations, post_operations Statement separator for multi-statement SQL blocks.

SQLX Context Functions

Each action maps to an action target (database.schema.name) in BigQuery. By default, name is the action’s filename (without extension), schema is defaultDataset (or defaultAssertionDataset for assertions), and database is defaultProject from workflow_settings.yaml.

Inside the SQL body and operation blocks of a .sqlx file (or via the ctx argument in JavaScript API callbacks), the following context methods are available:

| Function | Signature | Description | |—|—|—| | ref | ref(name) / ref(schema, name) / ref(database, schema, name) / ref({ database?, schema?, name, includeDependentAssertions? }) | Resolves an action target to its backtick-quoted SQL identifier and registers it as an upstream dependency. | | resolve | resolve(name) / resolve(schema, name) / resolve(database, schema, name) / resolve(targetObj) | Resolves an action target to its backtick-quoted SQL identifier without adding it as a dependency. | | self | self() | Returns the fully qualified, backtick-quoted SQL identifier of the current action’s target. | | name | name() | Returns the finalized table/action name of the current action (including namePrefix if set). | | schema | schema() | Returns the finalized dataset (schema) name of the current action (including datasetSuffix if set). | | database | database() | Returns the finalized Google Cloud project (database) of the current action (including projectSuffix if set). | | when | when(condition, trueCase, falseCase = "") | Returns trueCase when condition is truthy and falseCase otherwise. | | incremental | incremental() | Returns true when compiling the incremental update query of an incremental table, or false when compiling its initial/full-refresh query (table, incremental, view, dataPreparation). |


Action Types Reference

Full field definitions for all action configurations are documented in the Dataform Configs Reference.

table

Creates or replaces a BigQuery table (CREATE OR REPLACE TABLE). See TableConfig.

-- definitions/marts/orders_by_day.sqlx
config {
  type: "table",
  schema: "marts",
  description: "Daily aggregated order metrics",
  columns: {
    order_date: "UTC date of the order",
    total_orders: "Count of completed orders"
  },
  bigquery: {
    partitionBy: "order_date",
    clusterBy: ["country_code"],
    labels: {
      domain: "commerce"
    }
  },
  assertions: {
    uniqueKey: ["order_date", "country_code"],
    nonNull: ["order_date"]
  },
  tags: ["daily"]
}

SELECT
  DATE(order_timestamp) AS order_date,
  country_code,
  COUNT(*) AS total_orders
FROM ${ref("stg_orders")}
GROUP BY 1, 2

Key table config properties:


incremental

Builds a table from scratch on first run (or --full-refresh), and inserts or merges new rows on subsequent runs when the target table already exists. See IncrementalTableConfig.

-- definitions/marts/events_incremental.sqlx
config {
  type: "incremental",
  uniqueKey: ["event_id"],
  bigquery: {
    partitionBy: "DATE(event_timestamp)",
    updatePartitionFilter: "event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)"
  },
  protected: false
}

SELECT
  event_id,
  event_timestamp,
  user_id
FROM ${ref("raw_events")}
${when(incremental(), `WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM ${self()})`)}

Incremental-specific config properties (in addition to table properties):


view

Creates a virtual view (CREATE OR REPLACE VIEW) or a BigQuery materialized view (materialized: true). See ViewConfig.

-- definitions/staging/stg_orders.sqlx
config {
  type: "view",
  materialized: true,
  bigquery: {
    partitionBy: "DATE(order_timestamp)",
    clusterBy: ["customer_id"]
  }
}

SELECT
  order_id,
  customer_id,
  order_timestamp
FROM ${ref("raw_orders")}

operations

Executes arbitrary SQL statements (DDL, DML, stored procedures, GRANT/REVOKE). Multiple statements are separated by ---. See OperationConfig.

-- definitions/ops/cleanup_temp_tables.sqlx
config {
  type: "operations",
  hasOutput: false,
  dependencies: ["orders_by_day"]
}

DELETE FROM `my-gcp-project.staging.scratch_events`
WHERE created_at < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)

When hasOutput: true is set, the operation registers a referenceable target at database.schema.name that downstream actions can reference with ${ref("<name>")} or ${self()}.


assertion and Built-in Assertions

An assertion is a data quality query that fails if it returns one or more rows. Assertions materialize as views in defaultAssertionDataset (dataform init sets "dataform_assertions"). See AssertionConfig.

-- definitions/assertions/assert_positive_order_amount.sqlx
config {
  type: "assertion",
  tags: ["critical"]
}

SELECT order_id, amount
FROM ${ref("stg_orders")}
WHERE amount <= 0

You can also configure built-in assertions directly inside a table, incremental, or view config block:

config {
  type: "table",
  assertions: {
    uniqueKey: ["order_id"],
    nonNull: ["order_id", "customer_id"],
    rowConditions: [
      "amount >= 0",
      "order_timestamp <= CURRENT_TIMESTAMP()"
    ]
  }
}

declaration

Declares an external BigQuery table or view (not managed by Dataform) so it can be referenced via ${ref("<name>")}, documented, and shown in the dependency graph. See DeclarationConfig.

-- definitions/sources/raw_orders.sqlx
config {
  type: "declaration",
  database: "my-raw-data-project",
  schema: "stripe",
  name: "raw_orders",
  description: "Raw Stripe orders landing table"
}

test (Unit Tests)

Validates the SQL logic of a target dataset (table, view, or incremental action) by replacing its upstream ref() inputs with static mock queries (input "<ref_name>" { ... }) and comparing the output against an expected query. Executed via dataform test or dataform run --run-tests.

-- definitions/tests/orders_by_day_test.sqlx
config {
  type: "test",
  dataset: "orders_by_day"
}

input "stg_orders" {
  SELECT TIMESTAMP("2026-01-01 10:00:00 UTC") AS order_timestamp, "PL" AS country_code
  UNION ALL
  SELECT TIMESTAMP("2026-01-01 12:00:00 UTC") AS order_timestamp, "PL" AS country_code
}

SELECT DATE("2026-01-01") AS order_date, "PL" AS country_code, 2 AS total_orders

notebook

Configures a Jupyter Notebook (.ipynb) action for execution in Google Cloud Dataform (via Vertex AI Colab Enterprise). See NotebookConfig.

# definitions/actions.yaml
actions:
  - notebook:
      name: train_churn_model
      filename: notebooks/train_churn_model.ipynb
      dependencyTargets:
        - name: orders_by_day

dataPreparation

Configures a BigQuery Data Preparation action from a .dp.yaml / .yaml definition (via actions.yaml) or a .dp.sqlx / .sqlx file (type: "dataPreparation"). See DataPreparationConfig.


propertyGraph (graph.yaml)

Defines a BigQuery Property Graph (CREATE OR REPLACE PROPERTY GRAPH) from a definitions/**/graph.yaml file (at most one graph.yaml per project) following the Google Semantic Ontology schema (version: 0.2.0.dev0/google). See the BigQuery Property Graphs guide for full syntax and examples.


actions.yaml Action Configs

Plain .sql and .ipynb files can be registered as Dataform actions via definitions/**/actions.yaml files validated against dataform.ActionConfigs:

# definitions/actions.yaml
actions:
  - declaration:
      name: raw_users
      project: my-raw-project
      dataset: raw
  - view:
      name: stg_users
      filename: staging/stg_users.sql
      dependencyTargets:
        - name: raw_users
  - table:
      name: dim_users
      filename: marts/dim_users.sql
      dependencyTargets:
        - name: stg_users
      partitionBy: created_date

JavaScript API (Session Globals)

Inside definitions/**/*.js files (and js {} blocks in .sqlx files), Dataform attaches the active Session instance to global.dataform and binds its action-builder methods as global functions:

| Global Function / Object | Signature | Description | |—|—|—| | publish | publish(name, queryOrConfig?) | Creates a table, view (default if type is omitted), or incremental action. Returns a Table, View, or IncrementalTable builder (.query(ctx => ...)). | | operate | operate(name, queriesOrConfig?) | Creates an operations action. Returns an Operation builder (.queries(ctx => ...)). | | assert | assert(name, queryOrConfig?) | Creates an assertion action. Returns an Assertion builder (.query(ctx => ...)). | | declare | declare(config) | Declares an external data source ({ database?, schema?, name, description?, columns? }). | | test | test(name) | Creates a unit test action (.dataset(target).input(refName, sql).expect(sql)). | | notebook | notebook(config) | Creates a Notebook action ({ name?, filename, location?, project?, dependencyTargets?, tags?, disabled?, description? }). | | getContents | getContents(relativeMdPath) | Reads a .md file inside the project directory (resolved relative to the caller file) and returns its contents as a string for description fields. | | jitData | jitData(key, value) | Registers structured data in CompiledGraph.jitData for Just-in-Time (JiT) compilation workflows. | | dataform.projectConfig | dataform.ProjectConfig | Read-only access to the resolved project configuration, including custom variables (dataform.projectConfig.vars). |

Example (definitions/generate_staging_views.js):

const sourceTables = ["orders", "customers", "products"];

sourceTables.forEach((tableName) => {
  declare({
    schema: "raw",
    name: `raw_${tableName}`,
  });

  publish(`stg_${tableName}`, {
    type: "view",
    schema: "staging",
    description: getContents(`./docs/stg_${tableName}.md`),
  }).query((ctx) => `SELECT * FROM ${ctx.ref("raw", `raw_${tableName}`)}`);
});

Package Exports and Bundled .proto Files

TypeScript / JavaScript Exports

When imported as an npm library (import * as dataformCore from "@dataform/core"), @dataform/core exports the following symbols (with TypeScript typings in bundle.d.ts):

Bundled Protocol Buffer Schemas

The published @dataform/core package includes the canonical .proto definitions at the package root: