@dataform/core)@dataform/core is the open-source SQLX compiler, JavaScript workflow SDK, and Protocol Buffer schema package for Dataform. It compiles SQLX (.sqlx), JavaScript (.js), action configuration (actions.yaml), BigQuery Property Graph (graph.yaml), and Jupyter Notebook (.ipynb) definitions into a dependency-ordered execution graph (dataform.CompiledGraph) for BigQuery.
For command-line execution (dataform init, compile, test, run, format), see the Dataform CLI Reference (@dataform/cli) and the root repository overview.
workflow_settings.yaml)Session Globals).proto Filesworkflow_settings.yaml (Recommended)In both Dataform in Google Cloud and @dataform/cli, specify dataformCoreVersion in your project’s workflow_settings.yaml:
defaultProject: my-gcp-project
defaultLocation: US
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
dataformCoreVersion: <version>
When dataformCoreVersion is set in workflow_settings.yaml, @dataform/cli automatically installs that version of @dataform/core in an isolated temporary directory at compile time (see Stateless installs).
package.json / npmIf you manage @dataform/core and external JavaScript packages with package.json (omitting dataformCoreVersion from workflow_settings.yaml), or if you consume @dataform/core programmatically:
npm i @dataform/core
A standard Dataform project has the following layout:
my-dataform-project/
├── workflow_settings.yaml # Project-level settings and defaults
├── definitions/ # Action definitions (.sqlx, .js, actions.yaml, graph.yaml, .sql, .ipynb)
│ ├── sources.sqlx
│ ├── staging/
│ │ └── stg_orders.sqlx
│ └── marts/
│ └── fct_orders.sqlx
└── includes/ # Shared JavaScript modules and constants
└── constants.js
workflow_settings.yaml: Configures default project/dataset/location, custom variables (vars), target prefixes/suffixes, and runtime options.definitions/: Every .sqlx and .js file under definitions/ (and any subdirectory) is evaluated to register actions in the compilation session. definitions/**/actions.yaml files register YAML-configured actions, and definitions/**/graph.yaml defines a BigQuery Property Graph.includes/: Top-level includes/*.js files are automatically attached to the global scope by their file basename (for example, includes/constants.js is available as constants inside any .sqlx or .js file in definitions/). Nested files under includes/ can be imported via standard require("includes/...").Note on Project Execution & Security: Dataform compiles SQLX and JavaScript project code using Node.js’s native
node:vmcontext. While module resolution is scoped to the project directory,node:vmis not a security sandbox against untrusted code. Only compile and run Dataform projects from trusted sources.
workflow_settings.yaml)workflow_settings.yaml is validated against the dataform.WorkflowSettings schema.
| Field | Type | Default | Description |
|—|—|—|—|
| dataformCoreVersion | string | — | @dataform/core version to compile against (for example, 3.1.5). |
| defaultProject | string | — | Required. Default Google Cloud project ID (database) for compiled actions. |
| defaultDataset | string | — | Required. Default BigQuery dataset (schema) for tables, views, and operations. |
| defaultLocation | string | — | Required. Default BigQuery dataset location (for example, US, EU, us-central1). |
| defaultAssertionDataset | string | — | Required. Default BigQuery dataset (schema) for assertion views. |
| vars | map<string, string> | {} | Custom string key-value variables accessible via dataform.projectConfig.vars.<key>. |
| projectSuffix | string | — | Suffix appended to target Google Cloud project IDs (<project>_<projectSuffix>). |
| datasetSuffix | string | — | Suffix appended to target dataset names (<dataset>_<datasetSuffix>). |
| namePrefix | string | — | Prefix prepended to target action names (<namePrefix>_<name>). |
| builtinAssertionNamePrefix | string | — | Prefix prepended to auto-generated built-in assertion names. |
| disableAssertions | boolean | false | Marks all manual and built-in assertions as disabled: true in the compiled graph. |
| defaultReservation | string | — | Default BigQuery reservation for query execution (projects/<p>/locations/<l>/reservations/<r> or none). |
| includeTestsInCompiledGraph | boolean | false | Includes unit test actions (type: "test") in CompiledGraph.tests. |
| preserveGovernanceControls | boolean | false | Preserves existing BigQuery column governance controls (such as Data Policies) when recreating tables. |
| defaultJobLabels | map<string, string> | {} | Default labels applied to BigQuery jobs executed for actions in the project. |
| defaultNotebookRuntimeOptions | object | — | Default runtime options for Notebook actions (outputBucket, runtimeTemplateName, repositorySnapshotDestination). |
| defaultIcebergConfig | object | — | Project-level defaults for BigQuery Apache Iceberg tables (bucketName, tableFolderRoot, tableFolderSubpath, connection). |
| extension | object | — | External compilation extension package configuration (name, compilationMode, options). |
| lineage | object | — | Knowledge Catalog Data Lineage settings (lineage: { enabled: true }). See Data Lineage guide. |
| defaultLakehouseConfig | object | — | Project-level defaults for 4-part Lakehouse targets (catalog, namespace). |
A .sqlx file combines a config {} metadata block, optional JavaScript (js {}) and hook blocks, and a SQL body with ${...} JavaScript template interpolation.
| Block | Supported Action Types | Description |
|---|---|---|
config { ... } |
All .sqlx types |
Declares the action type (defaults to "operations" if omitted), target overrides, dependencies, assertions, and BigQuery options. |
js { ... } |
All .sqlx types |
Local JavaScript variables and helper functions scoped to the current .sqlx file. |
pre_operations { ... } |
table, incremental, view |
SQL statements executed before the main table/view query runs (multiple statements can be terminated with ; or split into separate compiled array entries with ---). |
post_operations { ... } |
table, incremental, view |
SQL statements executed after the main table/view query completes (multiple statements can be terminated with ; or split into separate compiled array entries with ---). |
incremental_where { ... } |
incremental |
Optional SQL predicate that sets Table.where to filter rows during incremental updates (SELECT * FROM (<query>) AS subquery WHERE <predicate>). Not supported with compilation_mode: "jit". |
input "<target>" { ... } |
test |
Mock SQL query replacing a referenced upstream dependency (ref("<target>")) during a unit test. |
--- |
operations, pre_operations, post_operations |
Statement separator for multi-statement SQL blocks. |
Each action maps to an action target (database.schema.name) in BigQuery. By default, name is the action’s filename (without extension), schema is defaultDataset (or defaultAssertionDataset for assertions), and database is defaultProject from workflow_settings.yaml.
Inside the SQL body and operation blocks of a .sqlx file (or via the ctx argument in JavaScript API callbacks), the following context methods are available:
| Function | Signature | Description |
|—|—|—|
| ref | ref(name) / ref(schema, name) / ref(database, schema, name) / ref({ database?, schema?, name, includeDependentAssertions? }) | Resolves an action target to its backtick-quoted SQL identifier and registers it as an upstream dependency. |
| resolve | resolve(name) / resolve(schema, name) / resolve(database, schema, name) / resolve(targetObj) | Resolves an action target to its backtick-quoted SQL identifier without adding it as a dependency. |
| self | self() | Returns the fully qualified, backtick-quoted SQL identifier of the current action’s target. |
| name | name() | Returns the finalized table/action name of the current action (including namePrefix if set). |
| schema | schema() | Returns the finalized dataset (schema) name of the current action (including datasetSuffix if set). |
| database | database() | Returns the finalized Google Cloud project (database) of the current action (including projectSuffix if set). |
| when | when(condition, trueCase, falseCase = "") | Returns trueCase when condition is truthy and falseCase otherwise. |
| incremental | incremental() | Returns true when compiling the incremental update query of an incremental table, or false when compiling its initial/full-refresh query (table, incremental, view, dataPreparation). |
Full field definitions for all action configurations are documented in the Dataform Configs Reference.
tableCreates or replaces a BigQuery table (CREATE OR REPLACE TABLE). See TableConfig.
-- definitions/marts/orders_by_day.sqlx
config {
type: "table",
schema: "marts",
description: "Daily aggregated order metrics",
columns: {
order_date: "UTC date of the order",
total_orders: "Count of completed orders"
},
bigquery: {
partitionBy: "order_date",
clusterBy: ["country_code"],
labels: {
domain: "commerce"
}
},
assertions: {
uniqueKey: ["order_date", "country_code"],
nonNull: ["order_date"]
},
tags: ["daily"]
}
SELECT
DATE(order_timestamp) AS order_date,
country_code,
COUNT(*) AS total_orders
FROM ${ref("stg_orders")}
GROUP BY 1, 2
Key table config properties:
name, schema (or dataset), database (or project), description, columns, tags, disabled, dependencies, dependOnDependencyAssertions, hermetic, metadata (overview, extraProperties).partitionBy, partitionExpirationDays, requirePartitionFilter, clusterBy, labels, additionalOptions, and iceberg (all accepted at top level or nested under bigquery: {}; for view, bigquery: {} accepts partitionBy, clusterBy, labels, and additionalOptions), plus top-level preserveGovernanceControls.assertions (uniqueKey, uniqueKeys, nonNull, rowConditions), preOperations, postOperations, reservation, jobLabels, lakehouse, compilation_mode ("aot" or "jit").incrementalBuilds a table from scratch on first run (or --full-refresh), and inserts or merges new rows on subsequent runs when the target table already exists. See IncrementalTableConfig.
-- definitions/marts/events_incremental.sqlx
config {
type: "incremental",
uniqueKey: ["event_id"],
bigquery: {
partitionBy: "DATE(event_timestamp)",
updatePartitionFilter: "event_timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)"
},
protected: false
}
SELECT
event_id,
event_timestamp,
user_id
FROM ${ref("raw_events")}
${when(incremental(), `WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM ${self()})`)}
Incremental-specific config properties (in addition to table properties):
uniqueKey: Column name(s) used to MERGE rows on incremental updates; if omitted, new rows are appended (INSERT).incrementalStrategy: "merge" (requires uniqueKey) or "insert_overwrite" (requires partitionBy) in SQLX and JavaScript (or "INCREMENTAL_STRATEGY_MERGE" / "INCREMENTAL_STRATEGY_INSERT_OVERWRITE" in actions.yaml).updatePartitionFilter / incrementalPredicates: Mutually exclusive SQL filter(s) restricting target partitions scanned during incremental updates (setting both is a compile error; prefer incrementalPredicates).onSchemaChange: "IGNORE" (default), "FAIL", "EXTEND", or "SYNCHRONIZE".protected (default true): Prevents the table from being dropped and rebuilt from scratch on --full-refresh. Set protected: false to allow full refreshes.viewCreates a virtual view (CREATE OR REPLACE VIEW) or a BigQuery materialized view (materialized: true). See ViewConfig.
-- definitions/staging/stg_orders.sqlx
config {
type: "view",
materialized: true,
bigquery: {
partitionBy: "DATE(order_timestamp)",
clusterBy: ["customer_id"]
}
}
SELECT
order_id,
customer_id,
order_timestamp
FROM ${ref("raw_orders")}
operationsExecutes arbitrary SQL statements (DDL, DML, stored procedures, GRANT/REVOKE). Multiple statements are separated by ---. See OperationConfig.
-- definitions/ops/cleanup_temp_tables.sqlx
config {
type: "operations",
hasOutput: false,
dependencies: ["orders_by_day"]
}
DELETE FROM `my-gcp-project.staging.scratch_events`
WHERE created_at < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
When hasOutput: true is set, the operation registers a referenceable target at database.schema.name that downstream actions can reference with ${ref("<name>")} or ${self()}.
assertion and Built-in AssertionsAn assertion is a data quality query that fails if it returns one or more rows. Assertions materialize as views in defaultAssertionDataset (dataform init sets "dataform_assertions"). See AssertionConfig.
-- definitions/assertions/assert_positive_order_amount.sqlx
config {
type: "assertion",
tags: ["critical"]
}
SELECT order_id, amount
FROM ${ref("stg_orders")}
WHERE amount <= 0
You can also configure built-in assertions directly inside a table, incremental, or view config block:
config {
type: "table",
assertions: {
uniqueKey: ["order_id"],
nonNull: ["order_id", "customer_id"],
rowConditions: [
"amount >= 0",
"order_timestamp <= CURRENT_TIMESTAMP()"
]
}
}
declarationDeclares an external BigQuery table or view (not managed by Dataform) so it can be referenced via ${ref("<name>")}, documented, and shown in the dependency graph. See DeclarationConfig.
-- definitions/sources/raw_orders.sqlx
config {
type: "declaration",
database: "my-raw-data-project",
schema: "stripe",
name: "raw_orders",
description: "Raw Stripe orders landing table"
}
test (Unit Tests)Validates the SQL logic of a target dataset (table, view, or incremental action) by replacing its upstream ref() inputs with static mock queries (input "<ref_name>" { ... }) and comparing the output against an expected query. Executed via dataform test or dataform run --run-tests.
-- definitions/tests/orders_by_day_test.sqlx
config {
type: "test",
dataset: "orders_by_day"
}
input "stg_orders" {
SELECT TIMESTAMP("2026-01-01 10:00:00 UTC") AS order_timestamp, "PL" AS country_code
UNION ALL
SELECT TIMESTAMP("2026-01-01 12:00:00 UTC") AS order_timestamp, "PL" AS country_code
}
SELECT DATE("2026-01-01") AS order_date, "PL" AS country_code, 2 AS total_orders
notebookConfigures a Jupyter Notebook (.ipynb) action for execution in Google Cloud Dataform (via Vertex AI Colab Enterprise). See NotebookConfig.
# definitions/actions.yaml
actions:
- notebook:
name: train_churn_model
filename: notebooks/train_churn_model.ipynb
dependencyTargets:
- name: orders_by_day
dataPreparationConfigures a BigQuery Data Preparation action from a .dp.yaml / .yaml definition (via actions.yaml) or a .dp.sqlx / .sqlx file (type: "dataPreparation"). See DataPreparationConfig.
propertyGraph (graph.yaml)Defines a BigQuery Property Graph (CREATE OR REPLACE PROPERTY GRAPH) from a definitions/**/graph.yaml file (at most one graph.yaml per project) following the Google Semantic Ontology schema (version: 0.2.0.dev0/google). See the BigQuery Property Graphs guide for full syntax and examples.
actions.yaml Action ConfigsPlain .sql and .ipynb files can be registered as Dataform actions via definitions/**/actions.yaml files validated against dataform.ActionConfigs:
# definitions/actions.yaml
actions:
- declaration:
name: raw_users
project: my-raw-project
dataset: raw
- view:
name: stg_users
filename: staging/stg_users.sql
dependencyTargets:
- name: raw_users
- table:
name: dim_users
filename: marts/dim_users.sql
dependencyTargets:
- name: stg_users
partitionBy: created_date
Session Globals)Inside definitions/**/*.js files (and js {} blocks in .sqlx files), Dataform attaches the active Session instance to global.dataform and binds its action-builder methods as global functions:
| Global Function / Object | Signature | Description |
|—|—|—|
| publish | publish(name, queryOrConfig?) | Creates a table, view (default if type is omitted), or incremental action. Returns a Table, View, or IncrementalTable builder (.query(ctx => ...)). |
| operate | operate(name, queriesOrConfig?) | Creates an operations action. Returns an Operation builder (.queries(ctx => ...)). |
| assert | assert(name, queryOrConfig?) | Creates an assertion action. Returns an Assertion builder (.query(ctx => ...)). |
| declare | declare(config) | Declares an external data source ({ database?, schema?, name, description?, columns? }). |
| test | test(name) | Creates a unit test action (.dataset(target).input(refName, sql).expect(sql)). |
| notebook | notebook(config) | Creates a Notebook action ({ name?, filename, location?, project?, dependencyTargets?, tags?, disabled?, description? }). |
| getContents | getContents(relativeMdPath) | Reads a .md file inside the project directory (resolved relative to the caller file) and returns its contents as a string for description fields. |
| jitData | jitData(key, value) | Registers structured data in CompiledGraph.jitData for Just-in-Time (JiT) compilation workflows. |
| dataform.projectConfig | dataform.ProjectConfig | Read-only access to the resolved project configuration, including custom variables (dataform.projectConfig.vars). |
Example (definitions/generate_staging_views.js):
const sourceTables = ["orders", "customers", "products"];
sourceTables.forEach((tableName) => {
declare({
schema: "raw",
name: `raw_${tableName}`,
});
publish(`stg_${tableName}`, {
type: "view",
schema: "staging",
description: getContents(`./docs/stg_${tableName}.md`),
}).query((ctx) => `SELECT * FROM ${ctx.ref("raw", `raw_${tableName}`)}`);
});
.proto FilesWhen imported as an npm library (import * as dataformCore from "@dataform/core"), @dataform/core exports the following symbols (with TypeScript typings in bundle.d.ts):
version (string): The installed @dataform/core semantic version string (for example, "3.1.5").session (Session): The singleton compilation Session instance.compiler ((code: string, path: string) => string): Transpiles .sqlx, .yaml/.yml, .sql, .ipynb, and .md source strings into executable JavaScript modules.main ((coreExecutionRequest: Uint8Array | string) => Uint8Array | string): Entry point invoked by the compilation VM sandbox; accepts an encoded dataform.CoreExecutionRequest proto and returns an encoded dataform.CoreExecutionResponse proto containing CompiledGraph.jitCompiler ((rpcCallback: RpcCallback) => IJitCompiler): Creates a Just-in-Time (JiT) action compiler bound to a database adapter RpcCallback.supportedFeatures (dataform.SupportedFeatures[]): Feature capability flags supported by this @dataform/core version (such as ARRAY_BUFFER_IPC).IDataformExtension (interface): TypeScript interface for compilation extensions.IJitCompiler (interface): TypeScript interface for Just-in-Time (JiT) action compilers.
The published @dataform/core package includes the canonical .proto definitions at the package root:
core.proto — CompiledGraph, Table, Operation, Assertion, Declaration, Test, Notebook, DataPreparation, PropertyGraph, Target, ProjectConfig, CoreExecutionRequest, CoreExecutionResponse.configs.proto — WorkflowSettings, ActionConfigs, ActionConfig (TableConfig, ViewConfig, IncrementalTableConfig, AssertionConfig, OperationConfig, DeclarationConfig, NotebookConfig, DataPreparationConfig).property_graph_config.proto — PropertyGraphConfig (SemanticModelConfig, GraphAiContextConfig, GraphDeploymentConfig, GraphEntityConfig, GraphRelationshipConfig, GraphMetricConfig, GraphCustomExtensionConfig).db_adapter.proto — Field, Fields, TableMetadata, ExecutionMetadata.jit.proto — DbAdapter service, JitCompilationRequest, JitCompilationResponse, JitTableResult, JitIncrementalTableResult, JitOperationResult, JitAssertionResult.extension.proto — Extension (ExtensionCompilationMode).configs.proto)Session Reference@dataform/cli)graph.yaml).dataformignore)