dataform

Stateless installs

<- Back to Home.

When dataformCoreVersion is set in workflow_settings.yaml, the CLI doesn’t use a node_modules directory in the project. Instead, each time it compiles the project, it:

  1. copies the project to a temporary directory,
  2. writes a package.json there that depends on @dataform/core at that version,
  3. runs npm i in the copy,
  4. compiles the copy, and
  5. deletes the copy.

The project itself must not contain package.json, package-lock.json or node_modules.

The copy uses ignore rules to skip potentially large directories, such as .venv or build output. This page describes what it includes.

Always copied

Ignore files can’t exclude any of these.

Always excluded

Outside the locations above, .git and node_modules directories are never copied. Ignore files can’t re-include them.

.gitignore and .dataformignore

Everything else is filtered by the .gitignore in the project root, then by the .dataformignore in the project root, if either exists.

With --verbose, the CLI prints which ignore files it applied.

When a skipped file is needed

A file outside definitions/ and includes/ that an ignore file excludes isn’t available during installation or compilation. For example:

Compilation or installation then fails, even though the file is in the project. If project code catches the failed load, it carries on, possibly with different behavior.

When the CLI can tell that a missing module was excluded from the copy, it prints a note alongside the error:

Note: 'queries' is in the project but not in the copy compiled for dataformCoreVersion, because .gitignore or .dataformignore excludes it. If module 'queries/example.sql' is there, re-include it in .dataformignore.

To make such a file available, re-include it in .dataformignore, or move it under definitions/ or includes/.