What's New

Release notes for the @sqlanvil/cli and @sqlanvil/core packages. Full notes live on GitHub Releases.

1.29.0 — insert_overwrite on BigQuery, and upstream Dataform 3.0.62

Synced to upstream dataform 3.0.62. The user-facing parts:

• New incremental strategy: incrementalStrategy: "insert_overwrite" (BigQuery). Instead of upserting rows on a uniqueKey, an incremental run replaces every partition its output touches — the right shape when a run recomputes a date range rather than matching individual rows. Requires bigquery: { partitionBy }; incrementalPredicates scope it further. The default (merge) is unchanged, so nothing moves unless you ask for it.

  config {
    type: "incremental",
    incrementalStrategy: "insert_overwrite",
    bigquery: { partitionBy: "event_date" }
  }

  It is BigQuery-only, and says so at compile time. On Postgres, Supabase or MySQL the strategy is rejected with that reason rather than accepted and quietly executed as a merge — which would have been a run that succeeds and writes different rows than you asked for.

• preserveGovernanceControls in workflow_settings.yaml and on table configs: keeps existing BigQuery column governance controls (e.g. data policies) when a table is recreated. BigQuery only — no other warehouse has the concept.

• A skipped task now records why it was skipped — run timed out, run cancelled, cancelled before the action started — instead of just showing SKIPPED. When a long run is cut short, the run detail now distinguishes “never started” from “stopped in flight”.

• Trailing statement comments no longer break parsing (upstream #2108), and JiT assertion compilation accepts both result shapes (#2221).

sqlanvil --version now reports Dataform core 3.0.62.

1.28.0 — Migration converts far more of a BigQuery project, and reports the rest as a to-do list

Migration is now three phases, not two: convert → introspect → fix. The middle phase is why — a converted declaration has no column list until introspect fills one in, and several BigQuery constructs cannot be rewritten without knowing the columns. Everything below came out of porting a real 880-declaration Dataform project.

• New command: sqlanvil migrate-fix. Run it after scripts/introspect_all.sh, in the converted project. It rewrites what only the introspected schema makes possible: SELECT * EXCEPT (a, b) expands to the explicit column list, and GROUP BY ALL becomes positional ordinals. Sites it can’t resolve — a star over a source that was never introspected — are listed in the report rather than half-rewritten. --dry-run shows the changes without writing.
• The converter rewrites BigQuery calls from their arguments, not by pattern: SAFE_CAST becomes a pg_input_is_valid guard that preserves the “invalid input yields NULL” contract (a plain CAST would abort the query on the first bad row — the whole reason the original used SAFE_CAST), plus SAFE_DIVIDE, SPLIT, DATE_DIFF, OFFSET/ORDINAL subscripts and the BigQuery type names. Nested calls converge.
• Lexical differences are handled where they change meaning. BigQuery treats backticks as identifiers and double quotes as strings; PostgreSQL is the reverse, so a mechanical copy silently turns column references into string literals. Also raw strings (r'…'), # comments, and EXTRACT(DAYOFWEEK) (1-based Sunday vs PostgreSQL’s 0-based dow).
• COLLATE(x, '') and NOT ENFORCED constraints. The collation wrapper is stripped (it’s BigQuery’s no-op default). NOT ENFORCED keys — advisory metadata BigQuery never checks — are rewritten into the PostgreSQL form that would work and left commented: the declared model stays visible, and enforcing integrity a project has never had is a deliberate, per-table decision rather than a silent one.
• ARRAY(SELECT AS STRUCT …) gets a recommendation and the SQL to apply it, chosen per site from how the column is actually read across the project — collapse it when every read is an UNNEST (the array is a round trip), a child table when a reader does more, jsonb as the fallback it should be rather than the default. Not rewritten automatically: it restructures the data model, so it belongs in front of a person.
• migration-report.md is now a to-do list, grouped by class and split by who can finish it — mechanical items an agent or a sed can apply, versus the ones needing a decision about the business (they are ranked with changes-meaning above compile errors, because a query that still runs and returns different numbers is the more expensive failure). It is written for a coding agent to work through with you, and updated by all three phases rather than only the first.
• Behaviour change: extracted and introspected column names fold to lower case. PostgreSQL folds unquoted identifiers to lower case and has no case-insensitive identifier mode, so carrying Email across from BigQuery meant every reference had to be written "Email" forever and every reference the original SQL spelled email stopped resolving — at run time, not compile time. Folding once, where the column is materialized, removes the class. Source columns differing only in case now fail loudly, naming both, instead of one silently overwriting the other. If you already have extracted tables with mixed-case columns, re-run the extract; existing quoted references to the old spelling will need updating.

1.27.2 — Large CLI outputs no longer truncate; query --json handles counts

• Piped output over 64 KB arrived truncated. The CLI exited before the OS accepted all of its stdout, silently chopping big outputs mid-stream when piped (redirecting to a file was unaffected) — found when a hosted run-detail capture parsed sqlanvil query --json on a 1,050-action project and received exactly one pipe buffer. Stdio now flushes before exit.
• query --json / inspect --json no longer crash on count(*)/sum(). DuckDB returns int64 aggregates as BigInt, which the JSON serializer rejected (“Do not know how to serialize a BigInt”); they now serialize as numbers (decimal strings beyond 2⁵³).

1.27.1 — Runner-extract hardened for real project scale

Three fixes from the first full hosted run of a large migrated project (206 extracted sources against a Supabase warehouse):

• NUMERIC values load correctly. BigQuery returns NUMERIC/BIGNUMERIC as decimal objects; the extract serialized them as quoted literals, which Postgres rejected with invalid input syntax for type numeric: ""55"". They now load losslessly as numerics.
• Extracts stream instead of buffering. A runner-extract source is now read page-by-page and batch-inserted with backpressure — memory stays bounded no matter how large the source (previously a big table could buffer up to 512 MB per extract and exhaust the runner’s heap). Row/byte caps unchanged.
• Connection use is capped. Extracts are DAG roots, so a full run used to start all of them simultaneously — each with its own connection pool, exhausting Supabase’s session pooler (max clients reached). At most 4 extracts now run concurrently, each on a single connection.

1.27.0 — Declarations are inert until referenced

• Unreferenced source declarations no longer run — or fail — anything. In Dataform, a declaration nothing references is inert; SQLAnvil now matches. Previously every declaration on a runner-extract connection became an extract a full sqlanvil run would materialize — copying source tables no model reads, and failing outright when a declaration deliberately points at a table that doesn’t exist yet (a common pattern: pre-declare the source system’s whole catalog, pull tables as needed). Extracts with no downstream consumers are now pruned from the compiled graph; add a ref() later and the extract is back on the next compile. A referenced declaration whose source is missing still fails at run time — exactly like a hand-written select * from bad_project.bad_table. Found migrating a production project where only 206 of 880 declared sources are actually consumed.
• scripts/introspect_all.sh finishes the whole pass. The generated scaffold script no longer aborts on the first failed introspect (set -e); failures — typically stale declarations for dropped tables — are collected and summarized at the end, so one bad declaration can’t block the other 879.

1.26.2 — introspect on BigQuery needs no credentials file

• BigQuery connections now work with gcloud ADC alone. sqlanvil introspect (and the generated scripts/introspect_all.sh in migrated projects) no longer demands a .df-credentials.json when the connection is BigQuery — with no connections.<name> entry (or no file at all), the client falls back to Application Default Credentials, matching Dataform and the migration guide’s “a machine with BigQuery read access” promise. Postgres and MySQL source connections still require the file — they carry real secrets (host/user/password).

1.26.1 — One script scaffolds every declaration’s columnTypes

• migrate-dataform now generates scripts/introspect_all.sh — the concrete sqlanvil introspect command for every converted declaration (the report previously showed only per-connection templates). Run it on a machine with BigQuery read access (gcloud auth application-default login), commit what it writes, and the extract layer of a migrated project is scaffolded in one pass — found migrating a production project with 881 declarations.

1.26.0 — Migrate from Dataform: auth like Dataform, test before production

• Converted BigQuery projects authenticate the way Dataform does — out of the box. migrate-dataform --target-warehouse bigquery now generates the secretless ADC-mode .df-credentials.json (project + location only, gitignored; your actual credentials are still never copied): anyone with gcloud auth application-default login can run sqlanvil validate immediately, exactly like a local dataform run. For CI and production, see the new Running in production mapping (attached service account = ADC; keyless WIF accessToken; per-workflow environments).
• First runs can’t touch production. The converter scaffolds a ready-made test environment — sqlanvil run . --environment test writes to <dataset>_test datasets, leaving production untouched until you deliberately run without it. Prefer a separate GCP project for tests? One field: environments.test.defaultDatabase.
• init --interactive asks before anything runs. The BigQuery paths gained an auth step — detect gcloud ADC (offering to launch gcloud auth application-default login right there), or point at a service-account key file, or defer — and the convert path a test-target step (dataset suffix, separate project, or none). The next-steps it prints follow the same safe sequence: compile → validate (read-only) → test run → production.

1.25.2 — validate understands self-references

• Self-referencing models now validate correctly. A query that reads its own table — the append/UNION-history pattern, or an incremental’s MAX(...)-watermark subquery — used to fail with “Not found” against the validation shadow namespace (the model’s own stub can’t exist before it validates). ${self()} references in queries now resolve to the production relation, matching what a real run reads; validation stays read-only.
• Pre/post-ops are dry-run in run order, against the fresh stub (BigQuery). The ALTER TABLE ${self()} ADD PRIMARY KEY … NOT ENFORCED idiom from the metadata guide used to false-fail; each op is now dry-run individually after the model’s shadow stub is created — a constraint-free relation, exactly like the freshly-built table a real run ALTERs. A failing op fails the model and blocks its dependents. On Postgres/MySQL, DDL ops are no longer fed to EXPLAIN (which always rejected them) — they’re simply not validated there.
• Proven on a 1,500-file production BigQuery project: 83 more models PASS and 52 fewer are blocked, with the remaining failures reduced to genuine issues.

1.25.1 — Clear errors for invalid config values (instead of a compile crash)

• Config type mistakes now fail loudly, per file. An object or array in a scalar config field — the classic case is dbt-style partitioning, partitionBy: { field, dataType, granularity }, where Dataform/SQLAnvil syntax is a SQL string — used to crash the entire compile deep in serialization (ERR_INVALID_ARG_TYPE). It now produces a normal compilation error naming the file, the property, and the expected type. Notably, Dataform silently coerces the same mistake to an empty string (that table was never actually partitioned) — found while migrating a 1,500-file production BigQuery project, where the new error caught a real, silent misconfiguration. Valid configs are unaffected: enum names as strings (fileFormat: "PARQUET"), coercible primitives, and map/array shapes validate exactly as before.

1.25.0 — Generated projects teach agents the metadata contract

• The scaffolded AGENTS.md now covers metadata for AI/BI — every project generated by init, migrate-dataform, or the Cloud wizard tells coding agents to treat description and columns: {} as part of the model, not decoration: they persist into the warehouse catalog on every run (COMMENT ON for Postgres/Supabase; table and incremental-table comments on MySQL/MariaDB — views can’t carry them; table + nested column descriptions on BigQuery), so anything reading the warehouse — BI tools, other agents — sees them. The guidance also has agents name join targets in column descriptions, add PK/FK constraints via post_operations (NOT ENFORCED on BigQuery; wrapped in ${when(!incremental(), …)} so one-time DDL doesn’t re-run on appends), and use uniqueKey/uniqueKeys correctly (they’re mutually exclusive).
• npm page housekeeping — the @sqlanvil/cli and @sqlanvil/core pages now link Homepage to sqlanvil.com with explicit Repository/Issues links to GitHub, and the README’s NOTICE/LICENSE/AGENTS.md links no longer 404 on npm (they were relative links npm can’t resolve).

1.24.0 — Keep BigQuery when leaving Dataform; every project ships agent guidance

• migrate-dataform --target-warehouse bigquery — the tooling swap: “I don’t want to use Dataform anymore — convert the code and keep running against the warehouse I already have in BigQuery.” SQL bodies, bigquery: {} config blocks, and declarations pass through untouched (BigQuery is a first-class sqlanvil warehouse, so nothing needs translating); defaultProject/defaultLocation carry through, dataset casing is preserved, and no Supabase account is involved anywhere. Only the settings file converts (sqlanvilCoreVersion:) and the compile global renames. The default (supabase, or postgres) still moves the warehouse as before. The init --interactive convert path now asks which you want — keep-BigQuery is its default, and that answer skips the Postgres credentials Q&A in favor of gcloud ADC guidance. sqlanvil validate (BigQuery dry-run) is the completion check.
• Every generated project now includes AGENTS.md — a warehouse-tailored guide in the cross-agent AGENTS.md standard (read natively by Codex, Cursor, Antigravity, Gemini CLI, and 30+ tools), plus a one-line CLAUDE.md bridge for Claude Code. Written by init (all modes, including --bare), by migrate-dataform (with a converted-project addendum pointing at the migration report), and by the Cloud wizard scaffold. Point any coding agent at your repo and it knows the dialect. Existing AGENTS.md/CLAUDE.md files are never overwritten.

1.23.0 — Guided setup: init --interactive (and init --bare)

• sqlanvil init --interactive — a guided Q&A for starting a project. First question: start fresh, or convert an existing Dataform project?
  • The fresh path walks warehouse choice, project directory, default schema, whether to include the sample project (and its cross-warehouse BigQuery source), then a credentials Q&A that writes the gitignored .df-credentials.json for you — Supabase defaults to the Session-pooler shape and warns if you paste the IPv6-only direct db.<ref> host. It ends with your numbered next steps (compile, validate).
  • The convert path runs migrate-dataform (source stays read-only), then offers the same credentials Q&A and points you at the migration report.
  • Explicit flag only — plain sqlanvil init is unchanged for scripts and CI, and --interactive fails fast with guidance when there’s no terminal to talk to.
• sqlanvil init --bare — scaffold the directory structure and config only, skipping the sample project files. For when you’re about to write (or generate) your own models.

1.22.0 — Migrate from Dataform

• sqlanvil migrate-dataform <srcDir> <outDir> — convert a Dataform/BigQuery project into a SQLAnvil-on-Postgres project. The source directory is read-only; the output gets a converted project plus migration-report.{md,json}. Declarations stay in BigQuery as named runner-extract connections (one per source GCP project); target SQL gets safe lexical rewrites plus inline SQLANVIL-MIGRATE: markers on everything needing dialect review; credentials in the source are never copied. Proven against a 1,404-file production project: 745 declarations → 6 connections, zero compile errors on the converted output.
• Runner-extract declarations: schema: now does the whole job. A declared schema: overrides the connection’s dataset (so one connection per source project serves many datasets) and names the Postgres schema the extract materializes into — ods.zip_code stays ods.zip_code, keeping schema-qualified ref()s working after a migration. Declarations without a schema: keep the previous <connection>_ext placement.
• Empty columnTypes now compiles on runner-extract declarations (introspect them incrementally); the extract fails at run time — before touching or billing the source — with the exact sqlanvil introspect command to fix it. FDW-mode declarations still require columnTypes at compile (a foreign table can’t exist without its columns).

1.21.0 — A real starter project from sqlanvil init

• sqlanvil init now generates a working sample project in the shape real ones take: source declarations (app_orders — a table your app already has — plus Google’s public ZIP data from BigQuery), staging views over them in intermediate/, output tables built from the staging layer (daily_sales, product_revenue, and orders_by_region joining both sources), and a business-rule assertion. compile is green out of the box on every warehouse; validate pinpoints exactly which sources still need wiring before run.
  • On Postgres/Supabase the BigQuery source rides a bigquery_public connection in mode: runner-extract — no Vault secret, no wrappers extension. On BigQuery it’s a native declaration; MySQL skips it (cross-warehouse connections need a Postgres-family warehouse).
  • The Supabase credentials template now points at the Session pooler (aws-1-<region>.pooler.supabase.com, user postgres.<project-ref>) — the direct db.<ref>.supabase.co host is IPv6-only and unreachable from most networks.
  • Postgres/Supabase projects default defaultDataset to public; scaffold directories survive the first commit (.gitkeep); --warehouse mysql documented in init help.
• Selective runs no longer execute unrelated extracts. run --actions/--tags previously ran every runner-extract source in the project regardless of selection — re-reading (and re-billing) sources nothing selected depended on, and failing runs that never touched them. Extracts now prune like every other action; --include-deps still pulls one in when a selected model reads it, and unselective runs are unchanged.

1.20.1 — Relative import/export paths are project-relative

• Fix: a relative local location: on imports/exports now resolves against the project directory, not the CLI’s own working directory. Previously sqlanvil run <projectDir> from anywhere else couldn’t find files a script action had just staged (scripts run with cwd = project dir); the two now always agree. Absolute paths and s3:///gs:// URIs are unaffected.

1.20.0 — Python script actions

• python: script actions — run Python file-staging and glue steps as first-class DAG nodes, declared in actions.yaml against a plain .py file. The ingestion slot Dataform/dbt leave outside the pipeline: download → import → staging → marts in one sqlanvil run, ordered by dependencies and covered by run history.
  • Execution-time only — compile never runs Python (compiles stay hermetic and deterministic). Compile checks config shape, file existence, and that pythonVersion is a well-formed specifier.
  • No warehouse credentials injected — scripts stage files; type: "import" is the loading boundary.
  • You own the environment; SQLAnvil validates it — sqlanvil validate checks the interpreter against pythonVersion, requirements: against the installed packages (offline, nothing is installed), and the script’s syntax — without executing it. Failures block dependents, like models.
  • Contract: cwd = project dir, SA_VARS (your vars as JSON) + SA_ACTION_NAME env, optional venv: interpreter, args:, exit 0 = success, 30-min default timeout (timeoutMillis overrides).
  • Language-neutral underneath (script: with a language field — python: is sugar), so future runtimes are additive.
  • The mailing_list example’s OpenAddresses ingestion is now fully in-DAG: a python: action stages the CSV (bundled sample, or a real openaddresses.io download via the oa_source_url var) and the import depends on it.
  • SQLAnvil Cloud: hosted runs reject script actions at compile time (like local file paths) — user code doesn’t execute on the shared runner. Local CLI and your own CI both run them today.

1.19.0 — The mysql: {} block is feature-complete

• FULLTEXT / SPATIAL indexes — indexes: [{ columns: ["title"], type: "fulltext" }] (mutually exclusive with unique; a spatial index needs a NOT NULL SRID geometry column, which usually means an ALTER TABLE … MODIFY in post_operations first).
• Index prefix lengths — a column may carry MySQL’s own prefix syntax, "body(50)" → `body`(50) (required to index TEXT/BLOB columns).
• rowFormat table option — emitted as ROW_FORMAT= (e.g. DYNAMIC, COMPRESSED).
• Fix: the mysql: {} block on a materialized view (type: "view", materialized: true) now compiles through — earlier versions silently ignored it there, despite the docs showing it.
• With these, the mysql: {} block covers table options, all index forms, and partitioning — closing issue #35.

1.18.0 — MySQL/MariaDB as a read-only source

• MySQL/MariaDB cross-warehouse sources — a platform: mysql connection can now feed a Postgres/Supabase warehouse. Declare a source with connection: and SQLAnvil reads database.table at run time and materializes it as a plain ref()-able table — the same runner-extract mechanism as keyless BigQuery sources (capped at 1M rows / 512 MB). MySQL sources are runner-extract only (there is no Postgres FDW for MySQL); an explicit mode: fdw is a compile-time error. Source credentials (host, port, user, password) live in .df-credentials.json’s connections map. Validated against MySQL 8 and MariaDB 11.
• sqlanvil introspect maps MySQL types to Postgres types — scaffolded columnTypes from a MySQL source now arrive as warehouse-valid types (datetime → timestamp, double → double precision, json → jsonb, blobs → bytea, …), since they define the extract table’s columns in the warehouse.
• Fix: re-running an extract over an existing plain table no longer fails on the leftover drop foreign table (drops are now relation-kind-aware; this affected BigQuery extracts too).
• Fix: an explicit schema: on a connection’d declaration was silently ignored — both the FDW foreign-table bridge and extracts now honor it as the source schema/database.

1.17.1 — run --graph works from any directory

• Fix: run --graph no longer requires the working directory to contain a workflow_settings.yaml — with a stored graph, the directory is only used for credentials and artifacts, so a bare directory works (as the 1.17.0 notes intended).

1.17.0 — run --graph (execute a stored compiled graph)

• sqlanvil run --graph <file> — run the JSON emitted by compile --json directly, without compiling the project. What executes is exactly what was compiled — environment overrides are baked in at compile time — which makes a stored graph a true release artifact: compile once, run the same graph unchanged later or elsewhere. Selection flags (--actions, --tags, --include-deps, --include-dependents), --full-refresh, and BigQuery --dry-run all apply to the loaded graph.
• Guards: --environment with --graph is rejected (compile with --environment instead, and pass --credentials explicitly); --dry-run on Postgres/Supabase/MySQL is rejected (validate needs the project source). The CLI warns when the graph was compiled by a different core major.minor.
• This powers pinned releases in SQLAnvil Cloud: scheduled workflow runs can execute the exact stored snapshot a release captured.

1.16.0 — Synced to upstream Dataform 3.0.61

• BigQuery column types: BIGNUMERIC, JSON, and INTERVAL source columns now map correctly (to numeric/string) when reading table schemas.
• getContents(path) — read a file’s raw contents inside a compilation (e.g. inline a .sql or Markdown snippet); paths are resolved relative to the calling file and confined to the project directory.
• Assertion metadata — assertions accept a metadata block, carried through to the action descriptor.
• actions.yaml guard — referencing a .sqlx file from actions.yaml now raises a clear error; .sqlx files are compiled directly from definitions/.
• Internal: the build migrated to Bazel 7.3.2 + bzlmod (no user-facing change).

1.15.0 — runner-extract connection mode (keyless cross-warehouse sources)

• mode: runner-extract on a BigQuery source connection — an alternative to the live Foreign Data Wrapper. Instead of a foreign table, SQLAnvil reads the source at run time and materializes the rows into a plain table (ref()-able just like the FDW bridge). It needs no Vault secret and no wrappers/postgis extensions, so it works on bare/ephemeral databases where an FDW can’t be provisioned. Source auth (a short-lived accessToken, a key, or ADC) goes in .df-credentials.json’s connections map. Keep the default fdw mode when you want a live foreign table on a persistent warehouse. See the Foreign Data Wrappers guide.

1.14.0 — Keyless BigQuery via access token

• accessToken on BigQuery credentials — a new optional field in .df-credentials.json lets BigQuery authenticate with a short-lived OAuth2 access token instead of a JSON key. Precedence is accessToken → JSON credentials → Application Default Credentials, so existing setups are unaffected. This enables keyless runs: a caller that already runs inside GCP (e.g. SQLAnvil Cloud’s runner) can mint a token by impersonating a service account and hand it to the CLI — no key is ever written. Tokens are short-lived (≈1h), suited to a single run.

1.13.0 — Cross-project BigQuery FDW (billingProject)

• billingProject on a BigQuery named connection lets you read a dataset you can read but not bill — e.g. bigquery-public-data — by billing the FDW query jobs to your own GCP project. Set project to where the data lives and billingProject to your project; SQLAnvil bills yours and reads the source via a full-FQN subquery. Without it, reading a public/other dataset failed with bigquery.jobs.create permission denied. The SA needs bigquery.jobUser on the billing project. See the Foreign Data Wrappers guide.

1.12.0 — File imports (type: "import")

• type: "import" loads a Parquet/CSV/JSON file into a table in the warehouse — the symmetric inverse of type: "export". The loaded table is ref()-able by downstream models. On BigQuery it compiles to native LOAD DATA OVERWRITE|INTO … FROM FILES(…) (gs:// only); on Postgres/Supabase the CLI loads it via the bundled DuckDB bridge (read the file, write into the warehouse) — filling the gap where managed Postgres/Supabase can’t read Parquet itself. overwrite defaults true (replace), false appends. MySQL is not supported yet. See the Imports guide.

1.11.0 — MySQL partitioning + introspect

• MySQL/MariaDB native partitioning via the mysql: {} block: partition: { kind, expression, partitions, count } — RANGE/LIST (explicit child partitions, e.g. values less than (2024) / values in (...)) and HASH/KEY (PARTITIONS <n>). See the MySQL guide.
• sqlanvil introspect now supports MySQL — scaffold a declaration from an existing MySQL/MariaDB table’s columns, types, and comments (platform: mysql connection).

1.10.0 — Queryable artifacts + catalog

• Queryable Parquet artifacts. compile writes a catalog and run writes run history to target/ as Parquet (via the bundled DuckDB) — portable and readable by DuckDB, pandas, agents, or external tables. Best-effort; --no-artifacts to disable.
• sqlanvil query "<sql>" runs SQL over the artifacts (views: actions, dependencies, columns, runs); sqlanvil inspect prints counts + latest-run summary + failures; sqlanvil docs generates a self-contained HTML catalog at target/docs/index.html. See the Artifacts & Catalog guide. (Add target/ to .gitignore.)

1.9.0 — sqlanvil validate

• New command: sqlanvil validate — catch broken SQL before you run it. It walks the project DAG in dependency order and validates every model against the warehouse planner (EXPLAIN on PostgreSQL/Supabase/MySQL, dry-run on BigQuery) without executing, using an isolated, auto-dropped shadow schema so whole-project validation works even before any model is built. Results are PASS / FAIL / BLOCK / SKIP with error locations; non-zero exit on failures, so it drops into CI. Works on all four warehouses. See the Validate guide.
• Fix: run --dry-run on PostgreSQL/Supabase/MySQL now validates instead of executing — previously it applied changes while claiming “no changes will be applied”. BigQuery keeps its native server-side dry-run.

1.8.3 — MySQL pre/post-ops + DuckDB security migration

• MySQL/MariaDB now runs pre_operations / post_operations (and the incremental variants). They were silently ignored before — a pre_operations/post_operations block compiled and ran but never executed. They now bracket every build path, matching BigQuery/Postgres. (#44)
• Security: the DuckDB binding behind Postgres/Supabase file exports moved from the classic duckdb package to the official @duckdb/node-api. The old package’s node-gyp → tar chain reported 5 high-severity npm audit advisories (no fix available); the new client audits clean. No behavior change. (#40)

1.8.2 — Upstream 3.0.60 sync + assertion metadata

• metadata on assertions: the assertion config block now accepts a metadata field (mirroring table/view), letting you attach an overview and arbitrary key/value extraProperties that land on the assertion’s action descriptor. (#2208)
• Synced to upstream Dataform core 3.0.60 (sqlanvil --version now reports Dataform core 3.0.60).
• Security: bumped vm2 3.11.3 → 3.11.4 and protobufjs 7.5.5 → 7.5.8.

1.8.1 — Multiline rowConditions fix

• Bug fix: a multiline rowConditions assertion entry no longer compiles to invalid SQL on BigQuery. The condition label is now escaped to a single-line string literal (BigQuery single-quoted literals can’t span lines), so conditions formatted across multiple lines for readability work. Postgres/Supabase and MySQL were unaffected. (#42)

1.8.0 — File exports (type: "export")

• type: "export" writes a query result to a Parquet/CSV/JSON file. On BigQuery it compiles to native EXPORT DATA (gs:// only); on Postgres/Supabase the CLI exports runner-side via DuckDB (COPY (SELECT … FROM postgres_query('pg', …)) TO …) to s3://, gs://, Supabase Storage, or a local path — filling the gap where managed Supabase can’t write files itself.
• Cloud credentials live in a new storage section of the gitignored .df-credentials.json. The DuckDB dependency ships with the CLI and loads only when an export runs. See the File Exports guide.

1.7.0 — Named environments

• --environment <name> (on compile/run/test) loads a named environment from an environments: block in workflow_settings.yaml — its schemaSuffix, vars, defaultDatabase/defaultLocation, and its own credentials file. Precedence: explicit CLI flag > environment > workflow_settings defaults (vars merge per-key). Secrets stay in gitignored per-env .df-credentials*.json files. See the Environments guide.

1.6.0 — MySQL mysql:{} block, comments, materialized views

• mysql: {} config block — secondary indexes + table options (engine/charset/collation) on table/incremental models.
• COMMENT metadata — description:/columns: apply as real MySQL table/column comments (full column-definition reconstruction so nothing is dropped), read back from information_schema.
• Materialized views — type: "view", materialized: true is emulated as a refreshed table snapshot (drop + CTAS each run).

1.5.0 — MySQL / MariaDB adapter

• New mysql warehouse adapter — one adapter for MySQL 8 and MariaDB 11, generating portable MySQL DDL/DML (CTAS tables, CREATE OR REPLACE VIEW, ON DUPLICATE KEY UPDATE incremental upserts). sqlanvil init --warehouse mysql. See the MySQL/MariaDB guide.

1.1.0 — Named connections + schema introspection

Read foreign sources declaratively, without hand-writing FDW boilerplate.

• Named connections: define connections: in workflow_settings.yaml and tag a declaration with connection: — SQLAnvil auto-generates the read-only FDW bridge (BigQuery or Postgres source) and the table becomes ref()-able. warehouse: can name a connection; non-secret connection defs live in workflow_settings.yaml, secrets in .df-credentials.json. See the Foreign Data Wrappers guide.
• sqlanvil introspect: reads a source table’s schema and writes the declaration with columnTypes (and descriptions) filled in, mapping source types to your warehouse dialect.
• sqlanvil init now defaults to --warehouse supabase (was bigquery), fitting the Postgres/Supabase-first focus. Pass --warehouse postgres|bigquery to override.

1.0.3 — Cross-warehouse Foreign Data Wrappers

First-class Foreign Data Wrapper support in the wrapper() action — query another warehouse in place and join it with your operational data.

• Provider presets: wrapper({ provider: "bigquery", server, serverOptions, credential, foreignTables }) emits a complete, correct BigQuery FDW bridge (enables the wrappers extension, registers the FDW idempotently, creates the server). Generic FDW remains reachable on any Postgres via explicit wrapper/handler/validator.
• Ref-able foreign tables: each foreignTables[] entry becomes a ref()-able table that depends on the server setup, so downstream models consume external data like any other source.
• Safe credentials: references a pre-existing Supabase Vault secret by id — the service-account key JSON never enters SQLAnvil. Postgres-first design; only the credential path is Supabase-specific.
• New example: supabase_bigquery_mailing_list — a proximity mailing list joining Supabase orders with Google’s bigquery-public-data ZIP geo data via a live FDW + PostGIS.

See the Foreign Data Wrappers guide.

1.0.2 — Packaging fixes

• The published npm packages now include README.md, LICENSE, and NOTICE, so the npm package pages render the README and the Apache-2.0 license ships in the artifact.

1.0.0 — First public release

• Open-source SQL workflow tool for BigQuery, PostgreSQL, and Supabase, forked from Dataform OSS and extended with first-class Postgres/Supabase support.
• Idiomatic Postgres DDL/DML (native partitioning, INSERT … ON CONFLICT upserts, btree/gin/gist/brin indexes, materialized views).
• Supabase-native action types: RLS policies, Realtime publications, pgvector indexes.
• Flat warehouse: config + gitignored .df-credentials.json connection.