Working with Snowflake, dbt, and Propel

This guide is for users syncing data from Snowflake to Propel while using dbt.

​

What Is dbt (data build tool)?

dbt is an open-source tool that transforms warehouse data using SQL. It enables building, testing, documenting, and orchestrating data transformation pipelines.

​

Propel and dbt

Propel uses Snowflake streams to capture changes from tables and sync data. These streams are tied to their source tables. If a table is dropped, its associated stream is deleted. Without an active stream, Propel cannot sync data from that table.

This section covers what you need to know when working with dbt and Propel.

​

Materialize models as incremental

When syncing Snowflake tables generated with dbt to Propel, use the incremental materialization strategy. Unlike table materializations, which drop and recreate the entire table on each run (breaking the underlying stream and Propel syncs), incremental materializations add new rows without rebuilding the table. This approach preserves the Snowflake stream that Propel relies on for data synchronization.

​

Avoid doing a full-refresh

A dbt full-refresh operation deletes the Snowflake tables and recreates them. This process breaks the Snowflake stream that Propel relies on for syncing data, as the stream is tied to the original table. When Propel detects that the stream is broken, it fails the sync.

To protect the models synced to Propel from accidental full-refreshes, add the following configuration to your dbt models:

​

Pro-tip: Use dbt post hooks to enable change tracking

You can use dbt post hooks to enable change tracking for your tables.

This is useful so you don’t have to manually enable change tracking for each table.

Replace <YOUR_TABLE_NAME> with the name of your Snowflake table.

​

Recovering from a failed sync

If a sync fails due to a broken or stale stream, the recovery process depends on your table engine:

1. For ReplacingMergeTree tables:

   • Click the “Re-sync” button in the Propel console.
   • This will recreate the stream, re-ingest the data, and deduplicate it.

2. For MergeTree tables:

   • Create a new Data Pool with the same configuration.
   • Migrate the metrics from the old Data Pool to the new one.
   • Delete the old Data Pool.

These steps ensure your data remains synchronized and accessible in Propel.