Skip to main content

Working with Propel and dbt

This guide explains the best practices using dbt with Propel.

It covers what dbt is, the different use cases when you should consider it, and how it works with Propel.

  1. What Is dbt?
  2. For What Use Cases Should I Consider dbt?
  3. Propel and dbt

Requirements​

What Is dbt (data build tool)?​

dbt is an open-source tool to transform data in your warehouse by building a transformation pipeline using SQL, testing your data, documenting it, and orchestrating the execution of your pipeline.

Overall, dbt can help you to improve the quality of your data and reduce the time and effort required to transform it. It is particularly useful for teams that are collaborating on data transformation processes or that need to maintain a high degree of confidence in their data.

For What Use Cases Should I Consider dbt?​

Some of the main use cases for dbt include:

  • Data transformation: dbt allows developers to transform data in a scalable and efficient manner, using SQL to build transformation pipelines.
  • Data enrichment: dbt can be used to join tables and create wide analytics tables.
  • Data testing: dbt includes testing functionality that can be used to ensure the quality of data, including testing for completeness, accuracy, and consistency.
  • Data documentation: dbt includes features for documenting data pipelines, which can help with maintaining data lineage and auditing.
  • Data orchestration: dbt can be used to orchestrate the execution of data pipelines, allowing developers to automate the transformation process.

Propel and dbt​

Propel and dbt offer a powerful combination. You can use dbt to transform your data in your warehouse, and Propel to expose it via a low latency API to your data applications. There are a couple of considerations:

dbt “full-refresh” is a safe operation

When developing models with dbt, it is common to perform a full refresh that deletes and recreates the table along with its data in the warehouse. With Propel, a dbt "full-refresh" is a safe operation. This is because Propel uses a cursor-based synchronization approach that picks up new records from where the cursor was pointing.

Rebuilding tables, changing schemas, or backfilling values

Sometimes you need to rebuild the data to fix old data, change the schema, or backfill values. In this case, the data that is already been synced to Propel is now different from the underlying table. If you need to bring this data to Propel, you need to create a new Data Pool. Note that when you do so, you don’t need to recreate the metrics or change your application. You can migrate Metrics from the old Data Pool to the new one as long as the schema is compatible.

Models have to be materialized as tables or incremental

When working with dbt, you need to materialize your models as tables or incremental. At this point, Propel does not support views.