Today, we’re excited to announce Propel’s support for Snowflake dynamic tables. Recently announced by Snowflake in in public preview, we believe Snowflake dynamic tables are set to become the default pipeline and data transformation tool for Snowflake customers. With that in mind, at Propel, we knew that full support for this feature was critical for enabling the best possible customer-facing analytics use cases for customers using Snowflake as their data warehouse.
What are Snowflake dynamic tables?
Snowflake Dynamic Tables are a type of table that materializes the results of a SQL query. This means that the data in the table is always up-to-date, and you can query it just like any other table.
Dynamic Tables are a powerful tool because they make it easy to build and manage data pipelines. With Dynamic Tables, you can declaratively define the result of your pipelines using simple SQL statements. This means that you can focus on the business logic of your pipelines, and let Snowflake handle the details of scheduling, orchestration, and execution.
Dynamic Tables also support incremental updates, so you only need to process the data that has changed since the last refresh. This can significantly improve the performance of your pipelines, especially for large datasets.
As mentioned, Snowflake dynamic tables are now in public preview, meaning all Snowflake customers can access the feature. To learn more, check out the Snowflake dynamic tables documentation.
Designing Snowflake dynamic tables to power customer-facing analytics with Propel
Propel and Snowflake dynamic tables are a very effective combination to power customer-facing dashboards and APIs.
In this example, we are landing raw JSON events into Snowflake using Snowpipe. Once there, we’re extracting the JSON data out into two relational tables - all orders received, and a table listing all customers. When performing data analysis internally using Snowflake this setup is great; we can join on Orders and Customers to get the insights we want such as orders per customer.
When we want to prepare this data for use in customer-facing analytics, we want a single analytics table with all the data represented - both orders and customers. Connecting Propel to a dynamic table setup this way means the data as fast as possible to end customers, where low latency is critical.
Using a dynamic table here allows us to join both data sets together. It also means that as new data is added to either the underlying Orders or Customers table, the dynamic table incrementally updates. Propel will then pick up and synchronize from the dynamic table, making your analytics data ready for use in our platform.
How to get started
For existing customers, please contact us to get dynamic table support activated on your account. If you don’t have a Propel account yet, click here to get started. Also, check out our Data Chaos podcast, where we deep dive into the entropy that exists in the world of data.
