Introducing new synchronization controls for Snowflake data warehouse integrations

New features enable fine-grained control over how you can synchronize data between Snowflake and Propel’s high-performance analytics platform

Photo: Propel

Today, we’re delighted to highlight a series of new features for Propel Data Pools and how they work with synchronizing data from your Snowflake warehouse. A Data Pool is a high-speed data store and cache optimized for serving data with low latency (sub-second response times) and high concurrency (thousands or millions of users). With these new features, customers have even more control and visibility into how data is synchronized into the Propel platform from Snowflake. Let’s dive in and take a look at these new features.

Sync Interval

Sync interval allows you to control how frequently the Data Pool will check for new data to synchronize. The lowest interval is 1 minute, ranging up to once a day. Determining what interval to use depends on the data freshness requirements for your application. A higher sync frequency only makes sense if your data is updating at the same cadence.

<aside>💡 Note: Setting a more frequent interval, such as 1 minute, can have cost implications as checking for new data will wake up your Snowflake warehouse in order to query.

</aside>

Sync Pausing

Synchronization can now be paused via the API or Console. When paused, Propel stops attempting to synchronize records to your Data Pool. As mentioned above, pausing or resuming syncing can shift the syncing interval, because the interval resets when syncing is resumed. Queries to the Data Pool for metrics still work while syncing is paused. So pausing can be useful if you need to rebuild a table in your warehouse.

Sync Visibility

Sometimes there is no data to synchronize. Propel now shows every attempt to synchronize your records, even if there were no new records to synchronize.

A screenshot of the Propel console showing Sync History with 3 successful previous syncs, two of which were where records have been added and one with no records added.

This can be helpful in fine-tuning the interval that is most appropriate for your use case. Additionally, you can use the toggle Show-only syncs with records (shown in the screenshot above) on the Data Pool overview page to quickly see the latest syncs that added records — particularly useful when debugging.

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5

Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.