Hi there!

Getting data in good shape is sometimes the hardest part about building data applications. It’s definitely going to be one of the hardest parts of building AI applications.

When you hear “we need to get our data in good shape first” it typically means doing one or all of these transformations:

• Flatten nested JSON into tabular form
• Combine data from multiple source tables through JOINs
• Calculate new derived columns from existing data
• Perform incremental aggregations
• Sorts rows with a different sorting key
• Filter out unnecessary data based on conditions
• De-duplicate rows

Luckily, we have announced Materialized Views. They automate all the underlying data transformation pipelines, allowing you to focus on the actual transformations in SQL. Not only do they do it in real-time, but there’s no more scheduling or full refreshes.

What’s New

Some months, a lot of small things land. In others, one big one lands.

This month, we’re excited to announce a big one: the new Materialized Views, a simple yet powerful way to transform your data with SQL.

Here are the highlights of what’s new 👇

• Materialized Views - Transform data with SQL in real time.
• Rockset migration service - the easiest way to migrate off Rockset.
• Customizable table engine and sorting key - More control over Data Pools.
• 2 bug fixes
• 11 improvements

✨ Materialized Views

We're introducing Materialized Views in Propel’s Serverless ClickHouse as a powerful tool for data transformation. Developers can leverage Materialized Views to reshape, filter, or enrich data with SQL. Materialized Views are persistent query results that update dynamically as the original data changes.

The key benefit? Data is transformed in real time. No scheduling. No full-refreshes.

Learn more about Materialized Views.

✨ Rockset Migration Service

OpenAI has announced the acquisition of Rockset, and as a result, the Rockset service will cease to operate. For those unfamiliar with Rockset, it was a cloud-hosted real-time analytics database that enabled millisecond-latency queries for aggregations and joins, similar to Propel.

We are pleased to announce the immediate availability of the Rockset Migration Service. This service is designed to offer a seamless transition for companies from Rockset.

Read our full blog post to learn more or get started with the migration process.

✨ Customizable table engine and sorting key for all Data Pools

We're thrilled to announce that Propel now supports customizable table engines and sorting keys for all Data Pools. What does this mean? Better query performance, more cost-efficient reads, and support for real-time updates and delete on any Data Pool type.

Table engines in Propel’s Serverless ClickHouse determine how tables store, process, read, and update their data.

The sorting key is a set of one or more columns that Propel uses to organize the rows within a table. It determines the order of the rows in the table and significantly impacts the query performance. If the rows are sorted well, Propel can efficiently skip over unneeded rows and thus optimize query performance.

This enhancement provides users with more flexibility and control over their data, allowing them to optimize their data pools for their specific use cases.

Learn more about the table engine and sorting key.

Latest Posts and Episodes

From Propel

• In-depth: What is Customer-facing analytics? (Blog post)
• How to select a table engine and sorting key (Guide)
• What is the default timestamp and how is it used? (Guide)
• When and how to choose a partition key (Guide)

From the community

• The Ultimate Guide to Finding Outliers in Your Time-Series Data - Deep dive into time series data.
• Apache Iceberg - What Is It? - Apache Iceberg has been all over the news. This is a nice summary and explanation.
• A Recap of the Data Engineering Open Forum at Netflix - All very relevant topics.

How to get started with Propel for free?

Sign up and start building with a generous free tier forever 👩🏽‍💻  🏗️  🚀!

Thank you for reading!

‍

Nico Acosta
Co-founder & CEO
Propel

‍If you find these product updates useful, please forward them to a colleague who can subscribe here. Alternatively, you can subscribe to our podcast, Data Chaos, on Spotify or Apple Podcasts.

‍