s

Synchronize data from over 350+ sources to Propel’s Serverless ClickHouse.

Get started with Airbyte

Step-by-step instructions to connect any Airbyte source to Propel.

Features

Data Pools created with the Airbyte destination are Webhook Data Pools that support the following features:

Feature nameSupportedNotes
Syncs inserts, updates, and deletesPropel collects inserts, updates, and deletes sent from Airbyte.
Re-syncYou can trigger a resync in Airbyte.
Configurable sync intervalYou can configure sync interval in Airbyte.
Sync Pausing / ResumingYou can pause / resume syncs in Airbyte.
Sync mode: Full refresh - OverwriteSupports the Full refresh - Overwrite Airbyte sync mode.
Sync mode: Full refresh - AppendSupports the Full refresh - Append Airbyte sync mode.
Sync mode: Incremental - AppendSupports the Incremental - Append Airbyte sync mode.
Sync mode: Incremental - Append + DedupedSupports the Incremental - Append + Deduped Airbyte sync mode.
Schema changesWhen Airbyte detects a column was added in your source table, Propel will automatically add it to the payload in the _propel_payload column. It will not add it as a column in your destination Data Pool.
Delete Job APISee Delete Job API.
API configurableSee API docs.
Terraform configurableSee Terraform docs.

How does the Airbyte destination work?

Propel creates a Webhook Data Pool for each Airbyte source table. Airbyte streams inserts, updates, and deletes as individual records to Propel.

Data Pools mirror the source table’s structure, with additional columns from Airbyte and Propel:

ColumnTypeDescription
_propel_received_atTIMESTAMPThe timestamp when the record was received by Propel in UTC.
_propel_payloadJSONThe JSON Payload of the record.
_airbyte_raw_idSTRINGThe unique ID of the record generated by Airbyte.
_airbyte_extracted_atTIMESTAMPThe timestamp when the data was extracted from the source by Airbyte in UTC.

After connecting, Airbyte performs initial and scheduled syncs, sending data to Propel’s Webhook Data Pool API. Propel then processes this data, making it available for SQL and the Query APIs.

Key configurations

  • Default Timestamp: _airbyte_extracted_at
  • Primary Key: _airbyte_raw_id
  • Sort Key: Combination of primary key and default timestamp

You can modify the sort key using a Materialized View.

Data handling

Airbyte streams capture inserts, updates, and deletes. Process these changes using a Materialized View to create a new, updated Data Pool.

Schema changes

Source schema changes are reflected in the _propel_payload JSON, but not in the Data Pool schema after initial setup.

Data types

The following table illustrates how we transform Airbyte data types into Propel-supported types:

Airbyte TypeAirbyte FormatPropel Type
(Not specified or Null)-String
String-String
StringDateDate
StringDateTime (TimestampWOTZ)String
StringDateTimeTimestamp
StringTimeString
Boolean-Boolean
Number-Double
Integer-Int64
Object-JSON
Array-JSON

Transforming data

Once your data is in a Webhook Data Pool, you can use Materialized Views to: