Today, we’re launching a new way to power your customer-facing analytics: Webhooks. In addition to syncing data from your database and data warehouse, you can now send JSON events from your application or SaaS platform directly into Propel via an HTTP POST request.

Webhooks are also an ideal way to integrate event buses such as Segment, Ruddestack, or AWS EventBridge, workflow tools like Zapier or IFTTT, or streaming sources like Kafka or Kinesis.

What are Webhooks?

Webhooks are HTTP requests triggered by specific events in applications. When an event occurs, the application sends the event, typically in JSON format, to the specified URL as an HTTP request, i.e., the webhook. They are a simple yet powerful way to push data from one application to another, making them ideal for real-time analytics.

How Lumeo powers computer vision analytics with Webhooks

Lumeo enables enterprises to run computer vision pipelines on their installed commercial camera systems. When processing the video streams, Lumeo’s platform publishes the results of its computer vision models to Propel in real time via webhooks to power their customer-facing dashboards.

Benefits

Bring data together from multiple sources

Webhooks provide the flexibility to ingest data from various data sources in real-time, making your analytics more comprehensive.

Seamless integration

Webhooks can easily integrate with your existing applications, SaaS platforms, or streaming infrastructure, requiring minimal changes. This ensures you can get the data you need to power your analytics without expensive data engineering projects.

Real-time analytics

With Webhooks, you can process data the moment it’s generated, enabling you to provide real-time insights. This is particularly useful for applications dealing with fast-changing data or requiring immediate response based on the incoming data.

Reduced complexity

Webhooks simplify the data pipeline by reducing the steps and tools needed to ingest data. By directly pushing data from your application to Propel, you eliminate the need for intermediary data storage or transformation tools, simplifying your data architecture and reducing potential points of failure.

Elastic scalability

Webhooks are designed with scalability in mind. As your data grows, Webhooks can easily handle increased volumes, ensuring your analytics continue to run smoothly and efficiently.

Key features

Flexible schema definition and automatic JSON flattening

With Propel's Webhooks, you can customize the schema according to your JSON payload. It allows you to unpack and flatten both top-level and nested fields. Each unpacked field will then become its own column in the Data Pool. As events are received in the Webhook Data Pool the values are flattened into their corresponding columns.

Individual or batch event collection

You can send individual events or batches of JSON events to the Webhook Data Pool by making an HTTP POST request to the Data Pool's URL.

The examples below demonstrate how to send individual events and batches of events for the TacoSoft dataset from the Quickstart.

Sending individual events

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
-d '{
      "customer_id": 5,
      "order_id": 34,
      "store_id": 4445,
      "order_details": {
        "taco_count": 7,
        "total_price": 25.90,
        "checkout_time": "2023-07-31T15:20:10Z"
      }
      "created_at": "2023-07-31T14:50:35Z"
    }'

Sending batches of events

curl https://webhooks.us-east-2.propeldata.com/v1/WHK... \
-X POST \
-H "Content-Type: application/json" \
-d '[
  {
    "customer_id": 5,
    "order_id": 34,
    "store_id": 4445,
    "order_details": {
      "taco_count": 7,
      "total_price": 25.90,
      "checkout_time": "2023-07-31T15:20:10Z"
    },
    "created_at": "2023-07-31T14:50:35Z"
  },
  {
    "customer_id": 8,
    "order_id": 22,
    "store_id": 1199,
    "order_details": {
      "taco_count": 3,
      "total_price": 15.75,
      "checkout_time": "2023-07-31T12:40:21Z"
    },
    "created_at": "2023-07-31T12:30:55Z"
  },
  {
    "customer_id": 12,
    "order_id": 67,
    "store_id": 876,
    "order_details": {
      "taco_count": 10,
      "total_price": 42.20,
      "checkout_time": "2023-07-31T09:15:50Z"
    },
    "created_at": "2023-07-31T09:10:25Z"
  },
  {
    "customer_id": 3,
    "order_id": 45,
    "store_id": 662,
    "order_details": {
      "taco_count": 2,
      "total_price": 10.50,
      "checkout_time": "2023-07-31T18:05:15Z"
    },
    "created_at": "2023-07-31T18:00:45Z"
  },
  {
    "customer_id": 9,
    "order_id": 56,
    "store_id": 256,
    "order_details": {
      "taco_count": 6,
      "total_price": 35.60,
      "checkout_time": "2023-07-31T21:30:05Z"
    },
    "created_at": "2023-07-31T21:20:30Z"
  }
]'

Real-time ingestion

The Webhook Data Pool ingests data in real-time and makes it available via the query APIs and UI components.

Real-time updates

The Webhook Data supports real-time updates. It uses each record’s primary key, consisting of the primary timestamp and a unique ID, to determine whether it should be inserted or updated within the Data Pool. This also means that if you send the same record twice, it will not be duplicated.

Updates are useful in the following scenarios:

• When retrying requests: You can safely retry requests without worrying about creating duplicates.
• When updating the properties of an object: For example, if you need to update the "taco_count" property of the "order_details" object from "7" to "8.”

Configurable authentication

You can optionally secure the Webhook Data Pool URL with HTTP basic authentication. This prevents third-party actors from posting events to your endpoint without your consent.

Availability

It’s available right now to all customers!

Getting started

Now that you understand the power and flexibility of Webhooks in real-time data ingestion, it's time to start building! Whether you're a developer aiming to build analytics, or a data engineer seeking to serve your data through a low-latency API, Webhooks will accelerate your customer-facing analytics efforts.

If you don’t have a Propel account yet, check our Docs and our Quickstart.

To learn more about Webhooks, read the Webhook Data Pool documentation, or if you are ready to get started, read the How to ingest JSON events into Propel guide.