Event-driven architecture has become popular for building scalable service-oriented architectures and microservice-based systems. Knowledge of basic event-driven architecture concepts is a valuable tool to have in your toolbox.
An event-driven architecture may be a useful choice for you if:
- You need to design and build a loosely coupled system that enables components to grow, scale, and fail independently.
- You have many different components and systems that need to operate in response to an event.
- You need to replicate data across systems, even when the systems are owned by different teams.
The components in an event-driven architecture could be separate services or separate modules within a single service. Shopping carts, real-time stock tickers, and membership sites are some examples of systems that may benefit from event-driven architecture. For example, when a user adds an item to a shopping cart, a specific component of an e-commerce system would send emails to the user with recommendations for similar products. This could be implemented in terms of events: adding an item to a shopping cart produces an event, and the email sub-system consumes the event.
In this article, you will learn the basic concepts of event-driven architecture, the benefits and challenges of implementing event-driven systems, and specific situations for which event-driven architecture is a good fit.
Fundamentals of Event-Driven Architecture
Event-driven systems are built around discrete units of work called events. Events indicate that something of interest has happened and represent new information or a meaningful change in state that may be of interest to one or more system components. Events are most useful when they have business relevance within or outside the system. For example, the fact that a customer purchased an item is an event that is relevant to an e-commerce business whose primary goal is to generate revenue from selling products. An event-driven architecture enables system components to detect and react to events efficiently and on demand.
You can think of an event-driven architecture in terms of three major components:
- A producer that detects events and publishes them with an event broker.
- An event broker that stores events in one or more queues or logs until a consumer consumes the events.
- One or more consumers that connects to the event broker and read events off various queues or logs for processing.
Event-driven architecture is an alternative to the more traditional "request/response" style of software architecture. Traditional software architecture patterns tend to focus on user-initiated commands as the primary driver of processing action within a system. The difference between events and commands is subtle. A command is a request to perform an action that has not yet happened. An event represents something that has already happened. A critical part of what makes event-driven architectures distinct from other software architectures is the increased focus on events relative to commands. This increased focus on events enables you to build event-driven systems with components that are highly decoupled from one another and easier to change over time.
Producers and Consumers
Producers are components that detect events and send notifications about these events to other systems or components. Consumers are components that listen for specific types of events and perform some action when they receive notification of an event that they are listening for. Producers and consumers can be any type of code or hardware, including database servers, services, devices, and human users. Event producers and consumers can be part of the same software application or be separate software applications. A single component can be both a producer and consumer within the same system.
The diagram above shows an example of an event-driven architecture for e-commerce operations and the associated data analytics with two producers (orders and memberships), three consumers (ingestion, fulfillment, and notifications), a data warehouse (e.g. Snowflake), and a data analytics API platform (e.g. Propel). The e-commerce system enables customers to place orders, cancel orders, sign up for membership, and cancel membership. In the example architecture diagram, each producer and consumer is a separate software application.
If you visit this e-commerce website and place an order for a pair of shoes, the orders service will detect that you placed an order and send a notification via the event broker that an OrderPlaced event has occurred. The OrderPlaced event will contain information about the items you ordered, the total price of the order, and any other pieces of information necessary for consumers to process the order. The fulfillment service will consume the OrderPlaced event and perform the necessary actions to prepare your order at the product warehouse. The notifications service will also consume the OrderPlaced event, and will send you one or more emails about your order. Finally, the data ingestion service will consume the OrderPlaced event and update the Snowflake data warehouse with the new data about your purchase, thereby making it available for purchase analysis and reporting via the Propel data analytics API platform.
You may also choose to cancel an order you have previously placed. In this case, the orders service will detect that you canceled a previous order and send a notification via the event broker that an OrderCanceled event has occurred. The OrderCanceled event will contain information about the order that was canceled, the reasons for the cancellation, and any other pieces of information necessary to process the order cancellation. The fulfillment service will consume the OrderCanceled event, and reverse the prior fulfillment of the order if possible. The notifications service will also consume the OrderCanceled event and send you one or more emails with information about the status of your order cancellation. The data ingestion service will also consume the event and update one or more data stores with data that may be useful for further analysis.
This fictitious e-commerce website also offers you discounts if you are a paid member. If you cancel your membership, an event-driven process similar to the order placement and order cancelation takes place. The memberships service will detect that you canceled your membership and send a notification that a MembershipCanceled event has occurred. The notifications service will consume the MembershipCanceled event, and send you one or more emails begging you to renew your membership. The data ingestion service will also consume the MembershipCanceled event, and will update the data warehouse with the new data about your membership cancellation. The data about your membership cancellation will be available for membership churn analysis and reporting almost as soon as it occurred.
Event Brokers
Event-driven architectures rely heavily on event brokers to propagate events throughout the system and deliver events to interested consumers. Event brokers, also known as message brokers, are systems that receive events from producers and transmit the events to consumers. Event brokers enable asynchronous messaging, allowing producers and consumers to send messages to one another without waiting for responses. They also enable interoperability between systems, which allows different systems to exchange data and messages.
Event brokers support both push and pull consumption models. In the push model, consumers subscribe to specific events and the event broker pushes events to subscribed consumers. In the pull model, consumers request events from the event broker. The consumption model and event broker you choose depend on your specific use case. Apache Kafka, RabbitMQ, or AWS EventBridge are event brokers that are widely used to implement event-driven architectures.
Benefits of Event-Driven Architecture
You can derive several benefits for your systems when you adopt event-driven architecture.
Reduced Latency
In contrast to more traditional approaches such as synchronous request/reply or Remote Procedure Calls (RPC), components in an event-driven architecture do not need to wait for responses to their messages. Asynchronous messaging in event-driven architectures enables systems to reduce perceptible latency associated with transporting data across network connections.
Loose Coupling
Event-driven architectures make it easier to build loosely coupled systems. Loosely coupled systems are easier to maintain since code and functionality changes in one component are less likely to require changes in other components.
Scalability
Event-driven architectures make scaling your system easier by allowing you to simply add extra producers and consumers, eliminating the need to rewrite code or deal with the complexities of direct, inter-process communication. You can scale components horizontally across multiple instances through replication of state while maintaining high availability among all instances.
Independent Scaling and Fault Tolerance
Event-driven systems scale and fail independently. For example, if one consumer becomes overloaded with processing tasks, you could allocate more memory or CPU to that consumer without affecting any other consumers or producers. Furthermore, because components in an event-driven system are loosely coupled, the failure of one consumer or producer is unlikely to affect other consumers and producers, thereby improving fault tolerance.
Auditing and Monitoring
Some implementations of event-driven architectures are particularly well-suited for monitoring and auditing since events are typically stored and/or accessible from a central location (i.e. the message broker) instead of being spread across different databases or log files.
Event-driven analytics
Since events represent a change that happened in the world, the data they carry is optimal for aggregate analysis or analytics. Analytics on event data can help answer business questions, track key performance indicators, or
Near Real-Time Data Processing
Event-driven architecture enables systems to receive and process data in near real-time. This is particularly useful for business analytics since your analysis will be based on data that includes the most recent state of your system, thereby increasing the likelihood that you will make better business decisions.
Challenges of Event-Driven Architecture
While event-driven architecture can offer many benefits. It's not without its challenges.
Deciding How Much Information to Include in Events
When implementing an event-driven architecture, you may find it difficult to determine the right amount of information to put into each event without driving up storage costs or causing system performance to degrade. If you include all the information that consumers could possibly need, you can simplify the processing logic required of consumers. However, placing too much information within a single event could have a negative impact on system performance due to large event sizes. By including only a minimal amount of information into your events, you'll likely reduce network time and cost, but consumers will need to perform additional lookups to retrieve all the data they need for processing.
Complexities of Asynchronous Communication
Asynchronous messaging increases complexity due to overhead as the message passes between multiple distributed processes, potentially hosted on different machines. Due to the inherently asynchronous nature of event-driven architectures, you may find it difficult to tell if and how two events are related in time. This makes troubleshooting and debugging event-driven systems tricky in certain scenarios.
Delivery Guarantees and idempotency
Some event brokers, such as AWS EventBridge, make an at-least-once delivery guarantee. This means that events might be duplicated. Ensuring that your application can process events idempotently is crucial for event-driven architectures. Meaning that the same event would not be processed twice. So if a `chargeAccount` event is fired twice, your application needs to make sure it knows that this particular event was already processed to avoid charging the account twice.
Conclusion
This article has explored the basics of event-driven architecture and how events, producers, and consumers work together to create systems that solve real problems. You should now understand the benefits and challenges of event-driven architecture, why it is a good fit, and how it helps you to build loosely coupled, fault-tolerant, and scalable systems.
Event-driven architectures are well-suited for data analytics applications and platforms. If you are looking to develop a robust data analytics platform as part of your event-driven architecture, Propel may be the right tool for you.
