Event Platform
The Wikimedia Event Platform exists to enable software and systems engineers to build loosely coupled software systems with event driven architectures. Systems built this way can be easily distributed and decoupled, allowing for inherent horizontal scalability and versatility with respect to yet-unknown use cases.
Platform Architecture Diagram
Philosophy and data model
In typical web stacks, when data changes occur, those changes are reflected as an update to a state store. For example, if a user renames a wikipedia page title, that page's record in a database table would have its page title field updated with the new page title value. Modeling data in this way works if all you care about is the current state. But by keeping only the latest state, we discard a huge piece of useful information: the history of changes.
Instead of updating a database when a state change happens, we can choose to model the state change as what is is: an 'event'. An event is something happening at a specific time, e.g. 'page id 123 was renamed from title_A to title_B by user X at 2020-06-25T12:34:00Z'. If we keep the history of all events, we can always use them recreate the current state as well as the state at any point in the past. An event based data model decouples the occurrence of an event from any downstream state changes. Multiple consumers can be notified of the events occurrence, which enables engineers to build new systems based on the events without interfering with the producers or other consumers of those events.
Event Platform generic concepts
| event | A strongly typed and schema-ed piece of data, usually representing something happening at a definite time. E.g. 'revision create','user button click', 'page view', etc. |
| event schema (AKA 'schema') | A datatype of an event. The event schema describes the data model of any given event and is used for validation upon receipt of events, as well as for data storage integration (an event schema can be mapped to an RDBS schema, etc.). An event schema is just like a programming data type. |
| event stream (AKA 'stream') | A continuous collection of events (loosely) ordered by time. |
| event bus | A publish & subscribe (PubSub) message queue where events are initially produced to and consumed from. WMF uses Apache Kafka. (NOTE: here 'event bus' is meant to be a generic term and is not referencing the MW EventBus extension). |
| event producer | Producers create events and produce them to specific event streams in the event bus. |
| event consumer | Consumers consume events from specific event streams in the event bus. |
| event intake service | An HTTP service that accepts events, validates them against their schema, and produces them to a specific event stream in the event bus. WMF uses EventGate. |
| Event stream processing | Refers to any software that consumes, transforms, and produces event streams. This includes simple event processing, as well as complex event processing and stateful stream processing. This is usually done using a distributed framework of some kind, e.g. Apache Flink, Apache Spark, or Kafka Streams, but also includes simpler home grown technologies like Change-propogation. |
Background reading
- Martin Fowler - What do you mean by "Event-Driven"
- Confluent - Turning the Database Inside Out
- Confluent - Event Driven 2.0
- Confluent - Designing Event Driven Systems
- Confluent - The Data Dichotomy: Rethinking the Way We Treat Data and Services
- Confluent - Stream data platform 1
- Confluent - Messaging as the Single Source of Truth
- Confluent - Build Services on a Backbone of Events
- What they dont tell you about event sourcing
- Confluent - Kafka Streams: Stream Processing Made Simple (This is about Kafka streams, but the discussion of streams vs. table and stateful streaming is good.)
- WMF Tech Blog - Wikimedia’s Event Data Platform (part 1, part 2, part 3)
- Wikipedia’s Event Data Platform, Or: JSON Is Okay Too, Andrew Otto, Confluent Current 2022 (slides)