OEP-52: Event Bus Architecture#

OEP-52

OEP-0052

Title

Event Bus Architecture

Last Modified

2022-05-15

Authors

Feanil Patel <feanil@edx.org>, Robert Raposa <rraposa@edx.org>

Arbiter

Tobias Macey <tmacey@mit.edu>

Status

Accepted

Type

Architecture

Created

2021-08-16

Review Period

2022-06-09 - 2022-06-30

References

Follow up work: event bus project roadmap

Overview#

Note

This provisional OEP is expected to be updated with references to new sub-decision ADRs and documentation to add clarity as work progresses. See the event bus project roadmap for more details.

  • Adding an event bus to the Open edX platform allows asynchronous event messaging across services, which enables a number of improvements aligned with our architectural goals.

  • An abstraction layer will be provided to enable the choice of multiple technologies for implementing an Open edX event bus.

  • Kafka and Redis streams implementations of the event bus have been implemented.

  • Event bus events extend the Hooks Extension Framework events, and use an Avro schema to serialize the existing hooks signals.

Context#

The already accepted OEP-41: Asynchronous Server Event Message Format details the general format and conventions the Open edX platform should use for asynchronous event messaging across services. It also provides background on a set of Architectural Goals for the Open edX platform, including:

  • Align with the Architecture Manifesto themes of decentralization and asynchronous communication in support of loose coupling between services.

  • Eliminate blocking, synchronous requests.

  • Eliminate expensive, batch synchronization.

  • Reduce the need for plugins.

  • Flexibly integrate with event producers.

  • Simplify integration to external systems.

However, this earlier OEP explicitly leaves out of scope the specific transport and libraries used for this messaging. At the time, the Open edX platform lacked a reliable way to send events to multiple consumers across services, following the publish-subscribe messaging pattern (pub/sub).

Decision#

  • Adding an event bus to the Open edX platform allows asynchronous event messaging across services, which enables a number of improvements aligned with our architectural goals.

  • For the purpose of this OEP, the focus of the event bus is to provide publish-subscribe messaging pattern (pub/sub) and event-driven capabilities.

  • An abstraction layer will be provided to enable the choice of multiple technologies for implementing an Open edX event bus.

  • An initial event bus implementation has been implemented using Kafka.

  • A secondary implementation has been implemented using Redis streams.

  • Event bus events extend the Hooks Extension Framework events, and use an Avro schema to serialize the existing hooks signals.

    • See ADR on External event bus and Django Signal events to learn more about how the OpenEdxPublicSignal internal used to send the same internal signal-based events across services, and then fire the signal-based events again within consuming services.

    • See ADR on External Event Schema Format and ADR on Event Schema Serialization and Evolution to learn about how the Avro Schema format is used for serializing external events published to Kafka, and consumed from Kafka. Additionally, learn about tooling to automatically handle the translation from OpenEdxPublicSignal internal events to Avro Schema formatted events, and back again. Also learn how this explicit schema format can aid in schema evolution.

Consequences#

  • Individual usage of any new Open edX event bus cannot be required until there is an implementation available as part of a named release of the Open edX platform.

  • Offering an abstraction layer for the event bus is the current intent, but will require additional work and support. Additionally, any alternative technology implementations (in addition to Kafka and Redis) will require implementation and support.

  • Although external events offer many benefits, it also requires a different mindset around eventual consistency. Proper education and experience will be needed to support the success of the event bus.

Change History#

2023-05-15#

  • Added link to event_bus_redis implementation ADR.

  • Updated OEP status to Accepted.

2023-05-12#

  • Added link to new openedx-events ADR

2022-11-29#

  • Removed disclaimer about provisional ADRs (most have since been accepted) and added follow-up work in References

2022-07-11#

  • Updated formatting.

  • Updated note about provisional updates, including reference to roadmap item.

2022-03-30#

Updated draft as follows:

  • Moved the Kafka technology choice to a separate edx.org ADR.

  • Added references and overview of a variety of other event bus related ADRs.

  • Moved ADR on schema representation to openedx-events, near a number of other event bus related ADRs.

2021-08-16#

Initial draft.