Deploying Open edX on Kubernetes Using Helm#
Status#
Accepted
Context#
For most of the history of the Open edX project, production deployments were done primarily using Virtual Machines that had been provisioned using the ansible scripts in the edx/configuration repository. With the adoption of the decision 0001 from OEP-0045 that selects Tutor as the replacement for edx/configuration some large instances and operators are encountering use cases that are not supported. Multiple instances of Open edX sharing some infrastructure is the first of such cases.
A popular way to deploy containerized applications is to use Kubernetes. However, Kubernetes has a lot of complexity so the overhead it brings is usually not worthwhile for small, singe-instance deployments. For that reason (as explained in OEP-45) as well as a general lack of experience with Kubernetes, to date there has been no officially recommended way to deploy Open edX on Kubernetes. Although Tutor does include basic support for Kubernetes, it is focused only on a basic working deployment, and lacks the flexibility and capability that operators require for large, multi-instance deployments.
Despite this, many major commercial Open edX hosting providers have migrated toward deploying Open edX on Kubernetes. In doing so, they developed several different approaches for doing so, each fairly different from each other but all focused on deploying Tutor-generated images into a Kubernetes cluster. This has resulted in duplicated effort and made it difficult to share best practices and lessons learned with each other. For further context, read the forum discussion Deploying multiple Open edX instances onto a Kubernetes Cluster with Tutor.
Decision#
For Open edX operators who wish to deploy Open edX on Kubernetes, community-maintained Helm Charts will be used to simplify the process of deploying a kubernetes cluster capable of loading one or more Open edX instances.
Building on top of the decision 0001 those single instances will in turn be managed by Tutor and the Tutor plugin ecosystem.
As the needs and deployment scale of Open edX operators can vary significantly, the Helm charts will be designed to be flexible, incorporating sensible defaults but allowing customization of almost every aspect of the deployment. This includes supporting various cloud providers and different underlying technologies for Kubernetes resources. E.g. Ingress controllers.
The goal is for these Helm charts to be developed and maintained collaboratively, by the Open edX Operators that are using them. All relevant best practices from both the Open edX and Helm projects should be followed from the start, such as OEP-55 Project Maintainers.
Consequences#
Helm Considerations#
Helm is the best known package manager for Kubernetes. It’s status as a graduated project from the Cloud Native Computing Foundation is a clear indication of the thriving adoption, open governance and commitment to the open community.
Helm makes it possible to package and publish charts, which can be rendered as kubernetes manifests. These charts provide robust support for versioning, deployment and rollback. At the same time Helm makes it easy for operators to customize application configurations during deployment and for developers to provide sensible defaults.
Charts are built with the Go language templating engine. This is a drawback, since it means adding a new language to the ecosystem with a new learning curve. Nevertheless, this is considered acceptable due to the large adoption of Helm in the devops community.
Best Practices#
Cloud Providers
As much as possible, the Helm chart should aim to be agnostic toward the underlying cloud provider used (e.g. AWS, Azure, Google Cloud Platform, etc.). In cases where provider-specific code is required, the Charts should be written in such a way that they can be configured to work with multiple providers, and not e.g. only support AWS.
Helm subcharts and modularity
The project should aim to make the charts available in a way that is composable and granular. Extending a single subchart and using it in the composition should be a better and faster alternative than forking the whole project.
Industry Best Practices#
The recommendations above are heavily inspired by the following resources:
Alternatives Considered#
Building more Kubernetes support into Tutor#
Enhancing the current support for tutor core was not considered since it would go against one of the goals of the project as explained by the author.
Custom Templating tool#
A custom templating tool built as a tutor plugin and leveraging kustomize for extension was demonstrated and considered as a possible alternative. It was ultimately rejected as Helm would provide more flexibility in extension without the need of maintenance.
Implementation Strategy#
Once this ADR has reached Provisional status:
A new repository will be created on GitHub, e.g. at openedx/openedx-helm-charts
The authors of this ADR will create a roadmap using GitHub Issues by defining the initial development issues.
Invite interested parties to become core contributors on the repo, and nominate some of them.
Participating core contributors will then complete the roadmap and begin developing an MVP.
References#
A working proof of concept that was written as part of the research for this ADR. See 2022-11-22 meeting recap for further information.
Change History#
2023–30#
Document updated to include community feedback.
2022-11-30#
Document updated to become an ADR of OEP-45 instead of a standalone OEP.
2022-08-19#
Document created