OEP-21: Deprecation and Removal#

OEP

OEP-21

Title

Deprecation and Removal

Last Modified

2024-04-04

Authors

Greg Sham <gsham@edx.org>, Nimisha Asthagiri <nimisha@edx.org> Diana Huang <dkh@edx.org> Sarina Canelake <sarina@axim.org>

Arbiter

Matt Tuchfarber <mtuchfarber@edx.org>

Status

Accepted

Type

Process

Created

2018-05-18

Review Period

2018-06-06 - 2018-06-20

TL;DR#

  • Removing unneeded or legacy code is crucial to optimizing programs and reducing costs: Having dead or broken code laying around can inadvertently introduce bugs, cause the runtime footprint to be larger than necessary, and cost time and money to maintain. Additionally, unneeded code may make it more difficult to understand or navigate a project.

  • It’s important to know exactly when and what to remove: Some code may be outdated and can be replaced with a new implementation, or some may have very low usage and is not worth holding on to. It’s important to analyze to what extent the code is no longer needed and what (if any) ripple effects it could have elsewhere. Monitoring can be a useful tool for this discovery.

  • Communicating out proposed changes can help guide the path to depreciation/removal: It is crucial to let others know you believe there’s code needing removal and how you plan to go about it. Inform individual stakeholders and use Open edX community channels such as Slack and Discourse to let others know your proposed plan. Collect and monitor feedback to avoid potential disruptions to other code/projects.

  • Others can take responsibility: In some cases, an item may be useful to others but not to you. That’s okay; part of the deprecation process is given others the ability to take over ownership of those parts of the system. Those who value a feature should support maintaining it.

  • The formal process outlined in this document can help make deprecation and removal projects happen more proactively: Having a defined process and common language takes the guesswork out and allows for easier access to get rid of dead code. The steps outlined in this document regarding analysis, communication, and documentation will help guide the process, and by using GitHub Issues as tickets to track the DEPR process (see the steps outlined in the Document section), these projects can be clearly tracked providing transparency on progress.

High-Level Overview of Code Removal Process#

../_images/Removal-Workflow.png

Best Practices for Proposing Code for Removal#

  • For further details on each of these steps, please see Proposed section. For DEPR ticket details, please refer to the Document process.

../_images/Best-Practices.png

Abstract#

“Remove dead code wherever possible. It gets in the way and slows you down.” [OREILLY]

This document defines a formal process for proposing, communicating, deprecating and removing legacy and unneeded code in the Open edX platform.

../_images/dead-code.png

Motivation#

Without a clearly defined and agreed upon process for removing legacy code, Open edX developers will hesitate to do so. As a result, the platform will have redundant implementations and dormant features that over time will slow down development and developer onboarding.

This is a problem shared by the software engineering industry and many have learned the benefits of investing in code removal:

  • It is undeniable that unnecessary code, like any other code, requires maintenance over time. It costs time and money. [OREILLY]

  • Extra code also makes it harder to learn the project, and requires extra understanding and navigating. [OREILLY]

  • It is harder to refactor, simplify, or optimize your program when it is bogged down by zombie code. [OREILLY]

  • There is a danger that someone inadvertently involves the ‘dormant’ code and introduces bugs. [SO] [SEC]

  • Dead code makes the runtime footprint larger than it needs to be. [INFOQ]

  • Dead code discourages a culture of treating the software as soft, and therefore always open to revision and improvement. [INFOQ]

Given the above, we define a common language and process to remove unneeded code, that is, code that may be superseded by a new implementation or may be deemed no longer necessary or supported. Following this process guarantees alignment across all stakeholders and a path towards complete removal of all unused code.

Specification#

This section explains each step in the process to remove a technology from the Open edX platform.

Process States#

Per the workflow chart, the next sections go through the details of each state and transition.

Proposed#

Do you think a piece of the codebase should be removed? If you even asked that question, chances are likely that it probably should be.

When to remove?#

Here are a few common cases for code removal:

  • There is now a new implementation that replaces the old implementation. If so, the old implementation should be deprecated and then removed in favor of the new.

  • Static analysis or runtime analysis shows that the code is never executed on your Open edX instance. If so, it should be removed (if no one in the community requires it) or should become a pluggable extension since it’s not core to all instances.

  • There is a legacy feature that never really saw the light of the day or was adopted by very few users. If so, this should be confirmed by usage analysis and then removed.

What to remove?#

Here are a few technologies that are commonly removed:

  • User features, including their APIs, frontend, and backend implementations

  • Modular components, such as Django apps, Frontend apps, XBlocks

  • Technologies, such as CoffeeScript, outdated frameworks

  • Feature toggles used for temporary rollout and testing

  • Interfaces, such as REST APIs and Plugin APIs

Analyze#

When proposing a removal, consider the following analysis:

  • Usage - Which users and services are currently using the code proposed for removal on your own Open edX instance? Perform a quick search across the edX codebase to gauge the level of impact and identify potential stakeholders. search. Also read how to monitor deprecation/removal to support this entire process.

  • Replacement - What, if any, is a viable replacement for the code being removed?

  • Migration path - If there is existing high usage in the community, what is a viable automated migration path from the deprecated code to the removed code?

  • Deprecation - Based on expected usage and effort to migrate, for how long should the deprecation period be?

  • Coordination - Who is going to take responsibility for moving the deprecation forward?

Timeline#

Deprecation can have one of two results: the feature gets removed, or someone else takes over the feature and becomes responsible for maintaining it.

To communicate a proposed removal, you will need to determine the earliest Open edX named release that could no longer have the removed functionality, based on the release’s cut date. Choosing the earliest named release where functionality could be removed is the simplest way to summarize what is often a lot of guesswork, and to clearly communicate to others how to prepare for the removal. The following discussion is meant to help determine the earliest named release.

When determining target dates to propose for the removal process, consider that it will vary depending on team resources and the technology being removed. The importance of removal (as described in Motivation) should be communicated with all team members so the removal can be prioritized and completed in a timely manner. A suggested timeline is shown in the diagram below, which considers the timing of the next Open edX named release.

A diagram that suggests having a 2 week time period between the *Proposed* and *Accepted* states, giving the community enough time to provide feedback. After which, the *Deprecated*, *Removing*, and *Removed* transition periods will vary by the type and scope of the technical change.
  • Proposed on Day 1

  • Communicated from Day 2 to Day 13

  • Accepted on Day 14 (depending on influx of feedback)

  • Deprecated/Removing/Removed - from Day 15 onwards (depending on resources and technology being removed). Consider when the next Named Release is cut; if it is very soon, you may wish to delay final removal until after the cut date.

Consider choosing deprecation and removal dates that allow for a full release cycle for transition planning. For example, a deprecation proposal could be accepted while Maple is being finalized, then implement the removal some time after Maple is released so that the removal itself will land in Nutmeg. (Removal could even happen as soon as a named release’s branches are cut, but this may interfere with fixes that need to be backported.) Any deployment following the named releases would then have a number of months to prepare before Nutmeg comes out.

This approach would be most appropriate for features that can be left in place for an extended period before removal and where a transition to an alternative would require a moderate to large amount of effort. For more trivial deprecations, it may be appropriate to simply deprecate and remove within the same release cycle.

Remember to use the named release’s cut date when determining the appropriate named release. Additionally, if the named release is far enough in the future that it only has a letter (and not a full name), just use the letter of the release. Reach out to the #wg-build-test-release in Slack if you’re still not sure what the most appropriate earliest named release would be.

Document#

Do the following to document your proposal:

  1. Create a GitHub Issue in the repo where the code being deprecated lives, and be sure to choose the “Deprecation (DEPR) Ticket” template. If your deprecation spans multiple repos, choose the primary/most relevant repo, or use the public-engineering project.

    Note

    While it is possible to create Issues with no template, it is strongly encouraged that you go to github.com/openedx/:repo/issues, click “New Issue”, and choose the DEPR template so you don’t miss any fields and automation works properly. The template fields help us more quickly address deprecation issues and reduce the amount of back and forth needed to make progress on work. If you must create an Issue outside the template, please preface your issue title with [DEPR].

  2. When writing the ticket, include the following information:

    1. Title: The title of the ticket should read “[DEPR]: <technology name being deprecated>”.

    2. Proposal Date: the day the proposal is being put up for consideration.

    3. Target Ticket Acceptance Date. A good default is 2 weeks from the Proposal Date. See Timeline for additional considerations.

    4. Earliest Open edX Named Release Without This Functionality. See Timeline for help choosing the earliest Open edX named release where the code will have been removed, based on its cut date.

    5. Rationale: A few sentences explaining the rationale for removing this technology.

    6. Removal: A description with links to what is being removed.

    7. Replacement: A description with links to what it is being replaced by.

    8. If you plan to mark the code for deprecation, explain how in the Deprecation section. See Deprecated for considerations.

    9. If automated migration will be needed, explain your migration plan in the Migration section.

    10. Additional Info:

      • If there is any additional publicly shareable information or data from your earlier analysis, include that in the Additional Info section.

      • If you know who plans to coordinate the ticket, include the name in the Additional Info section. The deprecation ticket is not active until there is a coordinator.

  3. Check that your ticket appears on the DEPR Project Board.

Note

You can create a ticket even if you don’t yet have all the information gathered yet. You simply must wait until you have completed the issue and have a coordinator before moving on to the formal Communication step.

(Optional) Start a preliminary discussion#

If you want to start a preliminary discussion or seek an owner for the deprecation ticket, you can post a message in the Development Discussion Board and any appropriate channels in the Open edX slack. This allows you to start a discussion and ask for feedback before somebody commits to coordinating the DEPR process. Note: This does not replace the formal Announce of the DEPR ticket.

Claim the DEPR Ticket as Coordinator#

A DEPR ticket is not active until it has a coordinator.

The coordinator is responsible for:

  • Ensuring the deprecation work happens

  • Following the DEPR workflow phases

  • Informing the community about any changes

  • Communicating if they are unable to continue coordinating the DEPR

When you are ready to coordinate the ticket, post a comment on the GitHub issue saying you’re doing so, and mark yourself as the GitHub issue’s owner. This can be done even if you don’t have write access to the ticket by making a comment on the ticket that says assign me.

Note

If the coordinator can no longer be responsible for the ticket, we ask that you find another coordinator if you can. If you can’t find another coordinator, it’s the responsibility of the current coordinator to:

  1. Remove yourself as the ticket owner

  2. Change the issue state back to Proposed

  3. Update stakeholders in whatever channel is appropriate

Now you are ready to communicate your proposal!

Communicated#

Announce#

Announce your proposal to deprecate and remove to the following communication channels.

To the Open edX Discourse#

Post a message to Open edX Discourse Deprecation Announcements, using the following template:

Subject: Deprecation/Removal: <Technology Name> <repo>#<issue-number>

Body:

Hi there,

We plan to deprecate and remove <Short description of the technology>.

Please read openedx/<repo>/issues/<issue-number> for more information and to post any questions/comments. The proposed deadline for comments before acceptance is <Target Accepted Date>.

Once the ticket is accepted, removal can happen at any time. If you believe that <technology name> is valuable and wish to make a case for retaining it and taking over its maintainership, please speak up on the issue linked above.

After acceptance, all future notifications around removal will only be posted to the GitHub issue, so be sure to watch that issue if you want further updates.

Thanks, <Your name>

Once the message is posted, include a link to the Discourse thread in the GitHub issue.

To openedx.slack.com#

The above discourse post should automatically get posted to the #open-edx-proposals and #general channels in the Open edX slack. If not, you could post something like:

Removal of <*Technology Name>:* We plan to deprecate and remove <Short description of the technology>.

Please read openedx/<repo>/issues/<issue-number> for more information and to post any questions/comments. The proposed deadline for comments before acceptance is <Target Accepted Date>. Once the ticket is accepted, removal can happen at any time.

Monitor Feedback#

Once announcements are made, update the GitHub Issue to the Communicated state and add a comment to the issue saying you’ve done so.

Allocate time to be responsive to any and all feedback and input on your DEPR ticket. Update the ticket and the proposal, if necessary, with any information that should be captured from the ongoing feedback. Continue to iterate and do this until the announced target Accepted date.

If during this time, there is a large amount of churn or concern, be open to adjusting the target dates and revisiting the proposal. If community alignment seems difficult, reach out to the Deprecation Working Group for directional guidance. In some cases, the proposal may need to be Abandoned entirely.

Accepted#

Once enough time is allotted for community feedback, all concerns on the DEPR ticket are responded to, and the target Accepted date has passed, update the state of the DEPR ticket to Accepted and make a comment on the issue saying you’ve done so.

For larger changes, it may be important to mention the upcoming deprecation in the release notes of the next named release.

Note

If there’s a new use of a feature once its deprecation ticket is accepted, then the contributor must provide an ADR justifying its usage. This is because using the deprecated feature obviously adds new technical debt to the system.

Deprecated#

If you decided to mark the code for deprecation during your Analyze or Monitor Feedback phases, invest time in doing so and update the state of the DEPR ticket to Deprecated once that is completed and make a comment on the issue saying you’ve done so.

Here are some common ways to mark a technology as deprecated:

  • REST API - Specify in the 1st line of the API’s docstring:

    "Deprecated <link-to-gh-issue>"
    
  • Javascript code - Add a log statement that executes once without being noisy:

    console.log("<Technology name> is deprecated. See <link-to-gh-issue>.")
    
  • Python code - Add a warnings.warn statement so it executes once without being noisy:

    warnings.warn("<Technology name> is deprecated. See <link-to-gh-issue>.", DeprecationWarning)
    
  • Feature toggles - Set the “Expiration Date” as described in OEP-17.

  • GitHub repo - See OEP-14 Archiving Open edX GitHub Repositories.

Removing#

When a team begins development work to remove the code, change the DEPR ticket’s state to Removing and make a comment on the issue saying you’ve done so.

During this phase, remember the following:

  • Implement the proposed and agreed upon migration path.

  • Remove related code from all places, including the frontend, APIs, and the backend, perhaps even in that order.

  • Remove any related documentation on docs.edx.org and elsewhere.

  • Continue to update the ticket with any delays or issues that may arise.

Removed#

When removal is complete:

  1. Add the DEPR ticket to the wiki page for the next Open edX named release to keep track of which removals occurred in which named release. Note: Eventually, this should be included in a .rst file bundled with the codebase.

  2. Update the DEPR ticket’s state to Removed and make a corresponding comment on the issue.

  3. Announce the removal in the #open-edx-proposals and #general Open edX slack channels.

  4. Optionally, celebrate with your team by banging and breaking a piñata!

../_images/pinata.png

References#

Change History#

2024-04-12#

  • Changed process to require every DEPR have a coordinator.

2023-06-21#

  • Update Discourse template to describe how to stay up to date on removal status.

2022-11-01#

  • Simplified guidance around the earliest named release and removal dates.

  • Clarify that proposal can be made while incomplete, as long as they aren’t communicated until they are complete.

2022-07-05#

  • Add suggestion for monitoring.

2022-06-03#

2022-02-08#

2021-04-26#

  • “Removal dates” has become synonymous with “Accepted dates”, so we are eliminating them to clarify the process.

  • Pull request #207

2021-02-05#

2019-01-02#

2018-11-02#

  • Describes the process in specific ordered steps for one to follow.

  • Adds references to external docs.

  • Updates information on the JIRA ticketing process.

  • Pull request #83

2018-06-21#