[DEPR]: All Insights related code, configuration, docs and repositories

[kdmccormick]

(@bmtcril wrote this up originally)

This ticket has been accepted, and the various steps will be carried out in the following sub-tickets:

Sub-tickets

• [DEPR]: Mark Insights repos as deprecated and move to edx-unsupported #227
• [DEPR]: Generalize / remove Insights references #271

Proposal Date

2023-12-13

Ticket Acceptance Date

2024-01-24

Technology Removal Date

Ongoing

First Open edX Named Release Without This Functionality

This will not be tagged for Redwood, even if no other changes happen.

Rationale

Insights as a product has been largely unsupported for several years, and the lack of community uptake on the project offers little incentive for anyone to take over maintenance. Effectively, the software required to run the product is entirely out of support (the project runs on Python 2.7, requirements haven't been updated in 2-3 years, likely because most packages no longer support 2.7).

Even without a fully released replacement, we should not be suggesting this as a solution.

Note: This is different from the Insights repository that was deprecated in 2019: https://github.com/edx-unsupported/insights

Removal

Insights is a large project that touches many things. This list is known to be incomplete and will grow over the deprecation project, but should give a good cross-section of the work:

The following repositories should be moved to the edx-unsupported organization and clearly labeled as deprecated:

• edx-analytics-pipeline
• edx-analytics-dashboard
• edx-analytics-data-api
• edx-analytics-data-api-client
• edx-analytics-configuration
• edx-analytics-exporter

References to those projects and Insights itself exist in at least the following locations:

edx-platform has settings-gated links to Insights, especially in the Instructor Dashboard, we should make sure that all of those settings make sense for plugging in replacements systems. Settings in edx-platform are:

• ANALYTICS_API_KEY
• ANALYTICS_API_URL
• ANALYTICS_DASHBOARD_URL
• ANALYTICS_DASHBOARD_NAME

Additionally, calls to the Analytics API should be removed or abstracted such that they can feasibly be reused by replacement systems.

The following repositories (and possibly more) have references to Insights that need to be understood, and made generic enough for replacements to use or (eventually) removed:

• edx-enterprise-data
• frontend-app-admin-portal
• paragon
• tubular
• devstack
• configuration
• edx-documentation

Localized strings exist in at least edx-platform and ecommerce that need to be revisited or removed.

Replacement

There are community based replacement systems such as Cairn, and home grown solutions that providers have been using instead of Insights for years. While the new Aspects system is not yet generally available, it should be by the time this ticket is ready for work.

Even if there were no viable replacements, it is neither reasonable nor safe to suggest that Insights, in its current state, is a viable solution for operators to adopt.

Deprecation

Given the scope of the work of this project and potential difficulties of migration for large operators that rely on Insights, a tiered deprecation is suggested, in this order:

• Complete a survey of existing use cases in edx-enterprise-data and frontend-app-admin-portal (and any other relevant places that come up) that seem to have hard dependencies on Insights, determine a course of action for each (deprecate, generalize, remove the links, etc)
• Remove Insights from Open edX releases
• Move the repositories listed above to edx-unsupported and clearly label them as deprecated. They should still be usable, but it should also be made clear that they are at-your-own-risk.
• Update documentation references to clearly state that they are historical / deprecated, or remove
• Complete generalization work such as replacing hard coded links or URL paths with settings

This is where breaking changes start...

• Complete removal work of things which cannot be generalized
• Remove remaining references in operational repos such as tubular, devstack, and configuration

Migration

For almost all installations no migration will be necessary. For those using Insights the migration will depend on whether they wish to continue using it or move to a new system. The steps for migrating to different new systems are outside the scope of this DEPR. The rest of these steps outline the process of keeping Insights running while this ticket is being worked on, and beyond.

During the non-breaking-changes steps the migrations would be things like updating settings / setting names. Deployment steps for the project may need to change if the deployment method doesn't follow GitHub redirects. Documentation links may need to be updated to point to provider-specific locations.

For breaking changes, most things in tubular, devstack, and configuration are either not working/unused or able to be pulled out of those locations and run from different locations.

Removal of code paths which cannot be generalized (such as explicit calls to analytics-api) can be delayed until suitable pluggability functionality can be added, or the code can be forked to support the use case if the Insights functionality is critical to the repository.

Additional Info

See #36 for additional context

[bmtcril]

This has had no dissenting voices in the comment period and has been moved to accepted. Next step is to write up a more detailed plan!

[kdmccormick]

@bmtcril It's likely that these repositories:

• tubular
• devstack
• configuration

will be deprecated out of the openedx org in the near-to-medium future, and then continued to be used by 2U on a fork. So, we could probably leave the Insights-related stuff in those repositories for now, which would save us some time, and save 2U time since they wouldn't need to figure out how to plug the Insights stuff back into those.

[pomegranited]

@kdmccormick @bmtcril

• tubular

The community still needs this one.. it runs the user retirement pipeline, cf user_retirement_pipeline/README, tutor-contrib-retirement.

And I'm happy to help out with this deprecation where needed, please let me know :)

[bmtcril]

@pomegranited @feanil is heading that up, but the scripts are moving here: openedx/edx-platform#34063

[feanil]

FYI, I posted hastexo/tutor-contrib-retirement#32 and we're not currently planning to update configuration as a part of the retirement code move so if that's valuable someone else should take that on.

[robrap]

This is a private link to a 2U ticket to look into this DEPR. I'm just noting it in case there are any updates that others in 2U want to communicate back to the team that will eventually be looking into this.

[MichaelRoytman]

Hello from 2U 👋!

As Robert mentioned, we'll be looking into how 2U adapts to these changes shortly. I'll be the point person for the discovery. Because the linked Jira ticket is private, please feel free to tag me on any relevant tickets or if you need my attention on anything. I'll be following this ticket for any updates. I'll be reviewing this DEPR ticket over the next few days, and, over the longer term, working to formulate a plan for the future of an insights product at 2U.

I'm anticipating having a lot of questions about this project. Is this the right place to leave them?

Thank you!

[bmtcril]

Hey Michael! The initial work is in the sub-ticket: #227 so you may wish to follow that one as well. Here is a fine place to discuss, or #wg-depr-slash-n-burn in Slack. I'm happy to do a synchronous meeting as well if that helps anything.

[HassanJaveed84]

Note that these repositories are still needed to support partner contractual obligations.
We have running pipelines which make use of these and are unrelated to insights.

• edx-analytics-pipeline
• edx-analytics-configuration
• edx-analytics-exporter

[bmtcril]

Hey @HassanJaveed84 ! I think your best bet is to fork them back into the edx org and run from there. If we move them to the openedx-unsupported org, as is currently the plan, it's likely that the pipelines will continue to work if they follow the Github redirects or are pulling from PyPI, but obviously changes can't be made there.

[kdmccormick]

@bmtcril Is this good to close out, or was there post-archival cleanup that you wanted to track with this DEPR ticket?

[bmtcril]

From the list above, these things still need to happen:

• Complete generalization work such as replacing hard coded links or URL paths with settings
• Complete removal work of things which cannot be generalized

There is a pre-task of trying to identify all of those places, but I'd rather not close this one out until those are done.

[bmtcril]

Note: There are a number of places with historical or references to old code that can stay, or are used in test data. All references in these places seem fine to remain:

• event-routing-backends
• docs.openedx.org
• XBlock
• aspects-dbt
• openedx-aspects
• openedx-translations
• edx-repo-health
• openedx-webhooks
• openedx-webhooks-data