[SIP-74] Superset 2.0 Breaking Changes

[suddjian]

[SIP-74] Superset 2.0 Breaking Changes

Status: Voting

The following GitHub project can be used to track the status of individual proposed changes:
https://github.com/apache/superset/projects/35

Motivation

Superset has built up a large number of complex behaviors with the aim of maintaining backwards compatibility. As such complexities build up, bugs appear and development becomes more difficult. Superset 2.0 is an opportunity to release a clean API, spartan feature flag options, and far smaller volume of deprecated code.

Proposed Change

Breaking Changes List:

• Remove PUBLIC_ROLE_LIKE_GAMMA
• Turn on SQLLAB_BACKEND_PERSISTENCE by default
• Enable flag for disabling Legacy Datasource Editor.
• Enable D&D w/ hybrid by default
• Remove old import/export code and turn on the VERSIONED_EXPORT flag by default.
• remove SUPERSET_WORKERS
• Remove ROW_LEVEL_SECURITY feature flag (permanently set to true)
• Upgrade minimum supported Python version to 3.8.
• Remove deprecated Celery CLI.
• Remove legacy SIP_15_* logic/flags. This feature has been in flight for a number of years which is sufficient time for institutions to have upgraded.
• remove OLD_API_CHECK_DATASET_OWNERSHIP
• remove legacy ALERTS code and config keys
• Remove ENABLE_REACT_CRUD_VIEWS flag, and sweep up old code
• change the ENABLE_JAVASCRIPT_CONTROLS from a config value to a feature flag
• remove SQLALCHEMY_DOCS_URL and SQLALCHEMY_DISPLAY_TEXT from the config and move to superset_text.js so that it's in line with the rest of the customizable text options.
• generally remove all code marked as "deprecated" * is deprecated and will be removed in version 2.0.0

We should plan what breaking changes go into this release well in advance to make it as painless as possible.

The blue-sky ideal is that the only breaking changes will be removal of deprecated code and related feature flags. This way, the code we're shipping will already be mature and production-validated.

Large projects introducing breaking changes bring excessive risk. If we can mature them behind a feature flag first, then we can remove the flag for 2.0 to make it a small, lower-risk change. Changes that cannot be kept behind a feature flag should maintain backwards compatibility so they can go in a minor release instead of a major one.

Projects that do not relate to planned breaking changes should be able to continue development as normal.

Migration Plan and Compatibility

For Long Term Support of Superset 1.x, we should back-port critical fixes to a 1.x.x release. "Critical" here generally means security fixes and high-impact bugs that are able to be back-ported. Porting features, and many of the more complex bugfixes, isn't likely to be practical considering the fluidity of the Superset codebase. The LTS release should be supported until the next major version, at which point 2.x becomes the LTS release.

To begin the major version release process, we should split off a lts-v1 branch. From that point, master will be considered to be 2.x only. This will give us space to actually make the breaking changes, test, and fix any issues before building the release. Future 1.x changes can be merged to the lts-v1 branch, and we can build 1.x.x releases from there.

We may want to offer some sort of migration assistant that will analyze your Superset configuration and give you hints of how you should prepare for the upgrade.

Rejected Alternatives

Never bumping major versions?

[rusackas]

Thanks for kicking this off! I, for one, am enamored with the idea of NOT adding any big projects or committing to any huge bodies of work to pull this off, but rather just removing deprecated code paths, reducing feature flag complexity, and just cleaning house. We need to get over the hump of 2.0, and use it as practice in order to better plan for new initiatives for 3.0+ with reduced risk.

It may also be worth mentioning that 2.0 has a huge emotional weight to it, almost as much as 1.0. But if we follow semver practices, we shouldn't be so afraid of major version bumps, in my opinion. When we complete the monorepo migration (new SIP pending), for example, we may see major version bumps much more often due to things like plugin deprecations or similar migration efforts.

As for the tasks on the board, I think each should indeed be discussed individually to gather consensus on each. This could happen by converting the notes to Issues and discussing there, or taking it to a discussion thread on the dev@ list in hope of reaching lazy consensus.

If anyone here does not have committer access to add suggestions to the Project, please feel free to add them in a comment here for consideration, and we can add them if there's a reasonable degree of alignment/interest.

[dpgaspar]

This is great!

Removing old API endpoints, by first migrating them and dropping a deprecation warning on the old code, is a pattern we have been following that helps our users prepare and know in advance what to expect on 2.0. This can probably be replicated for deprecating the old Druid Connector, and on all the CRUD MVC views or even others.
Make 2.0 less frighting :)

[villebro]

This has been brought up before in the form of SIP-11, but I want to also propose deprecating the native Druid connector in Superset 2.0. The question is, do we also want to remove the option of providing a custom datasource by only supporting the SQLAlchemy connector going forward, or leave the BaseDatasource class in place, just in case we fall out of love with SQL at some point? I'd personally vote for removing BaseDatasource and replacing it with SqlaTable, as the base datasource class creates considerable maintenance overhead for the SQLA connector.

[dpgaspar]

This has been brought up before in the form of SIP-11, but I want to also propose deprecating the native Druid connector in Superset 2.0. The question is, do we also want to remove the option of providing a custom datasource by only supporting the SQLAlchemy connector going forward, or leave the BaseDatasource class in place, just in case we fall out of love with SQL at some point? I'd personally vote for removing BaseDatasource and replacing it with SqlaTable, as the base datasource class creates considerable maintenance overhead for the SQLA connector.

+1 for BaseDatasource removal

[suddjian]

just in case we fall out of love with SQL at some point

How could you say such a thing

[suddjian]

I've made a couple organizational changes to the GitHub project, so it can act as our consensus mechanism. The goal is to eventually kick off a vote on this SIP with a specific set of planned changes.

• Add new proposals to the "Please consider" column.
• Move a proposal a single column to the right to vote for it.
• Move a proposal from any column to the "Considered unsuitable" column if you think it is unsuitable for inclusion in 2.0.
• If you mark a proposal as unsuitable, also leave a comment explaining why, so we can discuss.
• If a proposal is in Considered Unsuitable, don't move it to another column without reaching some agreement.

[nytai]

+1 on adding more deprecation warning and messages around the removal of the native druid connector, before completely removing it. While we did set the flag DRUID_IS_ACTIVE defaulted false, we could have done more to notify users that native druid is on the deprecation path. In my experience, those deploying superset end up flipping the DRUID_IS_ACTIVE flag, and end users are never aware that they're using a deprecated connector and that they should migrate to druid SQL.

[suddjian]

Some of us had a meeting to quickly discuss and select items to propose. I'll outline below the reasons for the sub-proposals we moved to "considered unsuitable":

• Creating new import/export functionality -- This does not seem to describe a breaking change. New import/exportable entities could be implemented in any release.
• Remove ability to add FilterBox! Remove display of FilterBoxes? Remove Filterbox code? -- Migration / feature parity of native filters isn't quite proven out yet as much as we'd like for this
• Remove Legacy Druid Connector -- As discussed above, more deprecation warnings should be added before removing this

There is a particular breaking change we need to make that is somewhat urgent, so with the goal of speeding up the road to 2.0 we decided it'd be a good idea to focus primarily on including changes at the level of config and the cli. The following changes don't really relate to this area, so we decided they'd probably be best left for a later major release:

• Rename Pivot Table in favor of Pivot Table V2
• Remove Legacy controls (support Full D&D and Hybrid D&D)
• Remove the CRUD MVC views that have been re-implemented in React
• Remove Legacy Datasource Editor
• Remove react-select
• Remove the /csrf_token/ route from core.py, as instructed by a todo comment there

[michael-s-molina]

How long do we plan to support LTS releases? 18, 30 months?

[suddjian]

The LTS release should be supported until the next major version, at which point the "current" version will become LTS. LTS, in our case, means primarily applying security fixes and whatever bugfixes the community is willing to back-port. Porting features, and many of the more complex bugfixes, to the LTS version isn't really likely to be practical considering the fluidity of the Superset codebase.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.