Migrate ckanext namespace package to PEP420 (or away from namespace packages)

[amercader]

Initially discussed in #8382 (comment).

The ckanext package used by plugins is a namespace package. CKAN currently uses a deprecated setuptools pkg_resources-based way of defining namespaces packages. The recommendation seems to be to start migrating away from this method.

• The modern way, know as native namespace packages or PEP 420
• @smotornyuk suggested options for the transition, including migrating to non-namespace packages.

To gather a bit more information my suggested plan is:

1. Migrate CKAN itself to native namespace packages (there are plenty of ckanext extensions in CKAN core)
2. Migrate a couple of extensions to native namespace packages
3. Run a suite of tests similar to these ones but tailored to our usual setups to identify the problematic combinations

I started work on 1 in the pep420-namespace-packages branch (it also includes #8456) and it is not as straightforward as it seems. Switching the namespace package method changes imports in subtle ways, that are reflected in seemingly unrelated test failures. For instance the last ones are caused by trying to capture an exception class defined in one of the tests modules (AuthTestException). Inspecting this in the work branch gives us:

<class 'datastore.tests.test_chained_auth_functions.AuthTestException'>

While on master:

<class 'ckanext.datastore.tests.test_chained_auth_functions.AuthTestException'>

Other worrying signs include warnings like this one, which don't occur in master:

python -c "from ckanext.datastore import helpers"
/home/adria/dev/pyenvs/pypackaging4/ckan/ckan/plugins/core.py:10: UserWarning: Module ckanext was already imported from None, but /home/adria/dev/pyenvs/pypackaging4/ckan is being added to sys.path
  from pkg_resources import iter_entry_points

Quick tests in extensions for point 2 also resulted in errors
.

All this to say that more investigation is required in order to plan a transition. Any suggestions from users with experience around these topics is appreciated.

[wardi]

To migrate an existing package, all packages sharing the namespace must be migrated simultaneously.

Should we create a new namespace for extensions to allow for the transition? e.g. ckanext for legacy pkg_resources-namespace and ckanx/ ckanext3 or something for the new way?