v14.0.0: upgrade to Nutmeg

- 💥 [Feature] Upgrade to Nutmeg: (by @regisb)
    - 💥 [Feature] Persistent grades are now enabled by default.
    - [Bugfix] Remove edX references from bulk emails ([issue](openedx/wg-build-test-release#100)).
    - [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
    - [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running

CHANGELOG.md

@@ -18,6 +18,16 @@ Every user-facing change should have an entry in this changelog. Please respect
 
 ## Unreleased
 
+## v14.0.0 (2022-06-09)
+
+- 💥[Feature] Upgrade to Nutmeg: (by @regisb)
+  - 💥[Feature] Hide a course from the `/course` search page in the LMS when the course visibility is set to "none" in the Studio. (thanks @ghassanmas!)
+  - 💥[Improvement] The `lms.env.json` and `cms.env.json` files are moved to `lms.env.yml` and `cms.env.yml`. As a consequence, plugin developers must reformat the following patches to use YAML format, and not JSON: "common-env-features", "lms-env-features", "cms-env-features", "lms-env", "cms-env", "openedx-auth".
+  - 💥[Feature] Persistent grades are now enabled by default.
+  - [Bugfix] Remove edX references from bulk emails ([issue](https://github.com/openedx/build-test-release-wg/issues/100)).
+  - [Improvement] For Tutor Nightly (and only Nightly), official plugins are now installed from their nightly branches on GitHub instead of a version range on PyPI. This will allow Nightly users to install all official plugins by running ``pip install -e ".[full]"``.
+  - [Bugfix] Start MongoDB when running migrations, because a new data migration fails if MongoDB is not running
+
 ## v13.3.1 (2022-06-06)
 
 - [Fix] Crashing celery workers in development (#681). (by @regisb)

docs/configuration.rst

@@ -8,7 +8,7 @@ Tutor offers plenty of possibilities for platform customisation out of the box.
 a. Modifying the Tutor :ref:`configuration parameters <configuration>`.
 b. Modifying the :ref:`Open edX docker image <customise>` that runs the Open edX platform.
 
-This section does not cover :ref:`plugin development <plugins>`. For simple changes, such as modifying the ``*.env.json`` files or the edx-platform settings, *you should not fork edx-platform or tutor*! Instead, you should create a simple :ref:`plugin for Tutor <plugins_yaml>`.
+This section does not cover :ref:`plugin development <plugins>`. For simple changes, such as modifying the ``*.env.yml`` files or the edx-platform settings, *you should not fork edx-platform or tutor*! Instead, you should create a simple :ref:`plugin for Tutor <plugins_yaml>`.
 
 .. _configuration:
 
@@ -31,7 +31,7 @@ Or from the system environment::
 
     export TUTOR_PARAM1=VALUE1
 
-Once the base configuration is created or updated, the environment is automatically re-generated. The environment is the set of all files required to manage an Open edX platform: Dockerfile, ``lms.env.json``, settings files, etc. You can view the environment files in the ``env`` folder::
+Once the base configuration is created or updated, the environment is automatically re-generated. The environment is the set of all files required to manage an Open edX platform: Dockerfile, ``lms.env.yml``, settings files, etc. You can view the environment files in the ``env`` folder::
 
     ls "$(tutor config printroot)/env"
 
@@ -83,7 +83,7 @@ Open edX customisation
 
 This defines the git repository from which you install Open edX platform code. If you run an Open edX fork with custom patches, set this to your own git repository. You may also override this configuration parameter at build time, by providing a ``--build-arg`` option.
 
-- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/maple.3"``)
+- ``OPENEDX_COMMON_VERSION`` (default: ``"open-release/nutmeg.1"``)
 
 This defines the default version that will be pulled from all Open edX git repositories.
 
@@ -294,16 +294,16 @@ Note that your edx-platform version must be a fork of the latest release **tag**
 
 If you don't create your fork from this tag, you *will* have important compatibility issues with other services. In particular:
 
-- Do not try to run a fork from an older (pre-Maple) version of edx-platform: this will simply not work.
+- Do not try to run a fork from an older (pre-Nutmeg) version of edx-platform: this will simply not work.
 - Do not try to run a fork from the edx-platform master branch: there is a 99% probability that it will fail.
-- Do not try to run a fork from the open-release/maple.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/maple.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/maple.master branch.
+- Do not try to run a fork from the open-release/nutmeg.master branch: Tutor will attempt to apply security and bug fix patches that might already be included in the open-release/nutmeg.master but which were not yet applied to the latest release tag. Patch application will thus fail if you base your fork from the open-release/nutmeg.master branch.
 
 .. _i18n:
 
 Adding custom translations
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/maple.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
+If you are not running Open edX in English, chances are that some strings will not be properly translated. In most cases, this is because not enough contributors have helped translate Open edX into your language. It happens! With Tutor, available translated languages include those that come bundled with `edx-platform <https://github.com/openedx/edx-platform/tree/open-release/nutmeg.master/conf/locale>`__ as well as those from `openedx-i18n <https://github.com/openedx/openedx-i18n/tree/master/edx-platform/locale>`__.
 
 Tutor offers a relatively simple mechanism to add custom translations to the openedx Docker image. You should create a folder that corresponds to your language code in the "build/openedx/locale" folder of the Tutor environment. This folder should contain a "LC_MESSAGES" folder. For instance::
 
@@ -324,9 +324,9 @@ Then, add a "django.po" file there that will contain your custom translations::
 .. warning::
     Don't forget to specify the file ``Content-Type`` when adding message strings with non-ASCII characters; otherwise a ``UnicodeDecodeError`` will be raised during compilation.
 
-The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/openedx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/django.po>`__.
+The "String to translate" part should match *exactly* the string that you would like to translate. You cannot make it up! The best way to find this string is to copy-paste it from the `upstream django.po file for the English language <https://github.com/openedx/edx-platform/blob/open-release/nutmeg.master/conf/locale/en/LC_MESSAGES/django.po>`__.
 
-If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/openedx/edx-platform/blob/open-release/maple.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
+If you cannot find the string to translate in this file, then it means that you are trying to translate a string that is used in some piece of javascript code. Those strings are stored in a different file named "djangojs.po". You can check it out `in the edx-platform repo as well <https://github.com/openedx/edx-platform/blob/open-release/nutmeg.master/conf/locale/en/LC_MESSAGES/djangojs.po>`__. Your custom javascript strings should also be stored in a "djangojs.po" file that should be placed in the same directory.
 
 To recap, here is an example. To translate a few strings in French, both from django.po and djangojs.po, we would have the following file hierarchy::
 

docs/dev.rst

@@ -92,7 +92,7 @@ The ``openedx-dev`` Docker image is based on the same ``openedx`` image used by
 
 - Additional Python and system requirements are installed for convenient debugging: `ipython <https://ipython.org/>`__, `ipdb <https://pypi.org/project/ipdb/>`__, vim, telnet.
 
-- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/maple.master/requirements/edx/development.in>`__ are installed.
+- The edx-platform `development requirements <https://github.com/openedx/edx-platform/blob/open-release/nutmeg.master/requirements/edx/development.in>`__ are installed.
 
 
 If you are using a custom ``openedx`` image, then you will need to rebuild ``openedx-dev`` every time you modify ``openedx``. To so, run::

docs/faq.rst

@@ -38,7 +38,7 @@ The `devstack <https://github.com/openedx/devstack>`_ is meant for development o
 Is Tutor officially supported by edX?
 -------------------------------------
 
-Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation method for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/installation/index.html>`__. 
+Yes: as of the Open edX Maple release (December 9th 2021), Tutor is the only officially supported installation method for Open edX: see the `official installation instructions <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-nutmeg.master/installation/index.html>`__. 
 
 What features are missing from Tutor?
 -------------------------------------

docs/install.rst

@@ -121,11 +121,11 @@ Major Open edX releases are published twice a year, in June and December, by the
 4. Test the new release in a sandboxed environment.
 5. If you are running edx-platform, or some other repository from a custom branch, then you should rebase (and test) your changes on top of the latest release tag (see :ref:`edx_platform_fork`).
 
-The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``quickstart`` command (see above). The single difference is that if the ``quickstart`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``quickstart``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Lilac to Maple and rebuild some Docker images, run::
+The process for upgrading from one major release to the next works similarly to any other upgrade, with the ``quickstart`` command (see above). The single difference is that if the ``quickstart`` command detects that your tutor environment was generated with an older release, it will perform a few release-specific upgrade steps. These extra upgrade steps will be performed just once. But they will be ignored if you updated your local environment (for instance: with ``tutor config save``) before running ``quickstart``. This situation typically occurs if you need to re-build some Docker images (see above). In such a case, you should make use of the ``upgrade`` command. For instance, to upgrade a local installation from Maple to Nutmeg and rebuild some Docker images, run::
 
     tutor config save
     tutor images build all # list the images that should be rebuilt here
-    tutor local upgrade --from=lilac
+    tutor local upgrade --from=maple
     tutor local quickstart
 
 .. _autocomplete:

docs/plugins/intro.rst

@@ -4,7 +4,7 @@
 Introduction
 ============
 
-Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.json`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
+Tutor comes with a plugin system that allows anyone to customise the deployment of an Open edX platform very easily. The vision behind this plugin system is that users should not have to fork the Tutor repository to customise their deployments. For instance, if you have created a new application that integrates with Open edX, you should not have to describe how to manually patch the platform settings, ``urls.py`` or ``*.env.yml`` files. Instead, you can create a "tutor-myapp" plugin for Tutor. Then, users will start using your application in three simple steps::
 
     # 1) Install the plugin
     pip install tutor-myapp

docs/quickstart.rst

@@ -22,7 +22,7 @@ Yes :) This is what happens when you run ``tutor local quickstart``:
 2. Configuration files are generated from templates.
 3. Docker images are downloaded.
 4. Docker containers are provisioned.
-5. A full, production-ready Open edX platform (`Maple <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-maple.master/platform_releases/maple.html>`__ release) is run with docker-compose.
+5. A full, production-ready Open edX platform (`Nutmeg <https://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/open-release-nutmeg.master/platform_releases/nutmeg.html>`__ release) is run with docker-compose.
 
 The whole procedure should require less than 10 minutes, on a server with good bandwidth. Note that your host environment will not be affected in any way, since everything runs inside docker containers. Root access is not even necessary.
 

docs/reference/patches.rst

@@ -53,21 +53,21 @@ File: ``apps/caddy/Caddyfile``
 ``cms-env``
 ===========
 
-File: ``apps/openedx/config/cms.env.json``
+File: ``apps/openedx/config/cms.env.yml``
 
 .. patch:: cms-env-features
 
 ``cms-env-features``
 ====================
 
-File: ``apps/openedx/config/cms.env.json``
+File: ``apps/openedx/config/cms.env.yml``
 
 .. patch:: common-env-features
 
 ``common-env-features``
 =======================
 
-Files: ``apps/openedx/config/cms.env.json``, ``apps/openedx/config/lms.env.json``
+Files: ``apps/openedx/config/cms.env.yml``, ``apps/openedx/config/lms.env.yml``
 
 .. patch:: dev-docker-compose-jobs-services
 
@@ -137,14 +137,14 @@ File: ``kustomization.yml``
 ``lms-env``
 ===========
 
-File: ``apps/openedx/config/lms.env.json``
+File: ``apps/openedx/config/lms.env.yml``
 
 .. patch:: lms-env-features
 
 ``lms-env-features``
 ====================
 
-File: ``apps/openedx/config/lms.env.json``
+File: ``apps/openedx/config/lms.env.yml``
 
 .. patch:: local-docker-compose-caddy-aliases
 
@@ -200,7 +200,7 @@ File: ``local/docker-compose.yml``
 ``openedx-auth``
 ================
 
-File: ``apps/openedx/config/partials/auth.json``
+File: ``apps/openedx/config/partials/auth.yml``
 
 .. patch:: openedx-cms-common-settings
 

requirements/plugins.txt

@@ -1,12 +1,12 @@
-# change version ranges when upgrading from maple
-tutor-android>=13.0.0,<14.0.0
-tutor-discovery>=13.0.0,<14.0.0
-tutor-ecommerce>=13.0.0,<14.0.0
-tutor-forum>=13.0.0,<14.0.0
-tutor-license>=13.0.0,<14.0.0
-tutor-mfe>=13.0.0,<14.0.0
-tutor-minio>=13.0.0,<14.0.0
-tutor-notes>=13.0.0,<14.0.0
-tutor-richie>=13.0.0,<14.0.0
-tutor-webui>=13.0.0,<14.0.0
-tutor-xqueue>=13.0.0,<14.0.0
+# change version ranges when upgrading from nutmeg
+tutor-android>=14.0.0,<15.0.0
+tutor-discovery>=14.0.0,<15.0.0
+tutor-ecommerce>=14.0.0,<15.0.0
+tutor-forum>=14.0.0,<15.0.0
+tutor-license>=14.0.0,<15.0.0
+tutor-mfe>=14.0.0,<15.0.0
+tutor-minio>=14.0.0,<15.0.0
+tutor-notes>=14.0.0,<15.0.0
+tutor-richie>=14.0.0,<15.0.0
+tutor-webui>=14.0.0,<15.0.0
+tutor-xqueue>=14.0.0,<15.0.0

tests/test_env.py

@@ -258,6 +258,6 @@ def test_current_version_in_latest_env(self) -> None:
             ) as f:
                 f.write(__version__)
             self.assertEqual(__version__, env.current_version(root))
-            self.assertEqual("maple", env.get_env_release(root))
+            self.assertEqual("nutmeg", env.get_env_release(root))
             self.assertIsNone(env.should_upgrade_from_release(root))
             self.assertTrue(env.is_up_to_date(root))

tutor/__about__.py

@@ -2,7 +2,7 @@
 
 # Increment this version number to trigger a new release. See
 # docs/tutor.html#versioning for information on the versioning scheme.
-__version__ = "13.3.1"
+__version__ = "14.0.0"
 
 # The version suffix will be appended to the actual version, separated by a
 # dash. Use this suffix to differentiate between the actual released version and

tutor/commands/k8s.py

@@ -468,7 +468,7 @@ def wait(context: K8sContext, name: str) -> None:
 @click.option(
     "--from",
     "from_release",
-    type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
+    type=click.Choice(["ironwood", "juniper", "koa", "lilac", "maple"]),
 )
 @click.pass_context
 def upgrade(context: click.Context, from_release: Optional[str]) -> None:

tutor/commands/local.py

@@ -154,7 +154,7 @@ def quickstart(
 @click.option(
     "--from",
     "from_release",
-    type=click.Choice(["ironwood", "juniper", "koa", "lilac"]),
+    type=click.Choice(["ironwood", "juniper", "koa", "lilac", "maple"]),
 )
 @click.pass_context
 def upgrade(context: click.Context, from_release: t.Optional[str]) -> None:

tutor/commands/upgrade/k8s.py

@@ -1,4 +1,5 @@
 from tutor import config as tutor_config
+from tutor import env as tutor_env
 from tutor import fmt
 from tutor.commands import k8s
 from tutor.commands.context import Context
@@ -27,6 +28,10 @@ def upgrade_from(context: Context, from_release: str) -> None:
         upgrade_from_lilac(config)
         running_release = "maple"
 
+    if running_release == "maple":
+        upgrade_from_maple(context, config)
+        running_release = "nutmeg"
+
 
 def upgrade_from_ironwood(config: Config) -> None:
     if not config["RUN_MONGODB"]:
@@ -102,3 +107,42 @@ def upgrade_from_lilac(config: Config) -> None:
         "upgrade from Lilac to Maple"
     )
     k8s.delete_resources(config, resources=["deployments", "services"])
+
+
+def upgrade_from_maple(context: Context, config: Config) -> None:
+    fmt.echo_info("Upgrading from Maple")
+    # The environment needs to be updated because the backpopulate/backfill commands are from Nutmeg
+    tutor_env.save(context.root, config)
+
+    # Start mysql
+    k8s.kubectl_apply(
+        context.root,
+        "--selector",
+        "app.kubernetes.io/name=mysql",
+    )
+    k8s.wait_for_pod_ready(config, "mysql")
+
+    # lms upgrade
+    k8s.kubectl_apply(
+        context.root,
+        "--selector",
+        "app.kubernetes.io/name=lms",
+    )
+    k8s.wait_for_pod_ready(config, "lms")
+    k8s.kubectl_exec(
+        config, "lms", ["sh", "-e", "-c", "./manage.py lms backpopulate_user_tours"]
+    )
+
+    # cms upgrade
+    k8s.kubectl_apply(
+        context.root,
+        "--selector",
+        "app.kubernetes.io/name=cms",
+    )
+    k8s.wait_for_pod_ready(config, "cms")
+    k8s.kubectl_exec(
+        config, "cms", ["sh", "-e", "-c", "./manage.py cms backfill_course_tabs"]
+    )
+    k8s.kubectl_exec(
+        config, "cms", ["sh", "-e", "-c", "./manage.py cms simulate_publish"]
+    )