Merge branch 'main' into recipe_test_workflow_prototype

.github/workflows/test.yml

@@ -36,12 +36,24 @@ jobs:
         run: |
           mamba --version 2>&1 | tee test_linux_artifacts_python_${{ matrix.python-version }}/conda_version.txt
           python -V 2>&1 | tee test_linux_artifacts_python_${{ matrix.python-version }}/python_version.txt
+          # this is how to export variables to the GITHUB var environment
+          echo "pver0=$(python -V)" >> $GITHUB_ENV
       - name: Install ESMValTool
         shell: bash -l {0}
         run: pip install -e .[develop] 2>&1 | tee test_linux_artifacts_python_${{ matrix.python-version }}/install.txt
       - name: Install Julia dependencies
         shell: bash -l {0}
         run: esmvaltool install Julia
+      - name: Export Python minor version
+        shell: bash -l {0}
+        run: echo "pver1=$(python -V)" >> $GITHUB_ENV
+      - name: Exit if Python minor version changed
+        shell: bash -l {0}
+        if: ${{ env.pver1 != env.pver0}}
+        run: |
+          echo "Python minor version changed after Julia install"
+          python -V
+          exit 1
       - name: Run flake8
         shell: bash -l {0}
         run: flake8

README.md

@@ -15,7 +15,7 @@
 
 - [**Documentation**](https://docs.esmvaltool.org/en/latest/)
 - [**ESMValTool Website**](https://www.esmvaltool.org/)
-- [**ESMValTool Tutorial**](https://esmvalgroup.github.io/ESMValTool_Tutorial/index.html)
+- [**ESMValTool Tutorial**](https://tutorial.esmvaltool.org/index.html)
 - [**ESMValGroup Project on GitHub**](https://github.com/ESMValGroup)
 - [**Gallery**](https://docs.esmvaltool.org/en/latest/gallery.html)
 - [**`conda-forge` package feedstock**](https://github.com/conda-forge/esmvaltool-suite-feedstock)
@@ -53,7 +53,7 @@ ESMValTool can run with the following types of [data as input](https://docs.esmv
 
 # Getting started
 
-Please see [getting started](https://docs.esmvaltool.org/en/latest/quickstart/index.html) on readthedocs as well as [ESMValTool tutorial](https://esmvalgroup.github.io/ESMValTool_Tutorial/index.html). The tutorial is a set of lessons that together teach skills needed to work with ESMValTool in climate-related domains.
+Please see [getting started](https://docs.esmvaltool.org/en/latest/quickstart/index.html) on readthedocs as well as [ESMValTool tutorial](https://tutorial.esmvaltool.org/index.html). The tutorial is a set of lessons that together teach skills needed to work with ESMValTool in climate-related domains.
 
 ## Getting help
 

doc/sphinx/source/api/esmvaltool.diag_scripts.seaborn_diag.rst

@@ -0,0 +1,9 @@
+.. _api.esmvaltool.diag_scripts.seaborn_diag:
+
+Seaborn Diagnostic
+==================
+
+.. automodule:: esmvaltool.diag_scripts.seaborn_diag
+   :no-members:
+   :no-inherited-members:
+   :no-show-inheritance:

doc/sphinx/source/api/esmvaltool.rst

@@ -28,3 +28,4 @@ Diagnostic Scripts
    esmvaltool.diag_scripts.monitor
    esmvaltool.diag_scripts.ocean
    esmvaltool.diag_scripts.psyplot_diag
+   esmvaltool.diag_scripts.seaborn_diag

doc/sphinx/source/community/diagnostic.rst

@@ -226,6 +226,7 @@ It is also possible to add more information for the implemented diagnostics usin
 - :code:`domains` a list of spatial coverage of the dataset
 - :code:`plot_types` a list of plot types if the diagnostic created a plot, e.g. error bar
 - :code:`statistics` a list of types of the statistic, e.g. anomaly
+- :code:`long_names` a list of long names of used variables, e.g. Air Temperature
 
 Arbitrarily named other items are also supported.
 

doc/sphinx/source/community/release_strategy/release_strategy.rst

@@ -81,6 +81,14 @@ Upcoming releases
 Past releases
 ^^^^^^^^^^^^^
 
+- 2.8.1 (Bugfix, Release Manager: `Valeriu Predoi`_)
+
++------------+---------------------------------------------------------------------------------------------+------------------------------------+
+|    Done    |                                            Event                                            |             Changelog              |
++============+=============================================================================================+====================================+
+| 2023-06-02 | `ESMValCore Release 2.8.1 <https://github.com/ESMValGroup/ESMValCore/releases/tag/v2.8.1>`_ | :ref:`esmvalcore:changelog-v2-8-1` |
++------------+---------------------------------------------------------------------------------------------+------------------------------------+
+
 - 2.8.0 (Release Manager: `Rémi Kazeroni`_)
 
 +------------+------------+---------------------------------------------------------------------------------------------+------------------------------------+

doc/sphinx/source/develop/recipe.rst

@@ -17,6 +17,6 @@ the specific parameters needed by a recipe that runs a personal diagnostic are:
 i.e. the full path to the personal diagnostic that the user needs to run.
 
 There is also a lesson available in the 
-`ESMValTool tutorial <https://esmvalgroup.github.io/ESMValTool_Tutorial/>`_
+`ESMValTool tutorial <https://tutorial.esmvaltool.org/>`_
 that describes in a step-by-step procedure how to write your own recipe. It can be found
-`here <https://esmvalgroup.github.io/ESMValTool_Tutorial/06-preprocessor/index.html>`_.
+`here <https://tutorial.esmvaltool.org/06-preprocessor/index.html>`_.

doc/sphinx/source/faq.rst

@@ -121,3 +121,10 @@ Moreover, recipe :ref:`recipes_psyplot_diag` and the corresponding diagnostic
 :ref:`psyplot_diag.py <api.esmvaltool.diag_scripts.psyplot_diag>` provide a
 high-level interface to the `Psyplot <https://psyplot.github.io/>`__ package
 which can be used to create a large variety of different plots.
+
+Similarly, recipe :ref:`recipes_seaborn_diag` and the corresponding diagnostic
+:ref:`seaborn_diag.py <api.esmvaltool.diag_scripts.seaborn_diag>` provide a
+high-level interface to the `Seaborn <https://seaborn.pydata.org>`__ package
+which can also be used to create a large variety of different plots.
+
+See also :ref:`general_purpose_diags`.

doc/sphinx/source/input.rst

@@ -358,7 +358,7 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 | MERRA2                       | sm (Lmon)                                                                                            |   3  | Python          |
 |                              | clt, pr, evspsbl, hfss, hfls, huss, prc, prsn, prw, ps, psl, rlds, rldscs, rlus, rlut, rlutcs, rsds, |      |                 |
 |                              | rsdscs, rsdt, tas, tasmin, tasmax, tauu, tauv, ts, uas, vas, rsus, rsuscs, rsut, rsutcs, ta, ua, va, |      |                 |
-|                              | tro3, zg, hus, wap, hur (Amon)                                                                       |      |                 |
+|                              | tro3, zg, hus, wap, hur, cl, clw, cli, clwvi, clivi (Amon)                                           |      |                 |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | MLS-AURA                     | hur, hurStderr (day)                                                                                 |   3  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+

doc/sphinx/source/quickstart/configuration.rst

@@ -23,6 +23,6 @@ the ``esmvaltool`` command can find the data on your system, before it can run
 a recipe.
 
 There is a lesson available in the
-`ESMValTool tutorial <https://esmvalgroup.github.io/ESMValTool_Tutorial/>`_
+`ESMValTool tutorial <https://tutorial.esmvaltool.org/>`_
 that describes how to personalize the configuration file. It can be found
-`at this site <https://esmvalgroup.github.io/ESMValTool_Tutorial/03-configuration/index.html>`_.
+`at this site <https://tutorial.esmvaltool.org/03-configuration/index.html>`_.

doc/sphinx/source/quickstart/installation.rst

@@ -37,10 +37,10 @@ The next sections will detail the procedure to install ESMValTool through each
 of these methods.
 
 There is also a lesson available in the
-`ESMValTool tutorial <https://esmvalgroup.github.io/ESMValTool_Tutorial/>`_
+`ESMValTool tutorial <https://tutorial.esmvaltool.org/>`_
 that describes the installation of the ESMValTool in more detail.
 It can be found
-`here <https://esmvalgroup.github.io/ESMValTool_Tutorial/02-installation/index.html>`_.
+`here <https://tutorial.esmvaltool.org/02-installation/index.html>`_.
 
 See `common installation issues`_ if you run into trouble.
 
@@ -510,11 +510,11 @@ estate, so there is no need to install ESMValTool if you are just running recipe
  - Met Office: `esmvaltool` is available on the Linux estate after login and module loading via `module load`;
    see the ESMValTool Community of Practice SharePoint site for more details.
 
-The ESMValTool Tutorial provides a `quickstart guide <https://esmvalgroup.github.io/ESMValTool_Tutorial/01-quickstart/index.html>`__
+The ESMValTool Tutorial provides a `quickstart guide <https://tutorial.esmvaltool.org/01-quickstart/index.html>`__
 that is particularly suited for new users that have an access to pre-installed version of ESMValTool.
 
 Information on how to request an account at CEDA-JASMIN and DKRZ-Levante and to get started with these HPC clusters
-can be found on the setup page of the tutorial `here <https://esmvalgroup.github.io/ESMValTool_Tutorial/setup.html>`__.
+can be found on the setup page of the tutorial `here <https://tutorial.esmvaltool.org/setup.html>`__.
 
 .. _install_with_docker:
 

doc/sphinx/source/quickstart/running.rst

@@ -14,9 +14,9 @@ Running your first recipe
 =========================
 
 There is a step-by-step tutorial available in the
-`ESMValTool tutorial <https://esmvalgroup.github.io/ESMValTool_Tutorial/>`_
+`ESMValTool tutorial <https://tutorial.esmvaltool.org/>`_
 on how to run your first recipe. It can be found
-`here <https://esmvalgroup.github.io/ESMValTool_Tutorial/04-recipe/index.html>`_.
+`here <https://tutorial.esmvaltool.org/04-recipe/index.html>`_.
 
 An
 `example recipe <https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/recipes/examples/recipe_python.yml>`_

doc/sphinx/source/recipes/broken_recipe_list.rst

@@ -16,10 +16,6 @@ More details can be found in the :ref:`broken recipe policy
      - Affected diagnostics
      - Problem
      - GitHub issue 
-   * - :ref:`recipe_autoassess_landsurface_soilmoisture.yml <recipe_autoassess_landsurface_soilmoisture.rst>`
-     - All
-     - Dependency on some external climatology files
-     - `#2309 <https://github.com/ESMValGroup/ESMValTool/issues/2309>`_
    * - `recipe_check_obs.yml`
      - `ERA5_native6`
      - Derivation of custom variables `rlus` and `rsus`

doc/sphinx/source/recipes/index.rst

@@ -10,6 +10,21 @@ ESMValTool for all available recipes can be accessed `here
 .. toctree::
    :maxdepth: 1
 
+.. _general_purpose_diags:
+
+General-purpose diagnostics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Recipes that use highly customizable diagnostics which are designed to plot a
+large variety of input data.
+
+.. toctree::
+   :maxdepth: 1
+
+   recipe_monitor
+   recipe_psyplot
+   recipe_seaborn
+
 Atmosphere
 ^^^^^^^^^^
 .. toctree::
@@ -121,9 +136,8 @@ Other
    recipe_ensclus
    recipe_esacci_lst
    recipe_examples
-   recipe_monitor
+   recipe_galytska23jgr
    recipe_multimodel_products
-   recipe_psyplot
    recipe_pv_capacity_factor
    recipe_rainfarm
    recipe_seaice

doc/sphinx/source/recipes/recipe_autoassess_landsurface_soilmoisture.rst

@@ -17,6 +17,7 @@ Performance metrics:
 
 Metrics are calculated using model and observation multi-year climatologies (seasonal means) 
 for meteorological seasons:
+
 * December-January-February (djf)
 * March-April-May (mam)
 * June-July-August (jja)
@@ -38,7 +39,6 @@ Recipes are stored in esmvaltool/recipes/
 
 Diagnostics are stored in esmvaltool/diag_scripts/autoassess/
 
-    * autoassess_area_base.py: wrapper for autoassess scripts
     * land_surface_soilmoisture/soilmoisture.py: script to calculate soil moisture
       metrics
     * plot_autoassess_metrics.py: plot normalised assessment metrics
@@ -47,21 +47,17 @@ Diagnostics are stored in esmvaltool/diag_scripts/autoassess/
 User settings in recipe
 -----------------------
 
-#. Script autoassess_area_base.py
+#. Script soilmoisture.py
 
    *Required settings for script*
 
    * area: must equal land_surface_soilmoisture to select this diagnostic
    * control_model: name of model to be used as control
    * exp_model: name of model to be used as experiment
-   * start: date (YYYY/MM/DD) at which period begins (see note on time gating)
-   * end: date (YYYY/MM/DD) at which period ends (see note on time gating)
-   * climfiles_root: path to observation climatologies
 
    *Optional settings for script*
 
-   * title: arbitrary string with name of diagnostic
-   * obs_models: unused for this recipe
+   none
 
    *Required settings for variables*
 
@@ -97,7 +93,8 @@ User settings in recipe
 Variables
 ---------
 
-* mrsos (land, monthly mean, longitude latitude time)
+* mrsos (from models: land, monthly mean, longitude latitude time)
+* sm (from observations: land, monthly mean, longitude latitude time)
 
 
 Observations and reformat scripts
@@ -122,56 +119,3 @@ Example plots
    :alt: Soilmoisture_Metrics.png
 
    Normalised metrics plot comparing a control and experiment simulation
-
-
-Additional notes on usage
--------------------------
-The ``landsurface_soilmoisture`` area metric is part of the ``esmvaltool/diag_scripts/autoassess`` diagnostics,
-and, as any other ``autoassess`` metric, it uses the ``autoassess_area_base.py`` as general purpose
-wrapper. This wrapper accepts a number of input arguments that are read through from the recipe.
-
-This recipe is part of the larger group of Autoassess metrics ported to ESMValTool
-from the native Autoassess package from the UK's Met Office. The ``diagnostics`` settings
-are almost the same as for the other Autoassess metrics.
-
-**Currently this recipe is marked as broken, because it only runs on Jasmin due to a dependency on some 
-external climatology files.**
-
-.. note::
-
-   **Time gating for autoassess metrics.**
-
-   To preserve the native Autoassess functionalities,
-   data loading and selection on time is done somewhat
-   differently for ESMValTool's autoassess metrics: the
-   time selection is done in the preprocessor as per usual but
-   a further time selection is performed as part of the diagnostic.
-   For this purpose the user will specify a ``start:`` and ``end:``
-   pair of arguments of ``scripts: autoassess_script`` (see below
-   for example). These are formatted as ``YYYY/MM/DD``; this is
-   necessary since the Autoassess metrics are computed from 1-Dec
-   through 1-Dec rather than 1-Jan through 1-Jan. This is a temporary
-   implementation to fully replicate the native Autoassess functionality
-   and a minor user inconvenience since they need to set an extra set of
-   ``start`` and ``end`` arguments in the diagnostic; this will be phased
-   when all the native Autoassess metrics have been ported to ESMValTool
-   review has completed.
-
-
-An example of standard inputs as read by ``autoassess_area_base.py`` and passed
-over to the diagnostic/metric is listed below.
-
-
-.. code-block:: yaml
-
-    scripts:
-      autoassess_landsurf_soilmoisture: &autoassess_landsurf_soilmoisture_settings
-        script: autoassess/autoassess_area_base.py
-        title: "Autoassess Land-Surface Soilmoisture Diagnostic"
-        area: land_surface_soilmoisture
-        control_model: IPSL-CM5A-LR
-        exp_model: inmcm4
-        obs_models: []
-        start: 1997/12/01
-        end: 2002/12/01
-        climfiles_root: '/gws/nopw/j04/esmeval/autoassess_specific_files/files'  # on JASMIN

doc/sphinx/source/recipes/recipe_bock20jgr.rst

@@ -82,7 +82,7 @@ User settings in recipe
    * time_avg: type of time average (currently only "yearly" and "monthly" are
      available).
    * ts_anomaly: calculates anomalies with respect to the defined reference
-     period; for each gird point by removing the mean for the given
+     period; for each grid point by removing the mean for the given
      calendar month (requiring at least 50% of the data to be
      non-missing)
    * ref_start: start year of reference period for anomalies
@@ -151,7 +151,7 @@ User settings in recipe
 
    * Required settings (variables)*
 
-   * reference_dataset: name of reference datatset
+   * reference_dataset: name of reference dataset
 
    *Optional settings (variables)*
 
@@ -229,8 +229,8 @@ User settings in recipe
      large multi-dimensional datasets, this might significantly reduce the
      computation time if only the multi-model mean dataset is relevant.
    * output_attributes: *dict*. Write additional attributes to netcdf files.
-   * seaborn_settings: *dict*. Options for :func:`seaborn.set` (affects all
-     plots).
+   * seaborn_settings: *dict*. Options for :func:`seaborn.set_theme` (affects
+     all plots).
 
 
 Variables
@@ -375,20 +375,20 @@ Example plots
    :align:   center
    :width:   9cm
 
-   Relative space-time root-mean-square deviation (RMSD) calculated from the 
-   climatological seasonal cycle of the CMIP3, CMIP5, and CMIP6 simulations 
-   (1980-1999) compared to observational data sets (Table 5). A relative 
-   performance is displayed, with blue shading being better and red shading 
-   worse than the median RMSD of all model results of all ensembles. A diagonal 
-   split of a grid square shows the relative error with respect to the reference 
-   data set (lower right triangle) and the alternative data set (upper left 
-   triangle) which are marked in Table 5. White boxes are used when data are not 
+   Relative space-time root-mean-square deviation (RMSD) calculated from the
+   climatological seasonal cycle of the CMIP3, CMIP5, and CMIP6 simulations
+   (1980-1999) compared to observational data sets (Table 5). A relative
+   performance is displayed, with blue shading being better and red shading
+   worse than the median RMSD of all model results of all ensembles. A diagonal
+   split of a grid square shows the relative error with respect to the reference
+   data set (lower right triangle) and the alternative data set (upper left
+   triangle) which are marked in Table 5. White boxes are used when data are not
    available for a given model and variable (Fig. 6).
 
 .. _fig_bock20jgr_5:
 .. figure::  /recipes/figures/bock20jgr/patterncor.png
    :align:   center
    :width:   9cm
 
-   Centered pattern correlations between models and observations for the annual 
-   mean climatology over the period 1980–1999 (Fig. 7). 
+   Centered pattern correlations between models and observations for the annual
+   mean climatology over the period 1980–1999 (Fig. 7).