From b17f95f917add50f22763397c2409f43c60da92c Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 15:13:58 +0100 Subject: [PATCH 01/12] #2461: Update release version and date --- CITATION.cff | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CITATION.cff b/CITATION.cff index 562e044ecb..c226e7a27c 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -210,11 +210,11 @@ authors: given-names: Diego cff-version: 1.2.0 -date-released: 2024-05-08 +date-released: 2024-06-20 doi: "10.5281/zenodo.3387139" license: "Apache-2.0" message: "If you use this software, please cite it using these metadata." repository-code: "https://github.com/ESMValGroup/ESMValCore/" title: ESMValCore -version: "v2.11.0rc1" +version: "v2.11.0rc2" ... From 42e22a7b7a4f09da5d7c4c10acaef2db2bab7b10 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 16:19:12 +0100 Subject: [PATCH 02/12] #2461: Update release notes --- doc/changelog.rst | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index e3e4b518d9..0fa179ba2d 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -19,6 +19,7 @@ Backwards incompatible changes TODO: add examples of how to deal with these changes - Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1` +- Use `iris.FUTURE.save_split_attrs = True` to remove iris warning (:pull:`2398`) by :user:`schlunma` Deprecations ~~~~~~~~~~~~ @@ -33,6 +34,9 @@ Bug fixes - Do not overwrite facets from recipe with CMOR table facets for derived variables (:pull:`2255`) by :user:`bouweandela` - Fix error message in variable definition check (:pull:`2313`) by :user:`enekomartinmartinez` - Unify dtype handling of preprocessors (:pull:`2393`) by :user:`schlunma` +- Fix bug in `_rechunk_aux_factory_dependencies` (:pull:`2428`) by :user:`ehogan` +- Avoid loading entire files into memory when downloading from ESGF (:pull:`2434`) by :user:`bouweandela` +- Preserve cube attribute global vs local when concatenating (:pull:`2449`) by :user:`bouweandela` CMOR standard ~~~~~~~~~~~~~ @@ -55,6 +59,7 @@ Computational performance improvements - Cache regridding weights if possible (:pull:`2344`) by :user:`schlunma` - Implement lazy area weights (:pull:`2354`) by :user:`schlunma` - Avoid large chunks in :func:`esmvalcore.preprocessor.climate_statistics` preprocessor function with `period='full'` (:pull:`2404`) by :user:`bouweandela` +- Load data only once for ESMPy regridders (:pull:`2418`) by :user:`bouweandela` Documentation ~~~~~~~~~~~~~ @@ -88,6 +93,8 @@ Installation - Pin pandas yet again avoid new ``2.2.1`` as well (:pull:`2353`) by :user:`valeriupredoi` - Update Iris pin to avoid using versions with memory issues (:pull:`2408`) by :user:`chrisbillowsMO` - Pin esmpy <8.6.0 (:pull:`2402`) by :user:`valeriupredoi` +- Pin numpy<2.0.0 to avoid pulling 2.0.0rcX (:pull:`2415`) by :user:`valeriupredoi` +- Add support for Python=3.12 (:pull:`2228`) by :user:`valeriupredoi` Preprocessor ~~~~~~~~~~~~ @@ -132,12 +139,13 @@ Improvements - Handle warnings about invalid units for iris>=3.8 (:pull:`2378`) by :user:`schlunma` - Added note on how to access ``index.html`` on remote server (:pull:`2276`) by :user:`schlunma` - Remove custom fix for concatenation of aux factories now that bug in iris is solved (:pull:`2392`) by :user:`schlunma` -- Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma` - Ignored iris warnings about global attributes (:pull:`2400`) by :user:`schlunma` +- Increase the version number for 2.11.0 release (:pull:`2410`) by :user:`chrisbillowsMO` +- Add draft release notes for v2.11.0rc1 (:pull:`2414`) by :user:`chrisbillowsMO` +- Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir (:pull:`2432`) by :user:`alistairsellar` .. _changelog-v2-10-0: - v2.10.0 ------- Highlights From a745184ef8e5c6be955bee1c1a9d9267301ddc9c Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 16:21:22 +0100 Subject: [PATCH 03/12] #2461: Update release notes --- doc/changelog.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index 0fa179ba2d..d9628e5cbc 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -140,8 +140,6 @@ Improvements - Added note on how to access ``index.html`` on remote server (:pull:`2276`) by :user:`schlunma` - Remove custom fix for concatenation of aux factories now that bug in iris is solved (:pull:`2392`) by :user:`schlunma` - Ignored iris warnings about global attributes (:pull:`2400`) by :user:`schlunma` -- Increase the version number for 2.11.0 release (:pull:`2410`) by :user:`chrisbillowsMO` -- Add draft release notes for v2.11.0rc1 (:pull:`2414`) by :user:`chrisbillowsMO` - Add native6, OBS6 and RAWOBS rootpaths to metoffice config-user.yml template, and remove temporary dir (:pull:`2432`) by :user:`alistairsellar` .. _changelog-v2-10-0: From 1f7d368a7c733bbcad2b607c6485500f3173c250 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 16:26:01 +0100 Subject: [PATCH 04/12] #2461: Remove outdated PR from release notes --- doc/changelog.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index d9628e5cbc..072d9f6a85 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -89,7 +89,6 @@ Fixes for datasets Installation ~~~~~~~~~~~~ -- Updated iris pin to ``iris>=3.6.1`` (:pull:`2286`) by :user:`schlunma` - Pin pandas yet again avoid new ``2.2.1`` as well (:pull:`2353`) by :user:`valeriupredoi` - Update Iris pin to avoid using versions with memory issues (:pull:`2408`) by :user:`chrisbillowsMO` - Pin esmpy <8.6.0 (:pull:`2402`) by :user:`valeriupredoi` From d0baeb776228d8ab3feb1c24a891393249bc1c2b Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 16:32:54 +0100 Subject: [PATCH 05/12] #2461: Use Sphinx code formatting --- doc/changelog.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index 072d9f6a85..c41b8502ae 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -19,7 +19,7 @@ Backwards incompatible changes TODO: add examples of how to deal with these changes - Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1` -- Use `iris.FUTURE.save_split_attrs = True` to remove iris warning (:pull:`2398`) by :user:`schlunma` +- Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma` Deprecations ~~~~~~~~~~~~ @@ -34,7 +34,7 @@ Bug fixes - Do not overwrite facets from recipe with CMOR table facets for derived variables (:pull:`2255`) by :user:`bouweandela` - Fix error message in variable definition check (:pull:`2313`) by :user:`enekomartinmartinez` - Unify dtype handling of preprocessors (:pull:`2393`) by :user:`schlunma` -- Fix bug in `_rechunk_aux_factory_dependencies` (:pull:`2428`) by :user:`ehogan` +- Fix bug in ``_rechunk_aux_factory_dependencies`` (:pull:`2428`) by :user:`ehogan` - Avoid loading entire files into memory when downloading from ESGF (:pull:`2434`) by :user:`bouweandela` - Preserve cube attribute global vs local when concatenating (:pull:`2449`) by :user:`bouweandela` From b3d715c4ab5b164ddcf6ed830cec47afc3ee1479 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Thu, 20 Jun 2024 16:44:21 +0100 Subject: [PATCH 06/12] #2461: Add highlights --- doc/changelog.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index c41b8502ae..a6db29f7bf 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -9,7 +9,18 @@ v2.11.0 ------- Highlights -TODO: add highlights +- Performance improvements have been made to many preprocessors: + + - Preprocessors :func:`esmvalcore.preprocessor.mask_landsea`, + :func:`esmvalcore.preprocessor.mask_landseaice`, + :func:`esmvalcore.preprocessor.mask_glaciated`, + :func:`esmvalcore.preprocessor.extract_levels` are now lazy + +- Several new preprocessors have been added: + + - :func:`esmvalcore.preprocessor.local_solar_time` + - :func:`esmvalcore.preprocessor.distance_metrics` + - :func:`esmvalcore.preprocessor.histogram` This release includes From a4eb6eb99fa7dfd6b1b60e6a91dc1c885cd0c86e Mon Sep 17 00:00:00 2001 From: Chris Billows <152496175+chrisbillowsMO@users.noreply.github.com> Date: Mon, 24 Jun 2024 16:29:28 +0100 Subject: [PATCH 07/12] #2461: Add Met Office release to highlights. --- doc/changelog.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index a6db29f7bf..26a773a70c 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -22,6 +22,12 @@ Highlights - :func:`esmvalcore.preprocessor.distance_metrics` - :func:`esmvalcore.preprocessor.histogram` +- NEW TREND: First time release manager shout-outs! + +This is the first ESMValTool release managed by the Met Office! We want to +shout this out - and for all future first time release managers to shout-out - +to celebrate the growing, thriving ESMValTool community. + This release includes Backwards incompatible changes From f3187887deae4977410e389eff748aa724777dca Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Tue, 25 Jun 2024 16:16:44 +0100 Subject: [PATCH 08/12] #2461: Add bullet point for first time release manager shout-outs --- doc/changelog.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index 26a773a70c..ee8e61751e 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -24,9 +24,9 @@ Highlights - NEW TREND: First time release manager shout-outs! -This is the first ESMValTool release managed by the Met Office! We want to -shout this out - and for all future first time release managers to shout-out - -to celebrate the growing, thriving ESMValTool community. + - This is the first ESMValTool release managed by the Met Office! We want to + shout this out - and for all future first time release managers to + shout-out - to celebrate the growing, thriving ESMValTool community. This release includes From 4f61b8e35d3e53380cac3e84d656c4605088a202 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Tue, 25 Jun 2024 16:23:07 +0100 Subject: [PATCH 09/12] #2461: Add details for Iris backwards incompatible change --- doc/changelog.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index ee8e61751e..d2e8938a0a 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -38,6 +38,15 @@ TODO: add examples of how to deal with these changes - Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1` - Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma` + - Since `v3.8.0`_, Iris explicitly distinguishes between local and global + netCDF attributes. ESMValCore adopted this behavior with v2.11.0. With + this change, attributes are written as local attributes by default, unless + they already existed as global attributes or belong to a special list of + global attributes (in which case attributes are written as global + attributes). See :class:`iris.cube.CubeAttrsDict` for details. + +.. _v3.8.0: https://scitools-iris.readthedocs.io/en/stable/whatsnew/3.8.html#v3-8-29-feb-2024 + Deprecations ~~~~~~~~~~~~ From 8c492056d9e50d7bfb4c11b4e5bfabd51a48a378 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Wed, 26 Jun 2024 08:49:38 +0100 Subject: [PATCH 10/12] #2461: Add details to one of the deprecation PRs --- doc/changelog.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index d2e8938a0a..aa25968702 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -51,6 +51,18 @@ Deprecations ~~~~~~~~~~~~ - Refactor regridding (:pull:`2231`) by :user:`schlunma` + + - This PR deprecated two regridding schemes, which will be removed with + ESMValCore v2.13.0: + + - ``unstructured_nearest``: Please use the scheme ``nearest`` instead. + This is an exact replacement for data on unstructured grids. ESMValCore + is now able to determine the most suitable regridding scheme based on + the input data. + - ``linear_extrapolate``: Please use a generic scheme with + ``reference: iris.analysis:Linear`` and + ``extrapolation_mode: extrapolate`` instead. + - Allow deprecated regridding scheme ``linear_extrapolate`` in recipe checks (:pull:`2324`) by :user:`schlunma` - Allow deprecated regridding scheme ``unstructured_nearest`` in recipe checks (:pull:`2336`) by :user:`schlunma` From 5eaa87c1e3318f31fee97a9770c758cf3d6d7cfd Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Wed, 26 Jun 2024 15:09:11 +0100 Subject: [PATCH 11/12] #2461: Remove TODO --- doc/changelog.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index aa25968702..e77b929135 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -33,8 +33,6 @@ This release includes Backwards incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -TODO: add examples of how to deal with these changes - - Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1` - Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma` From 86950411c6dae3fd4bdc0b1e46282d478c963284 Mon Sep 17 00:00:00 2001 From: Emma Hogan Date: Wed, 26 Jun 2024 15:13:41 +0100 Subject: [PATCH 12/12] #2461: Add details for the other backwards incompatible change --- doc/changelog.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index e77b929135..ad104a7b7d 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -34,6 +34,14 @@ Backwards incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Allow contiguous representation of extracted regions (:pull:`2230`) by :user:`rebeccaherman1` + + - The preprocessor function :func:`esmvalcore.preprocessor.extract_region` + no longer automatically maps the extracted :class:`iris.cube.Cube` to the + 0-360 degrees longitude domain. If you need this behaviour, use + ``cube.intersection(longitude=(0., 360.))`` in your Python code after + extracting the region. There is no possibility to restore the previous + behaviour from a recipe. + - Use ``iris.FUTURE.save_split_attrs = True`` to remove iris warning (:pull:`2398`) by :user:`schlunma` - Since `v3.8.0`_, Iris explicitly distinguishes between local and global