Draft changelog for v2.8.0

[remi-kazeroni]

To generate the changelog for v2.8.0, we need to:

• Draft changelog using PRs merged into main since v2.7.0, see list.
• Remove PRs that were listed for v2.7.1
• Add highlights (discussion in Highlights for ESMValCore v2.8.0 release #1954)
• Add documentation on backward incompatible changes

[remi-kazeroni]

@ESMValGroup/technical-lead-development-team, I'd like to propose to move to release rc1 without including the release highlights and without the instructions for backward incompatible changes and deprecated features. These will be included for a second release candidate, if any or push to main via a PR and cherry-picked into the release branch. The reasons for doing so would be:

• Getting a first release candidate asap and not delaying the release further. This would allow to start testing which is really needed...
• Giving people a bit more time to choose their favourite PRs for the highlights
• Agreeing on the documentation to provide for each PR marked "backward incompatible" or "deprecated feature" and also on which PR goes into which section. I'm realizing now that this needs extra thinking. More on this in my next post to this issue.

This issue remains until the changelog is complete.

Feel free to reply with a 👍 is you agree or a 👎 if you disagree

[remi-kazeroni]

@bouweandela and @schlunma, it would be great if you could help me drafting the recommendations and instructions for each PR listed as Backwards incompatible changes and Deprecations since you're the authors of these 7 PRs.

Could you please check that the description of these PRs look good, contains hyperlinks to the documentation and provide in this issue 1-2-3 sentences to be added to the Changelog? That would be super helpful as you're more specialist than me about these things 👍 Also, feel free to tell me if a PR should be moved to the other section.

Backwards incompatible changes

• Support wildcards in the recipe and improve support for ancillary variables and dataset versioning #1609 @bouweandela (a couple of sentences please)
• Remove deprecated features scheduled for removal in v2.8.0 or earlier #1826: Removed esmvalcore.iris_helpers.var_name_constraint (has been deprecated in v2.6.0; please use iris.NameConstraint with the keyword argument var_name instead) and the option always_use_ne_mask for esmvalcore.preprocessor.mask_landsea (has been deprecated in v2.5.0; the same behavior can now be achieved by specifying supplementary_variables for datasets/variables with the option skip: true).
• Add esmvalcore.local, a module to search data on the local filesystem #1835 @bouweandela (a couple of sentences and links to docs please)
• Update filename template for obs4MIPs to better match filenames #1866 @bouweandela (PR description empty, a couple of sentences please)

Deprecations

• Add esmvalcore.config module #1769 @bouweandela (a couple of sentences please)
• Combined offline and always_search_esgf into a single option search_esgf #1935: The configuration option/command line argument offline has been deprecated in favor for search_esgf. The old offline: true is now search_esgf: never (the default); the old offline: false is now search_esgf: when_missing. More details on how to adapt your workflow regarding these new options are given in the corresponding pull request descrpition and the documentation.
• Fixed race condition that may result in errors in cleanup and deprecate cleanup #1949: The preprocessor cleanup has been deprecated. Please do not use this anymore in the recipe (it is not necessary).

Once we have the material ready for the changelog, I'll open a PR to do the update. Thanks very much in advance! 🍻

[valeriupredoi]

that's a solid approach @remi-kazeroni - but pls make sure you mention the non-inclusion of those PRs only in the changelog, testing will be done with those - OK, sorry if I sound like Dr Obvious here 😁

[schlunma]

@remi-kazeroni I edited your comment and updated all the PR descriptions 👍

[remi-kazeroni]

@remi-kazeroni I edited your comment and updated all the PR descriptions 👍

Brilliant @schlunma! That is exactly what I needed! Thanks a lot 👍

[bouweandela]

@remi-kazeroni While #1769, #1835, #1609 deprecate older version(s) of the features they improve or contain small backward incompatible changes, they mostly add a lot of new functionality. Would it be possible to list those in the changelog twice? Once to describe the new feature and once (with a different text), to describe what was deprecated/changed in a backward incompatible way? I could open a draft pull request to demonstrate what I mean if that would be helpful.

[bouweandela]

About the titles used in the changelog:

• Notebook API (experimental) -> Can we change this to Python API? The new features listed here are not experimental and their use is not restricted to notebooks
• Bug fixes -> Could we separate this into two types of bugs? Now it looks like the previous release contained many bugs, while in fact most of these bugs were not part of the previous release but were introduced in the development of the current release. Maybe we could add an extra category 'development bugs' or something similar?
• User experience: arguably many of the changes aim to improve the user experience, why is there only 1 pull request listed here?

[remi-kazeroni]

Thanks for your offer @bouweandela! It would be great if you could open a draft PR to demonstrate what you have in mind. Drafting the changelog is not the easiest part of the release. At the moment, we have a test that checks that no PR is listed twice. I agree that is not great when we have nice and large PRs that could easily fit into several categories. In fact most PRs in the Core had 2 or more labels and it's not always easy to decide where these should go in the changelog. Feel free to open a PR based on this branch. Would be great to see how to improve the situation. for example, having to choose between backward incompatibility, deprecation and enhancements is not always possible, like for #1609.

Agreed with the suggested titles.

Bug fixes -> Could we separate this into two types of bugs?

What about keeping "Bugs" for things that went into a previous release and "Others", "Development bugs", "Internal", or "Miscellaneous",... for things that have never made it into a release?

User experience: arguably many of the changes aim to improve the user experience, why is there only 1 pull request listed here?

I guess this is because not many contributors have used the corresponding label in comparison to enhancement. Also because PRs can only be listed one.

In the long run, we should discuss the usage of labels and drafting changelog in the Tech Lead Team. It would be good to have common practices, for examples: which labels to apply when something is deprecated and where to list it in the changelog (when code is deprecated and when it is removed). It may be time to revise our approach on listing PRs only once in the changelog.

[remi-kazeroni]

Update on the changelog:

Notebook API (experimental) -> Can we change this to Python API

Done in #1959

User experience

I removed this section which has never been used in the changelog for any previous version. I moved the one PR into improvements.

Bug fixes -> Could we separate this into two types of bugs?

This is not something I could do alone. I would need each contributor to tell me which of their PRs fixed a released bug and which ones fixed a development bug... This would have worked better if we have had 2 distinct labels for these and had a common strategy for applying labels.

While #1769, #1835, #1609 deprecate older version(s) of the features they improve or contain small backward incompatible changes, they mostly add a lot of new functionality.

By mistake I mentioned the same PR twice in one commit, the tests failed, see: https://app.circleci.com/pipelines/github/ESMValGroup/ESMValCore/8708/workflows/9dab4883-3085-4612-a8a2-dcac1e4d02be/jobs/38795
The only thing I could offer (and actually did) is to advertise these PRs in the highlights section. This actually gives me better visibility than the Improvements section.

We have a discussion on the multiple use of the labels in #1365 Perhaps it would be a good time to revive it, agree on common practices for the next release and discuss at the next TLT call?

[valeriupredoi]

@bouweandela I like the idea of two or three groups of bugs but let's not do it now - if you could open an issue about that, then maybe we can discuss the categories there - for v2.9. Abut UX improvements - again - we should think what sort of labels we add when we add labels, and not when the RM is against the time wall to get the release done 😁

We have a discussion on the multiple use of the labels in #1365 Perhaps it would be a good time to revive it, agree on common practices for the next release and discuss at the next TLT call?

This ⬆️