Preparing the release of v2.5.0 #2535
Comments
Recipe testing against v2.5.0rc2 of the CoreThe result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc2/ @ESMValGroup/esmvaltool-recipe-maintainers Everyone can also help us by running a few recipes with the latest development installation in order to confirm these results. Settings for the recipe testing And special thanks to @bouweandela for making the recipe testing much more convenient 🥳 Edit: I reran 4 recipes using |
|
Below we summarize all the issues found by the automatic run. Feel free to expand this table as you see fit. Wontfix IssuesThese issues will not be fixed in this release. They are generally due to data that has become unavailable or is not yet fully integrated into ESMValTool. Missing data
Other problems
Outstanding IssuesMissing dataERA5
CMIP5
CMIP6
AUX
ESGF Data search / download incomplete
Other problemsDiagnostic issue (Python)
Diagnostic issue (NCL)
Diagnostic issue (R)
Core issue
Resolved IssuesMissing data
Other problems
|
|
The overview of issues found by running all recipes automatically is now complete (see post above). Please add already opened issues to the tables, we may have missed some. It seems that the missing CMIP5 issue has slightly worsened since the last release (see #2408). But we were told that some deleted data will be retrieved from the archive soon. We also have a few failures with the automatic download features (too long request on ESGF nodes, download failures, existing data but not found). Could someone else try to rerun some of the impacted recipes? I tried myself but got the same failures. I may not be using optimal settings for the automatic download. Any suggestion welcomed! |
|
stellar work, guys! What's the actual problem with the two Julia recipes? For |
|
also, I'm gonna run a bunch of those CMIP5 missing recipes since we have CMIP5 on JASMIN (unlike some German HPC that did a terribly ungerman thing and deleted all that data 🤣 ) |
|
OK here's me stuffs on JASMIN:
|
|
let me know if you guys want me to run any others but bugger me flat, am missing lots of OBS's at JASMIN 🔍 |
Here is the type of error I am getting. Did I do something wrong when installing Julia? I simply followed the instructions and did |
|
you need to purge the (most probably existing) |
|
We had two cases where using |
|
yes - very good find, Remi! That's a pretty serious performance bug 🐛 |
|
I agree that this is definitely something we need to check, but we probably can't fix this in time for v2.5. I suggest adding this to v2.6. |
|
am opening an issue as we speak |
|
also, I ran this is like shooting ducks with ballistic missiles: MRI has |
Should we do a comment with the list of full recipes and a checkbox for having them checked for the scientific ok - provenance and figures looking fine? Otherwise we never know if anyone actually looked at the output for the recipes that ran fine. |
|
@bettina-gier yes that is a good idea always - maybe this can be done during the freeze, like we did during other releases 🍺 |
|
I just tried to identify the issue with The weird thing is that I'm running this on a 256 GB node, and the output says |
|
no, it says Maximum memory used (estimate): 187.4 GB - but don't matter, if it was memory it would have said |
|
Yes, you're right!! It is related to this. Here is the output of some prints: The last indices are definitely not correct. I will try to fix this. (the 48.4 GB referred to my run with only 6 year; the 187.4 GB correspond to using all 20 years) |
|
Status of the release process and recipe testing: Using the release candidate v2.5rc2 for the Core, we initially got 80 of 119 public recipes to run successfully during the automated testing. Over the past week, the number of successful recipes was increased to 96 by:
The latest fixes to the Core warrants another (an probably last) release candidate for the Core. @schlunma will soon release v2.5rc3. This will be done at the same time as the feature freeze for the Tool. We will perform another round of recipe testing and ask maintainers to check the output later this week. |
|
Recipe testing against v2.5.0rc3 of the Core The result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc3/ Given the current uncertainties in the release process and the choice of iris version used for v2.5 (see ESMValGroup/ESMValCore#1509), we may not need to ask recipe maintainers to check the results of their recipes at this stage. We will do so for the next round of recipe testing (most likely needed if iris=3.1 is chosen) or contact maintainers to check this round of recipe testing later on. Nevertheless, feel all free to take a look at your favourite recipes. Summary of the main issues 6 recipes ran successfully with v2.5.0rc2 but not this time:
Among the 18 other failing recipes, 1 "new" issue discovered:
The 17 other failures remain unchanged:
|
|
I wonder if we could speed-up the search on ESGF nodes for some recipes ( |
|
|
cheers @remi-kazeroni and @schlunma - for the Julia ones, have you nuked the presumably existent |
Recipe testing against v2.5.0rc4 of the CoreThe result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc4/debug.html Among the 121 available recipes, 111 were run successfully and only 10 failed. This is the result of fixes to the Core, recipes/diagnostics and chasing missing data. A summary of the remaining failures in the next post. This last round of recipe testing also shows nice improvement in the automatic download of data thanks to the updated esgf-pyclient package. The data search through ESGF nodes takes only minutes in cases where it used to take hours to find all the fx files (see ESMValGroup/ESMValCore#1495). Thanks for your great work @bouweandela! @ESMValGroup/esmvaltool-recipe-maintainers, it would be great if you could check the output of the recipes you are responsible for and tick the corresponding box if happy with the output (checklist here and here). If something does not look right, please let us know if this issue. List of recipes:
|
|
Summary of the main issues
|
|
excellent work guys! I'll test the AA recipes on JASMIN, so we have only 8 crappe'd 😁 |
|
the issue coz of the ISCCP data on JASMIN - I can turn that dataset off, but I know for a fact that that should work if we fix the ISCCP data, what do you recommend @remi-kazeroni ? The other AA went through no problemo |
Thanks for double checking the AA recipes @valeriupredoi! We already have an issue open for the problematic dataset: ESMValGroup/ESMValCore#1238 If it's better to keep the dataset in the recipe, I'd suggest to address that for the next release 👍 |
|
awesome, yeah - I believe I opened that issue judging by the title containing semi-expletives 😅 |
|
Status of the recipe testing using the version v2.5.0rc4 for the Core: there are up to 115 recipes that can be run successfully. Among the remaining 6 recipes, there should not be any missing data problem any more (CMIP5, CMIP6, OBS,...). For each failing recipe, there is a dedicated issue or PR to fix it. In the (hopefully 😆) last round of testing, we need to pay closer attention to the |
|
@remi-kazeroni stellar work! Do we have a mention in the documentation of that recipe that needs gargantuan amounts of memory? |
|
For information, our ESMValTool project data from Mistral will be copied to Levante (successor machine) on March 14. This comprises user data (e.g. /work/bd0854/b3*****) but also OBS/RAWOBS data and data from different projects. After the copy, DKRZ will try to keep the data synchronize between the 2 machines until March 22. After that, it will be our responsibility to copy data from Mistral to Levante. More info here: https://docs.dkrz.de/blog/2022/mistral-levante-data-move.html I think this will require some actions from us to adjust the data paths, reorganize the directory structure of the group space... But for the time being, I'm hoping that we could finish the recipe testing and the release on Mistral only and configure things on Levante once the release is over. So hopefully we can finish the release sometime next week at the latest :) |
Recipe testing against v2.5.0rc6 of the CoreThe result of the recipe testing is accessible at: https://esmvaltool.dkrz.de/shared/esmvaltool/v2.5.0-test-rc6/debug.html Among the 121 available recipes, 114 were run successfully and only 7 failed. Since the outcome of this round of testing is identical to the previous one, there is no need for another thorough recipe output checking. Nevertheless, feel free to check the output of your favourite recipe if you wish. I think this round of testing is good enough to do the release next week 🍻 No new bug was discovered during this testing. Among the 7 known issues, we have:
|
|
make that 6 now 😁 so that's a 5% loss ratio - am very happy and it's a good start for my day off today - excellent work the Release Boys 🍺 |
|
I forgot to write that I needed to remove some output files (mostly netcdf files) for the recipe runs that required lots of disk space. The reason is that the recipe outputs are copied to the disk of the bot VM. Disk space there is limited to 1TB while some recipes produce 10GB - 100GB of data (even with the /preproc direcrotry removed). In some cases, there was not much left to check as some recipes do not produce plots but large nc files. To help checking recipes, I have rerun 2 recipes successfully: recipe_wflow (all netcdf files kept) and recipe_daily_era5 (some netcdf files kept). @axel-lauer: do you want to have another look at these 2 recipes? |
Thanks @remi-kazeroni. As far as I can tell, the output of the two recipes looks OK. I updated the list above. |
|
Closing this now since v2.5.0 is out 🎉 Thanks to everyone who contributed to this! An issue is open for every failing recipe; let's move the discussion to these. Cheers!! |
@ESMValGroup/esmvaltool-developmentteam
The next release of ESMValTool and ESMValCore are approaching (see schedule). As part of the release process, a candidate release has been made for the Core (v2.5.0rc2) following the feature freeze announcement.
We have now tested all the ESMValTool recipes against the candidate release rc2 for the Core (see next post).
Please note that the feature freeze for the Tool is schedule for Monday, Feb 21. Please let us know as soon as possible if you are still working on something for this release by adding your issue or pull request to our v2.5.0 milestone: https://github.com/ESMValGroup/ESMValTool/milestone/8
The text was updated successfully, but these errors were encountered: