unable to download ERA5 data using dfmt.download_ERA5

[thopri]

Hello,

The API interface for the ERA5 datasets seems to have changed, it uses a cds-beta in its url and the existing auth token no longer works (despite still being in the online account). I had to do a weird dance with ECMWF and CMEMS ending up with a new auth token which seems to pass authentication but I now get an "dataset dummy not found" error.

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds-beta.climate.copernicus.eu/api/retrieve/v1/processes/dummy/execute

I think the code at File "/home/thopri/miniforge3/envs/mod_build/lib/python3.11/site-packages/dfm_tools/download.py", line 129, in cds_credentials is calling a dummy dataset or call to check the auth and this is no longer valid. I will try commenting it out to see if the actual calls are still correct and update here.

[veenstrajelmer]

Yes, indeed the dummy dataset is called here:

dfm_tools/dfm_tools/download.py

Lines 125 to 129 in f6069de

	try:
	# checks whether CDS apikey is in environment variable or ~/.cdsapirc file
	c = cdsapi.Client()
	# checks whether authentication is succesful (correct combination of url and apikey)
	c.retrieve(name='dummy', request={})

I have implemented an exception there, but apparently the cdsapi has been changed so this is not catched anymore. Thanks for reporting this. As a temporary workaround, please do pip install cdsapi==0.7.0, then all will work just fine. In the meantime I will work on a fix that supports their new code.

Todo:

• update cdsapi minimal version to 0.7.2
• check if workaround code can be removed >> still required
• fix dummy error
• update whatsnew

[thopri]

commenting out the c.retrieve(dummy) seems to have "fixed" it. I did have to accept license terms to get it to work which seemed to involve making an random request on the ERA5 web data form thingy here:

https://cds-beta.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=download

This allows you to accept the licence and make API requests.

[veenstrajelmer]

Yes, I know, it is very inconvenient. It has been reported here, so feel free to comment this there: ecmwf/cdsapi#108. I also don't really know why they decided to move from a CDS account to a ECMWF account, but I have tried to accomodate dfm_tools as well as possible for it. FYI, the call to the dummy dataset is to check (and catch) whether the credenitals in your ~/.cdsapirc file are correct, since they have no API to check this themselves it seems (requested in ecmwf/cdsapi#111).

[thopri]

Yes I am having to add all this to my documentation/notes as its kinda confusing! I hope it becomes clearer out of beta.

[veenstrajelmer]

There is however a message when you supply your apikey to dfm_tools, which points to a useful article:

dfm_tools/dfm_tools/download.py

Lines 112 to 115 in 4ae7642

	print("Downloading CDS/ERA5 data requires a ECMWF API-key, copy "
	"your API-key from https://cds-beta.climate.copernicus.eu/profile "
	"(first register, login and accept the terms). "
	"More info in https://forum.ecmwf.int/t/3743.")

So to https://forum.ecmwf.int/t/3743. Maybe you already found this, but if not it might be helpful to not have to document everything yourself.

[veenstrajelmer]

As you might have seen the issue was fixed in the main branch. There is one fix pending (#962) before I will make a release, I will do this next week.