Intermittent 400 response when loading HLS data

[waltersdan]

Hi there,
I'm having intermittent issues accessing both the HLS S30 and L30 data. I am using pystac to search and stackstac to create xarray stacks for a series of tiles. Each tile is roughly 1000x1000 pixels, 300 timesteps, 7 bands. I have a local dask cluster running with 64 workers. I load the actual data into memory using something like:

arr = stack.to_numpy().astype('uint16')

And then I do some local processing on the data. This works for, say, 40 out of 45 tiles with no issue, each transferring to my local memory in about 5 minutes. However, sometimes I get an exception like:

Exception: 'RuntimeError("Error opening \'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSS30.020/HLS.S30.T09UXR.2022289T194411.v2.0/HLS.S30.T09UXR.2022289T194411.v2.0.B12.tif\': RasterioIOError(\'HTTP response code: 400\')")'

I have my download code wrapped with the retry module, with a delay of 10s, backoff 20s. Sometimes this clears after one error, other times it repeats. When it repeats, the offending .tif is sometimes the same, sometimes not.

I did try adding the GDAL http retry options (https://gdal.org/user/virtual_file_systems.html) per this suggestion (gjoseph92/stackstac#18):

dist_env = stackstac.DEFAULT_GDAL_ENV.updated(dict( GDAL_DISABLE_READDIR_ON_OPEN='TRUE', GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'), GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'), GDAL_HTTP_MAX_RETRY=5, GDAL_HTTP_RETRY_DELAY=5) )

which I pass to stackstac, but that doesn't seem to do anything, probably because GDAL retry codes don't include 400.

Any ideas would be much appreciated

[william-valencia]

@waltersdan Are you still having this issue?

[waltersdan]

I have not noticed it in some time, although I do wrap all my download code with a retry wrapper which works around it.