Is there a way to optimise looking for files required for recipes?

[ehogan]

The recipe_diurnal_temperature_index.yml recipe requires 12 files; in the main_log_debug.txt file there are 12 entries with the format Looking for files matching ['<filename>.nc'] in ['<data_path>']. However, the data_path we are using contains many files (38T!), which means that the searching step for each file takes up to 2 minutes, so for 12 files takes over 23 minutes!

The workaround we're using is to create a new directory, create links to the required files in this directory, then use this directory as the data_path.

Is there a way to optimise looking for files required for recipes?

[bouweandela]

That would require some changed to the code that looks for files. Maybe it can be taking into account together with #281.

[valeriupredoi]

we have a to-do item that will use wildcards to select all data files with a certain naming template see this too #589 - I will have to get my head around this and start working on it soon 🍺

[ehogan]

@bouweandela @valeriupredoi that sounds great, thanks! :)

[bouweandela]

@mattiarighi The work still needs to be done first..

[mattiarighi]

I closed because it looks like a duplicate of #589

[bouweandela]

It's a different problem, only related to the same code.

[bouweandela]

I will have to get my head around this and start working on it soon

@valeriupredoi Maybe you can try writing a cached version of os.scandir by using functools.lru_cache and then patch the glob module so it uses that?

[mo-tgeddes]

@ehogan and I ran this again and it is no longer a problem so I will close the issue. Thanks