Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make _find_input_files public #1618

Closed
wants to merge 2 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions esmvalcore/_data_finder.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,6 @@ def dates_to_timerange(start_date, end_date):
-------
str
``timerange`` in the form ``'start_date/end_date'``.

"""
start_date = str(start_date)
end_date = str(end_date)
Expand Down Expand Up @@ -283,7 +282,6 @@ def _truncate_dates(date, file_date):
same number of digits. If this is not the case, pad the dates with leading
zeros (e.g., use ``date='0100'`` and ``file_date='199901'`` for a correct
comparison).

"""
date = re.sub("[^0-9]", '', date)
file_date = re.sub("[^0-9]", '', file_date)
Expand Down Expand Up @@ -443,7 +441,7 @@ def get_rootpath(rootpath, project):


def _find_input_dirs(variable, rootpath, drs):
"""Return a the full paths to input directories."""
"""Return the full paths to input directories."""
project = variable['project']

root = get_rootpath(rootpath, project)
Expand Down Expand Up @@ -477,8 +475,36 @@ def _get_filenames_glob(variable, drs):
def _find_input_files(variable, rootpath, drs):
"""Find available input files.

Return the files, the directory in which they are located in, and
the file name.
Parameters
----------
variable : Dict[str: str]
[Need help here] Metadata (keys are metadata labels, values provide
value for the particular variable) that defines the variable (in the
ESMValTool sense i.e. piece of data which is being worked on) to
lookup. This will generally look something like
``{"short_name": "tas", "original_short_name": "t", "exp": "1pctCO2"...}``

rootpath : Dict[str: list[str]]
Rootpaths in which to search for files. Keys are projects (e.g. CMIP6)
and values are lists of paths to try as the rootpath.

drs : Dict[str: str]
DRS to use when searching for files. Keys are projects (e.g. CMIP6) and
values are the drs to use (e.g. ESGF).

Returns
-------
list[str], list[str], list[str]
Files (full path), directories in which they were searched for and
filename globs used to look for files. Note that the returned lists are
likely all of different lengths (i.e. don't correspond to each other in
any particular way).

Raises
------
KeyError
``variable`` does not contain all of the following keys:
``"short_name"``, ``"original_short_name"``
"""
short_name = variable['short_name']
variable['short_name'] = variable['original_short_name']
Expand Down