Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow wildcard searches when specifying fx variables in preprocessor #1082

Closed
wants to merge 45 commits into from
Closed
Changes from 3 commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
d644526
Search r0i0p0 if fx data not found under original ensemble
thomascrocker Apr 26, 2021
da8cf7e
typo. Moved second search under if statement
thomascrocker Apr 26, 2021
574f9f8
fixed line length complaint
thomascrocker Apr 26, 2021
4378116
fix by using globbing
thomascrocker May 5, 2021
54a5105
Merge branch 'master' into fix_fx_ensembles
valeriupredoi May 10, 2021
0c539c3
fix line too long
valeriupredoi May 10, 2021
5f5bb10
modded tests
valeriupredoi May 10, 2021
a5a933b
Added docs to additional places where FX variables are used
thomascrocker May 11, 2021
d471de2
Merge branch 'master' into fix_fx_ensembles
thomascrocker May 14, 2021
c17f85b
downgrade logging message to debug
thomascrocker May 18, 2021
7df08ee
addressing review comments and maintaining backwards compatibility
thomascrocker May 18, 2021
87b5f48
reverting tests to original
thomascrocker May 18, 2021
b53a991
refactored to resolve wildcards in separate function
thomascrocker May 19, 2021
3611451
CMIP6 wildcard FX test
thomascrocker May 19, 2021
a87d290
added CORDEX wildcard fx test
thomascrocker May 19, 2021
3f1b881
minor tweak to docs
thomascrocker May 19, 2021
8d3bf2d
flake8 fixes
thomascrocker May 19, 2021
443208f
Update _data_finder.py
thomascrocker May 19, 2021
62d091a
codacy fix
thomascrocker May 19, 2021
e2c76b1
minor changes to address review comments
thomascrocker May 20, 2021
ad9b702
additional tests
thomascrocker May 20, 2021
6909061
docs update
thomascrocker May 20, 2021
c88898b
Merge branch 'fix_fx_ensembles' of https://github.com/ESMValGroup/ESM…
thomascrocker May 20, 2021
a7bccf8
pylint fix
thomascrocker May 20, 2021
439ce43
Merge remote-tracking branch 'origin/master' into fix_fx_ensembles
schlunma May 20, 2021
3961cfc
Added further check on variable's ensemble during fx file retrieval i…
schlunma May 20, 2021
893aaaf
refactor to deal with more file path cases
thomascrocker May 21, 2021
c9cdbe4
further dir finder test
thomascrocker May 21, 2021
f0637f2
tweaks for codacy
thomascrocker May 21, 2021
fc61e06
Fixed globbing for fx files when latestversion tag is present
schlunma May 21, 2021
09543a0
Fixed output path for fx files when wildcards are used
schlunma May 21, 2021
92a6207
Fixed output path for fx files when wildcards are used (again)
schlunma May 21, 2021
3676f66
Fixed output path for fx files when wildcards are used (this time for…
schlunma May 21, 2021
3b1c0a2
Removed print() statement
schlunma May 21, 2021
845cc8d
Merge remote-tracking branch 'origin/master' into fix_fx_ensembles
schlunma May 21, 2021
b562249
Fixed tests
schlunma May 21, 2021
b50b094
Merge remote-tracking branch 'origin/main' into fix_fx_ensembles
schlunma Sep 9, 2021
c9b0f21
Merge branch 'main' into fix_fx_ensembles
valeriupredoi Nov 17, 2021
33cd03a
add jump over None dirs
valeriupredoi Nov 17, 2021
f1a3dcb
Merge remote-tracking branch 'origin/main' into fix_fx_ensembles
schlunma Feb 8, 2022
354f36b
Ran isor
schlunma Feb 8, 2022
5c455ed
Undo changes that are not relevant for this PR
schlunma Feb 8, 2022
dc171c6
add fixes
ledm Feb 10, 2022
3d544a1
added several fixes
ledm Feb 14, 2022
97f62eb
Merge remote-tracking branch 'origin/fix_fx_ensembles' into fix_fx_en…
ledm Feb 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions esmvalcore/_data_finder.py
Original file line number Diff line number Diff line change
Expand Up @@ -241,11 +241,21 @@ def _find_input_files(variable, rootpath, drs):

def get_input_filelist(variable, rootpath, drs):
"""Return the full path to input files."""
# change ensemble to fixed r0i0p0 for fx variables
# this is needed and is not a duplicate effort
if variable['project'] == 'CMIP5' and variable['frequency'] == 'fx':
variable['ensemble'] = 'r0i0p0'
schlunma marked this conversation as resolved.
Show resolved Hide resolved
(files, dirnames, filenames) = _find_input_files(variable, rootpath, drs)
if (
variable['frequency'] == 'fx'
and variable['ensemble'] != 'r0i0p0'
and dirnames == []
thomascrocker marked this conversation as resolved.
Show resolved Hide resolved
):
# fx files not found. Try to find fx files again with ensemble r0i0p0
logger.debug(
'fx data not found under %s, searching under r0i0p0',
variable['ensemble']
)
variable['ensemble'] = 'r0i0p0'
(files, dirnames, filenames) = _find_input_files(
variable, rootpath, drs
)
# do time gating only for non-fx variables
if variable['frequency'] != 'fx':
files = select_files(files, variable['start_year'],
Expand Down