Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply select preprocessor to area, volume and zonal statistics fx variables and add use of fx_files as list of dictionaries #439

Closed
wants to merge 81 commits into from
Closed
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
7d19611
routine to preprocess fx variables
valeriupredoi Jan 20, 2020
01afa17
fixed bug
valeriupredoi Jan 20, 2020
a96cae1
added opt arg
valeriupredoi Jan 20, 2020
ab887ef
added non functional test
valeriupredoi Jan 20, 2020
615ab3b
pick up already preprocced files as input
valeriupredoi Jan 20, 2020
f9c0047
made all fx vars run as ancestors
valeriupredoi Jan 20, 2020
2b575e1
fixed handling of tasks
valeriupredoi Jan 21, 2020
434ee12
fixed test
valeriupredoi Jan 21, 2020
d1cdbc6
fixed main var bug finally
valeriupredoi Jan 21, 2020
1ba5c74
fixed test proper
valeriupredoi Jan 21, 2020
2ae17bc
removed old flag
valeriupredoi Jan 21, 2020
7bb133d
added extra preproc sugg by Lee
valeriupredoi Jan 21, 2020
e99aff0
added extra preproc step sugg by Lee
valeriupredoi Jan 21, 2020
aa67226
added check on ancestry
valeriupredoi Jan 21, 2020
590e494
Merge branch 'preprocess_fx_variables' of https://github.com/ESMValGr…
valeriupredoi Jan 21, 2020
d897b1b
made it more robust
valeriupredoi Jan 21, 2020
c1e5141
correct profile applied
valeriupredoi Jan 21, 2020
5fc532f
replaced to dummy var
valeriupredoi Jan 21, 2020
9ed2f57
made it more robust
valeriupredoi Jan 22, 2020
f84f6e7
more power to the test
valeriupredoi Jan 22, 2020
e520166
allow for user defined fxvars dicts
valeriupredoi Jan 22, 2020
967423f
added needed tests for fxvars dicts
valeriupredoi Jan 22, 2020
e8711a2
added doc on new features for fx files
valeriupredoi Jan 22, 2020
a3b33dd
remove time preprocessors for fx mips
valeriupredoi Jan 23, 2020
aaa638f
added time preprocessors list
valeriupredoi Jan 23, 2020
c80d027
fixed test and asdded check on time preproc removal
valeriupredoi Jan 23, 2020
dbcd03e
changed declaration of fx files
valeriupredoi Jan 23, 2020
fe5a1ea
add raiser if no available fx vars
valeriupredoi Jan 23, 2020
e0ab9cb
actually use the recipe error correctly
valeriupredoi Jan 23, 2020
c6e4ca3
adding Lee's suggestion
valeriupredoi Jan 23, 2020
0750802
adding Lee's suggestion
valeriupredoi Jan 23, 2020
a4ff5fc
correct line too long
valeriupredoi Jan 23, 2020
15e1a67
added Manuel suggestion
valeriupredoi Jan 23, 2020
cba0cbb
removed redundant checks
valeriupredoi Jan 23, 2020
a733e0d
simplified approach and include activity
valeriupredoi Jan 23, 2020
44aa6ae
removed redundant test
valeriupredoi Jan 23, 2020
2a603cb
remove duplication in task ancestry and run task only once for unique…
valeriupredoi Jan 24, 2020
4c0f8be
use the preproc time preprocs list and robustify the testing as a con…
valeriupredoi Jan 24, 2020
a22a8fa
renamed tests and added extra case for activity switch
valeriupredoi Jan 24, 2020
16f8fd6
shuffled section at Monsegnor Mattia indication
valeriupredoi Jan 24, 2020
bd1a0ac
added note on running preproc steps on various time and non time fx vars
valeriupredoi Jan 24, 2020
68fb173
added to API
valeriupredoi Jan 24, 2020
1a6b551
added to API
valeriupredoi Jan 24, 2020
1a2bb90
fixed linter
valeriupredoi Jan 24, 2020
ef70bb3
removed check on OBS mips
valeriupredoi Jan 28, 2020
5460f44
made the preproccing of fx vars in preproc step default and always on
valeriupredoi Jan 28, 2020
aec6cad
removed redundant fx)var_preprocess flag
valeriupredoi Jan 28, 2020
e070f71
changed doc to remove pointers to defunct fx_var_preprocess flag
valeriupredoi Jan 28, 2020
c535aab
actual correct setter for fx files preproc
valeriupredoi Jan 28, 2020
f152ed4
actually fixed now
valeriupredoi Jan 28, 2020
af10679
Fixed prospector issues
schlunma Jan 29, 2020
13cf2b1
Small optimizations in fx file retrieval
schlunma Jan 29, 2020
2b46632
Expanded unit tests
schlunma Jan 29, 2020
c4877dd
Fixed docstrings
schlunma Jan 29, 2020
1001754
Added two more tests
schlunma Jan 29, 2020
99c2aae
fixed test
valeriupredoi Jan 30, 2020
147b151
slight docstring mod
valeriupredoi Jan 30, 2020
b4ccec4
slight docstring mod
valeriupredoi Jan 30, 2020
87931ba
fixed the last duplicate preproc file error reported by Lee, test wil…
valeriupredoi Jan 30, 2020
f827ce5
removed print statement
valeriupredoi Jan 30, 2020
b8aa4d9
overhauled method for identical ancestry removal
valeriupredoi Jan 31, 2020
8f97597
started work on the extra test
valeriupredoi Jan 31, 2020
3ed2dc7
bumped priority of clean task
valeriupredoi Jan 31, 2020
da7bbbb
completed test
valeriupredoi Jan 31, 2020
8a854a6
fixed bug related to single product duplicity
valeriupredoi Jan 31, 2020
431e02d
added inline comments
valeriupredoi Jan 31, 2020
ae128e2
tast now works at any time
valeriupredoi Jan 31, 2020
fc8d3bb
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Feb 3, 2020
5c0c7fe
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Feb 6, 2020
39a34b5
fixed case when there are no tasks at all
valeriupredoi Feb 6, 2020
8c90798
fixed crass bug noticed by Lee
valeriupredoi Feb 17, 2020
3b95671
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Feb 17, 2020
18cb3f6
priority ordering of tasks based on importance wrt ancestry
valeriupredoi Feb 18, 2020
b80e47e
rejiggled priorities since the sorting is done not in reverse order
valeriupredoi Feb 18, 2020
b129407
fixed major fx file generation bug
valeriupredoi Feb 19, 2020
d3548d6
special case for fx favrs
valeriupredoi Feb 19, 2020
0da1ea4
fixed test in light of changes
valeriupredoi Feb 19, 2020
dbe3c17
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Feb 21, 2020
f361395
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Feb 25, 2020
5f49f22
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Mar 6, 2020
dc30150
Merge branch 'master' into preprocess_fx_variables
valeriupredoi Mar 12, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions doc/esmvalcore/preprocessor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,92 @@ variable items e.g.:

.. _masking of missing values:


Using mask files for area/volume/zonal statistics
valeriupredoi marked this conversation as resolved.
Show resolved Hide resolved
-------------------------------------------------

Preprocessors like `volume_statistics` may take the optional key `fx_files` to allow
the computation of statistics with fx data; this data is retrieved in the same manner
as regular data files and can be ingested in the statistical computation as-is or it
can be preprocessed:

Key features:

- it is possible to run a set of preprocessor steps on the `fx_files: [vars or dicts of vars]` specified in the `area_statistics`, `volume_statistics` or `zonal_statistics`by setting the `fx_var_preprocess: True` in the `variable`; take for instance this example:

.. code-block:: yaml

preprocessors:
prep:
custom_order: true
extract_volume:
z_min: 0
z_max: 100
annual_statistics:
operator: mean
volume_statistics:
operator: mean
fx_files: [volcello, ]

diagnostics:
diag:
variables:
thetao700_Omon:
short_name: thetao
preprocessor: prep
fx_var_preprocess: True
mip: Omon
exp: historical
ensemble: r1i1p1f1
grid: gn
start_year: 2010
end_year: 2014
additional_datasets:
- {dataset: GFDL-CM4}
- {dataset: HadGEM3-GC31-LL}
- {dataset: UKESM1-0-LL}


this setup will run the custom-ordered preprocessor steps in `preprocessors/prep` before (and not including) `volume_statistics` for the fx variable `volcello`;

- it is also possible to request the same type of fx variables but as dictionaries, providing more information towards the retrieval of the correct data; take this snippet for example:


.. code-block:: yaml

preprocessors:
prep:
custom_order: true
extract_volume:
z_min: 0
z_max: 100
annual_statistics:
operator: mean
volume_statistics:
operator: mean
fx_files: [{short_name: volcello, mip: Omon, exp: historical}]

diagnostics:
diag:
variables:
thetao700_Omon:
short_name: thetao
preprocessor: prep
fx_var_preprocess: True
mip: Omon
exp: historical
ensemble: r1i1p1f1
grid: gn
start_year: 2010
end_year: 2014
additional_datasets:
- {dataset: GFDL-CM4}
- {dataset: HadGEM3-GC31-LL}
- {dataset: UKESM1-0-LL}


in this case, by providing the extra CMOR keys, we allow the code to find the `volcello` with mip `Omon` and experiment `historical`. This is particularly useful when the actual fx data is not found in the master variable's (`thetao` in this case) data experiment; this is also useful to force a certain time-dependent data (mip: Omon) for preprocessors that need time operations

Missing values masks
--------------------

Expand Down
Loading