State of the art FX variables handling without preprocessing #557

valeriupredoi · 2020-03-06T11:49:07Z

This PR is the abridged version of #511 and includes all the modifications from #511 and #439 that @schlunma and myself have implemented and @zklaus toiled to review and suggest a number of very useful changes; it however doesn't include the hacky way of preprocessing fx data as in #511 and #439 since we may want to hold that in the background until implementing the iris cell_measures parsing as @bouweandela suggested; it also includes functionality to solve @ChrisJones-MOHC issue reported in #547
Most of it has been reviewed by @zklaus and should be in good shape, but alas, another pass and testing is needed; review for tests is still needed as well.
Cheers 🍺

…bles (and some changes from yapf)

valeriupredoi · 2020-03-06T11:52:10Z

I will keep maintaining #439 because @ledm and @LisaBock are still using it to get IPCC stuffs done, but I reckon that upon approval of this PR we can safely close #511 and start thinking how to get us rid of ancestors and implement the fx vars preprocessing via cell_measures and iris magic. @bouweandela and myself can do that in the near future 🍺

ChrisJones-MOHC · 2020-03-06T16:03:41Z

thanks @valeriupredoi I tried some testing.

It seems to work perfectly for UKESM1. I processed historical data and it correctly picked up the piControl sftlf file. I used pre-processor:
preproc_weighting:
weighting_landsea_fraction:
area_type: land
fx_files: [{short_name: sftlf, mip: fx, exp: piControl}]

I have also combined it with annual meaning in the pre-processor:
preproc_weighting:
weighting_landsea_fraction:
area_type: land
fx_files: [{short_name: sftlf, mip: fx, exp: piControl}]
annual_statistics:
operator: mean

Both of these bit-compare with raw CMIP6 files I pulled back and processed by hand.

I'll also give it a try with some other models.

valeriupredoi · 2020-03-06T16:06:04Z

Note that now you can throw nothing or anything including the kitchen sink or anything inbetween at the preprocessors that use fx_files, see an example:

  prepocessor_namey:
    annual_statistics:
      operator: mean
    mask_landsea:
      mask_out: sea
      fx_files: [{short_name: sftlf, mip: fx, exp: piControl, ensemble: r1i1p1f2, grid: gn, activity: CMIP}]
    mask_landseaice:
      mask_out: sea
      fx_files: [{short_name: mrsofc, exp: piControl}]
    area_statistics:
      operator: mean
    volume_statistics:
      operator: mean
      fx_files: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p1f2, grid: gn, activity: CMIP}]

valeriupredoi · 2020-03-06T16:06:50Z

cheers muchly @ChrisJones-MOHC - was just writing a comment-example 🍺

ChrisJones-MOHC · 2020-03-06T16:29:09Z

just to confirm secondary test - it works also on CNRM-CM6-1 where it picked up sftlf from amip. Again results agree with processing manually.

esmvalcore/_recipe.py

bouweandela · 2020-03-09T12:56:39Z

esmvalcore/_recipe.py

-                for fxvar in var['fx_files']}
-            settings[step]['fx_files'] = fx_files_dict
-            logger.info(msg, step, pformat(fx_files_dict))
+    """Update fx settings depending on the needed method."""


This function is needlessly complicated

hang on, looking at it in a jiffy

ok simplified now courtesy to @zklaus last suggestions, happy Bouwe or unhappy Bouwe? 🍺

Thanks, much better! I still think the function inside a function _get_fx_vars_from_attribute is not really needed, or do you have a particular reason for this?

deal with it 😆

esmvalcore/_recipe.py

Co-Authored-By: Bouwe Andela <bouweandela@users.noreply.github.com>

Co-Authored-By: Klaus Zimmermann <klaus.zimmermann@smhi.se>

valeriupredoi · 2020-03-12T15:33:33Z

waddup, guys? 🍺

JaroCamphuijsen · 2020-03-24T15:33:07Z

doc/esmvalcore/preprocessor.rst

+        weighting_landsea_fraction:
+          area_type: land
+          exclude: ['CanESM2', 'reference_dataset']
+          fx_files: [{'short_name': 'sftlf', 'exp': 'piControl'}, {'short_name': 'sftof', 'exp': 'piControl'}]


Wouldn't it be better to use the keyword fx_variables instead of fx_files? The current name suggests you have to give it a path to a certain file which might be confusing (see for example https://esmvaltool.readthedocs.io/projects/esmvalcore/en/latest/recipe/preprocessor.html#extract-shape with the shapefile keyword.

Also this looks like it could be kept more readable by using the same format as for data sets:

datasets: - {dataset: CanESM2, project: CMIP5, exp: historical, ensemble: r1i1p1, start_year: 2001, end_year: 2004} - {dataset: UKESM1-0-LL, project: CMIP6, exp: historical, ensemble: r1i1p1f2, start_year: 2001, end_year: 2004, grid: gn} - {dataset: EC-EARTH3, alias: custom_alias, project: CMIP6, exp: historical, ensemble: r1i1p1f1, start_year: 2001, end_year: 2004, grid: gn}

replaced with fx_variables -> the user can specify whatever keys they want and in whatever order they want them

the user can specify whatever keys they want and in whatever order they want them

sorry for not being clear, I didn't mean the should add these specific keys and order, just the way it is formatted, so instead of the current:
fx_files: [{'short_name': 'sftlf', 'exp': 'piControl'}, {'short_name': 'sftof', 'exp': 'piControl'}]
have the other format in the documentation:

fx_files: - {'short_name': 'sftlf', 'exp': 'piControl'} - {'short_name': 'sftof', 'exp': 'piControl'}

But it's just a suggestion. Polishing nail extensions.

Also I saw you now replaced the argument fx_variables on lots of other places as well, but the docstrings still say that these fx_variables then should be a dictionary with {field: filename of fx variable}. There's also some other places where still the name fx_files is used. maybe check whether in these instances that you now changed it's really not the full filename but just the variable name? If you change these couple, then change the other occurrences as well. (e.g def tile_grid_areas(cube, fx_files):)

Hi @valeriupredoi how was the pizza? Did you have a look at this already?

great pizza, cheers, I can't remember the toppings though 😁 So here's the thing: the way the user can ask for fx_variables in the recipe (as arg of the preprocessor step) can be either as a list of short names eg fx_variables: ['sftlf', 'sftof'] or as a list of dicts eg fx_variables: [{short_name: sftlf, mip: fx}, {short_name: sftof, mip: fx}] so making the listing look like the variable dictionary will mean that the user will be confused as to why it could also be just a simple list; tile_grid_areas actually takes as argument the fx files and not the variables, so that's why I left it with fx_files in

good catch though - I just realized I missed adding documentation for these funcs (now added) 🍺

Cool, if you think this won't cause confusion I'm perfectly fine with this!

should be fine, RTFM will play a good part here 😁

JaroCamphuijsen · 2020-03-24T15:52:55Z

esmvalcore/_recipe.py

    var = dict(variable)
    var_project = variable['project']
+    get_project_config(var_project)


You don't seem to use the returned value from get_project_config() here, do you intend to only use the check and error raising? And if so, can't we make it more clear that you're doing that?

replaced with an actual RecipeError'

esmvalcore/_recipe.py

bouweandela · 2020-04-09T14:36:39Z

ping @valeriupredoi

valeriupredoi · 2020-04-09T16:40:22Z

Soorry folks, on Easter hols (staying at home this year 😁 ), will revisit this bright and early on Tuesday 🐰

valeriupredoi · 2020-04-14T16:35:43Z

ayt, guys, have now plugged them suggestions, what say yous? 🍺

JaroCamphuijsen

Looks good. I think it's ready to merge.

valeriupredoi · 2020-05-05T14:29:28Z

cheers @JaroCamphuijsen mate, nice review 🍺

valeriupredoi added 4 commits March 6, 2020 11:22

ported changes from PR439_child2 ONLY for better handling of fx varia…

928a82b

…bles (and some changes from yapf)

implement fx files as dicts everywhere

0799ab4

tests ONLY for the current changes

9451edd

changes for new implementation with fx files as dicts

f00b188

valeriupredoi added enhancement New feature or request preprocessor Related to the preprocessor labels Mar 6, 2020

valeriupredoi requested review from zklaus, bouweandela, schlunma and ChrisJones-MOHC March 6, 2020 11:49

valeriupredoi added 4 commits March 6, 2020 12:56

removed redundant logger message

668c523

added doc

2b44852

fixed bug for area volume zonal statistics

962d681

fixed test and wrote a composite one

857f8d0

made logger message more informative

503b09a

bouweandela reviewed Mar 9, 2020

View reviewed changes

esmvalcore/_recipe.py Outdated Show resolved Hide resolved

bouweandela reviewed Mar 9, 2020

View reviewed changes

schlunma added this to the IPCC AR6 milestone Mar 10, 2020

zklaus reviewed Mar 10, 2020

View reviewed changes

esmvalcore/_recipe.py Outdated Show resolved Hide resolved

esmvalcore/_recipe.py Outdated Show resolved Hide resolved

valeriupredoi and others added 4 commits March 10, 2020 14:30

Bouwe suggestion

1a402a0

Co-Authored-By: Bouwe Andela <bouweandela@users.noreply.github.com>

Klaus suggestion

0981f27

Co-Authored-By: Klaus Zimmermann <klaus.zimmermann@smhi.se>

Klaus simplification

5e7ae06

bad var names

442a32a

valeriupredoi added 2 commits March 10, 2020 15:04

func arg missing and changed wrong suggestion

c7baf7b

Merge branch 'master' into upgraded_fx_handling

55a4c09

JaroCamphuijsen reviewed Mar 24, 2020

View reviewed changes

bouweandela reviewed Mar 24, 2020

View reviewed changes

esmvalcore/_recipe.py Outdated Show resolved Hide resolved

valeriupredoi added 8 commits April 14, 2020 16:07

Merge branch 'master' into upgraded_fx_handling

dc527ae

replaced with fx_variables and deleted obsolete paragraph

6e745ef

replaced with fx_variables

84fbe47

replaced with fx_variables

e2dc5c5

replaced with fx_variables

84c3d34

plugged a few suggestions and replaced with fx variables

2ebcb36

replaced with fx_variables

20ed52c

replaced with fx_variables

6f35f2e

valeriupredoi added 2 commits May 5, 2020 12:26

Merge branch 'master' into upgraded_fx_handling

71d42e1

added doc for fx variables for missing functions

a0e0950

JaroCamphuijsen approved these changes May 5, 2020

View reviewed changes

valeriupredoi merged commit 54180f7 into master May 5, 2020

valeriupredoi deleted the upgraded_fx_handling branch May 5, 2020 14:31

valeriupredoi mentioned this pull request May 5, 2020

Fallback option for weighting_landsea_fraction in case specific models don't have fx data #547

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

State of the art FX variables handling without preprocessing #557

State of the art FX variables handling without preprocessing #557

valeriupredoi commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

ChrisJones-MOHC commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

ChrisJones-MOHC commented Mar 6, 2020

bouweandela Mar 9, 2020

valeriupredoi Mar 10, 2020

valeriupredoi Mar 10, 2020

bouweandela Mar 24, 2020

valeriupredoi May 5, 2020

valeriupredoi commented Mar 12, 2020

JaroCamphuijsen Mar 24, 2020

JaroCamphuijsen Mar 24, 2020

valeriupredoi Apr 14, 2020

JaroCamphuijsen Apr 24, 2020

JaroCamphuijsen Apr 24, 2020

JaroCamphuijsen May 5, 2020

valeriupredoi May 5, 2020

valeriupredoi May 5, 2020

JaroCamphuijsen May 5, 2020

valeriupredoi May 5, 2020

JaroCamphuijsen Mar 24, 2020 •

edited

Loading

valeriupredoi Apr 14, 2020 •

edited

Loading

bouweandela commented Apr 9, 2020

valeriupredoi commented Apr 9, 2020

valeriupredoi commented Apr 14, 2020

JaroCamphuijsen left a comment

valeriupredoi commented May 5, 2020

State of the art FX variables handling without preprocessing #557

State of the art FX variables handling without preprocessing #557

Conversation

valeriupredoi commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

ChrisJones-MOHC commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

valeriupredoi commented Mar 6, 2020

ChrisJones-MOHC commented Mar 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valeriupredoi commented Mar 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JaroCamphuijsen Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

valeriupredoi Apr 14, 2020 • edited Loading

Choose a reason for hiding this comment

bouweandela commented Apr 9, 2020

valeriupredoi commented Apr 9, 2020

valeriupredoi commented Apr 14, 2020

JaroCamphuijsen left a comment

Choose a reason for hiding this comment

valeriupredoi commented May 5, 2020

JaroCamphuijsen Mar 24, 2020 •

edited

Loading

valeriupredoi Apr 14, 2020 •

edited

Loading