Apply select preprocessor to area, volume and zonal statistics fx variables and add use of fx_files as list of dictionaries

[valeriupredoi]

Before you start, please read CONTRIBUTING.md.

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• This pull request has a descriptive title that can be used in a changelog
• Add unit tests
• Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
• If writing a new/modified preprocessor function, please update the documentation
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes
• If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.

---

Closes #436 #440

[valeriupredoi]

OK @ledm this is now functional, can you give it a test please? To derive fx vars for volume_statistics add fx_var_preprocess: True in variable eg:

        diagnostics:
          diagnostic_name:
            variables:
              tos:
                preprocessor: preproc
                fx_var_preprocess: True
                project: CMIP6
                mip: Omon
                exp: historical
                start_year: 2000
                end_year: 2005
                ensemble: r1i1p1f1
                grid: gn
                additional_datasets:
                  - {dataset: CanESM5}

[valeriupredoi]

aye, finally! This was tested by me with a (rather comperehensive ) recipe:

# recipe_fxprep.yml
---
documentation:

  description: test recipe for fx var preprocessing for area/volume/zonal/statistics

  authors:
    - predoi_valeriu

preprocessors:
    prep:
        custom_order: true
        extract_volume:
            z_min: 0
            z_max: 100
        annual_statistics:
            operator: mean
        volume_statistics:
            operator: mean
            fx_files: [volcello, ]

diagnostics:
  diag:
    variables:
      thetao700_Omon:
        short_name: thetao
        preprocessor: prep
        fx_var_preprocess: True
        additional_datasets:
          - {dataset: GFDL-CM4,        project: CMIP6, mip: Omon, exp: historical, ensemble: r1i1p1f1, grid: gn, start_year: 2010, end_year: 2014}
          - {dataset: HadGEM3-GC31-LL, project: CMIP6, mip: Omon, exp: historical, ensemble: r1i1p1f3, grid: gn, start_year: 2010, end_year: 2014}
          - {dataset: UKESM1-0-LL,     project: CMIP6, mip: Omon, exp: historical, ensemble: r2i1p1f2, grid: gn, start_year: 2010, end_year: 2014}

@mattiarighi and @ledm pls fire up tests yourselves. There may still be corner cases that I haven't looked at but since that recipe above works am quite happy 🍺 I need to add documentation!

[valeriupredoi]

Please have a look at this comment #436 (comment) and let's discuss the design there, before continuing.

Please have a look at #436 (comment)

[valeriupredoi]

@bouweandela let's get this one reviewed, please mate - we need these features and we need them to work proper and without having to worry about iris changes. These features work in this (rather laborious) way and have been tested quite extensively by @ledm - on top of it all this PR introduces some useful task filters too that I see useful not only for fx variables

[valeriupredoi]

OK guys - major plot twist: while trying to fix a bug that @ledm reported yesterday I discovered that there was a major issue with the fx files generation as it was until now - for the sake of not repeating analyses, I was removing ancestors with the same name and leaving just one - but that one was totally unclear with which preprocessing set of steps was being generated with. I fixed this by:

• not removing any ancestors anymore
• writing the output preprocessed fx file in with the parent variable data
• when loading for parent variable happens, loading for the attached fx variable data happens too -> pretty much what iris should be doing (but it doesn't, really)
• fixed tests too
  🍺

[valeriupredoi]

@ledm are you still using this branch? Can I close and nuke if not?

[valeriupredoi]

@ledm are you still using this branch? Can I close and nuke if not?

Frenly ping @ledm - even if you said you're still using it I'd be inclined to close it since fixing those conflicts is reason for a Nobel prize for literature

[ledm]

Lol, this is from feb 2020? What a simple beautiful time that was! We went out for a curry in Reading - we were planning to go to Singapore. Fun times.

I'm working on the main branch and it appears that this branch's functionality is already included there? Either that or I have some workaround in my recipes, but I had assumed that this was already long merged!

[valeriupredoi]

ah yes, the good old pre-pandemic days, all a warm blur now 😭

The fx handling functionality and API has changed massively with the introduction of handling of cell measures and variables, thanks to @sloosvel we are now in the 21st century on that. I'll nuke this even though I hope this was useful to you at least before we had the current implements, so farewell a few weeks of coding and sweat 🏳️