C3S_34e Meeting with Ouranos and CRIM: Jan 2021

[agstephens]

Proposed agenda - C3S_34e Meeting with Ouranos and CRIM: 20th Jan 2021

1. Minutes and actions from previous meeting

Previous minutes: #74

Previous actions:

• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67 - in progress
• AS: respond to Tom's message about a formal collaboration - done

2. Project update: c3s_34e [Ag]

• close to production service
• use of inventory files within our C3S service
• Changes to clisops/roocs_utils
• Testing, submodules, and banging our heads against the keyboard
• Travis-CI to GitHub actions
• Brexit impacts
• IS-ENES2 interest

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

4. CRIM/CEDA MOU - or letter of intent:

• the way forward?
• possible AI work?

5. Discussion:

• Averaging process
• Regridding work
• managing test datasets
• other activities

6. Agree next steps and next meeting

[agstephens]

Minutes - C3S_34e Meeting with Ouranos and CRIM: 20th Jan 2021

New actions

• AS: set up a meeting with CRIM & Ouranos - EO and ML. Agenda: Letter of intent; AI collaborations/opportunities; Other info. @cehbrecht can involve people from DKRZ and GERICS to the AI meeting.
• AS: push the idea to have an international 3-day sprint (running through the night across continents) for working on the Earth System Model STAC profile (Roadmap for merging with STAC NCAR/esm-collection-spec#21). @cehbrecht can involve people from DKRZ working on intake and STAC.

1. Minutes and actions from previous meeting

Previous minutes reviewed: #74

Previous actions:

• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67 - done
• AS: respond to Tom's message about a formal collaboration - done

2. Project update: c3s_34e [Ag]

• close to production service
• use of inventory files within our C3S service, used for:
  • limiting access to QA'd data only
  • returning original files if they are aligned with time subset
• Changes to clisops/roocs_utils:
  • Removing submodule; problems with per-file (get_file()) call; so using a single pytest.fixture to clone the test repo
• Travis-CI replacement:
  • some concern expressed about which service to use:
    • GitHub actions: drop-in alternative - but is not necessarily (or always) free
    • Internal GitLab (per institute), maybe with mirroring from GitHub - CEDA has some experience of this and can provide info if needed by Ouranos
• Brexit impacts: changes required for some of our non-ESGF data services that serve C3S
• IS-ENES3 interest: some interest in re-using our C3S WPS.

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos:

• Moving into catalog work:

  • setting up an intake catalog for THREDDS content
    • would like to use STAC later

• Pascal worked on a new clisops function to average over a polygon shape

  • planning to integrate over next month

• Travis has been working on PAVICS landing page and operational system.

• Put in a PR for improved subsetting feature:

  • using masking, and finding valid grid cells (e.g. on coastlines)
  • will look into issue that Elle raised

CRIM:

• keen to use and advance STAC usage

  • primarily focussing on STAC extension for machine-learning
    • for training datasets, adaptation datasets

• We will set up "Letter of Intent" with CEDA (see below)

4. CRIM/CEDA MOU - or letter of intent:

• the way forward?
• possible AI work?
• AS: set up meeting with CRIM & Ouranos - EO and ML. Agenda: Letter of intent; AI collaborations/opportunities; Other info

5. Discussion:

• Averaging process:
  • all agreed that CEDA will implement a simple average_over_dims function
  • we will also refactor clisops.core so that it has single functions that do one thing each - and then clisops.ops will join/chain them as required.
• Regridding work:
  • Martin demonstrated his latest investigations of adaptive masking with different regridding methods.
• managing test datasets:
  • we all agree that using the get_file() function was leading to HTTP GET failures due to github API limiting requests - we will simply use a single clone of the test data repo in testing
• other activities:
  • the rioxarray library is developing and is definitely worth tracking:
    • https://corteva.github.io/rioxarray/stable/rioxarray.html

6. Agree next steps and next meeting

• Set up a meeting about collaboration and AI
• Next monthly meeting: Thursday 18th February, https://ukri.zoom.us/j/98246249580