199 data request 20km regional projections for cordex cmip6 queensland future climate science

[charles-turner-1]

• Added Cordex CMIP6 Translators

• Beginning refactoring @rbeucher noted would be useful in Add support for BARPA py18 data store #211. There's a little more work to do here so I've left this as a draft PR for now.

@marc-white You added config/experiments/cmip6_ig45.yaml in this branch - I've separately added a config/metadata_sources/cordex_ig45.yaml which is doing largely the same thing but seemed to follow the other source conventions more closely. Are you able to confirm whether there's a convention I'm unaware of?

Closes #199

[codecov]

Codecov Report

Attention: Patch coverage is 98.55072% with 1 line in your changes missing coverage. Please review.

Project coverage is 96.89%. Comparing base (06a9d42) to head (b303840).
Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
src/access_nri_intake/catalog/translators.py	98.55%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #219      +/-   ##
==========================================
- Coverage   96.95%   96.89%   -0.06%     
==========================================
  Files           9        9              
  Lines         623      644      +21     
==========================================
+ Hits          604      624      +20     
- Misses         19       20       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[marc-white]

@charles-turner-1 there isn't a convention for naming the metadata sources that we're storing in the repository. This is a very recent change that was made to avoid repeating issues like #200. However, they're only reference copies, and the catalog itself should reference the metadata.yaml stored somewhere on /g/data. Therefore, you can really do what you like with this, so long as we stay internally consistent.

[marc-white]

You can just leave these as blank-blank, rather than a blank array.

[marc-white]

As above

[marc-white]

If this is the path where the 'live' metadata.yaml is being stored, then I'd recommend the reference copy's name/path within the repo match, i.e. be cmip6_ig45.

[marc-white]

I don't think this is a path where we want to be storing live YAML... see discussion on #200

[marc-white]

As it currently stands, it looks like you're going to load the same Intake-ESM catalogue into access-nri-intake-catalog twice, using two different translators and metadata.yaml files. I presume you wanted to use the cordex ones in the final merge?

[charles-turner-1]

Yup - the changes to cmip6.yaml seem to go with the draft metadata.yaml you added. The cordex experiments needed a different translator, so I've separated them out - just hadn't cleaned up that file - everything should be sensible now.

I'm not 100% sure I understand the discussion around YAML paths fully - either that or it appears we need to update a bunch of paths in config/*.yaml?

[marc-white]

@charles-turner-1 There's a few things about YAML paths:

• Typically, the metadata.yaml lives with the data. However, there are some "historical" experiments for which we don't have write access to their data area, so we have to store the metadata.yaml within our own xp65 project. Where in xp65 was debated, given there is also an access-nri-intake-catalog checkout in there somewhere we don't want to fiddle with/be dependent on in production.
• There were some experiments that we were already holding the metadata for in xp65, but it accidentally got deleted. That's why we're holding all the metadata for the live catalog in a single place (/g/data/xp65/admin/intake/metadata) to try and stop that happening again.
• Separately, @rbeucher asked that we start holding a reference copy of new experiment metadata.yamls within the access-nri-intake-catalog repository. This isn't for production use; it's just for reference in case the above problem happens again (and I guess might be handy for testing too, now that I think about it).