-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor CFWriter.save_datasets
and enable retrieval of equivalent xr.Dataset with scn.to_xarray()
#2259
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for starting this. I see a lot of TODO
s for refactoring. Do those still need to be done? Also, could/should this new Scene.to_xarray
be a keyword argument on to_xarray_dataset
(default to False)? I personally don't really like taking the generic name to_xarray
to be this very specific use case.
Lastly, can you add tests?
Also...isn't the full test suite run for draft PRs? |
...and why aren't the tests running? This is very strange. |
@djhoese no idea why the test is not running ... Few questions:
|
|
Hi @pytroll/satpy-core and @BENR0 . I will likely have time to work on this toward the end of the week. |
I think this should definitely be merged with the |
@ghiggi could you fix the merge conflicts? I'll have a shot at review this when it's done |
@mraspaud I might be able to give it a shot on Wednesday. Meanwhile ... tell me if
|
That's a lot of arguments, but if think this is necessary... |
I've been staring at this RTD issue for too long along with every other PR tonight. I have no idea what sphinx is complaining about. It seems to think |
I personally don't use ancillary variables, but I also don't use this writer. That said, ancillary variables are important and should be included in the NetCDF file. If there is a bug in the code so these never get added then we need to know why the tests don't show that as a problem. If there are two redundant pieces of code in the writer that look like they are collecting ancillary variables then yes one of them could/should be removed. |
No there are no redundant pieces of code. Currently, the CFWriter simply doesn't write ancillary variables to disk (since at https://github.com/pytroll/satpy/blob/main/satpy/writers/cf_writer.py#L808 we don't specify |
Maybe @zxdawn can comment it they're available? To summarize @ghiggi Can you tell if there are any tests that use ancillary variables (or some of the fake data has them)? |
Regarding the ancillary variables:
If you agree @djhoese, I would tend to ask you to merge the PR as it is, so that I can proceed with the CF refactoring. |
ha, it seems #1140 didn't pass the |
It may be hard to tell which ones use it since some may be loaded automatically by xarray as part of the normal
The nucaps way of doing it was a change by me (or rather, how I always did it). @mraspaud preferred including DataArrays in the Based on your private message to me on slack @ghiggi I am OK with this being merged and think the next PR should be a refactor-only PR (which can include converting tests to pytest). I could see an argument for include an ancillary variable fix, but that might be too much for a single PR and especially confusing with the amount of testing that would be needed. Given my lack of usage of the CF writer I don't think I should be the one to push the merge button. @mraspaud please give this one last review and merge it if acceptable while keeping in mind that @ghiggi plans to make more PRs after this one. |
@mraspaud milestone |
@ghiggi I answered yesterday the question about ancillary variables, is there still something to do there? |
@mraspaud I think no. Regarding the ancillary variables I currently kept the historical behaviour of the CF-writer:
If we merge this today, tomorrow I might be able to do a quick PR to reorg the functions in the |
Ok, let's merge this then, so that you can start on making it nicer. |
I agree, this is something I wanted to look into, and also profile in terms of extra computational efforts, e.g. when resampling. |
This PR:
CFWriter.save_datasets
Scene.to_xarray()
method to retrieve the equivalent xr.Dataset which is written by the CFWriter.Side note:
This PR could serve as an intermediate step to write in future a CF-compliant ZarrWriter, which essentially requires a different definition of the encodings. Nevertheless, after the merging of this PR, a CF-compliant Zarr dataset can be created with just
scn.to_xarray().to_zarr()
. Relates to Writer for Zarr format #2229Tests added
Fully documented
Add your name to
AUTHORS.md
if not there already