added xray netcdf read/write test #49
Conversation
|
Very cool Rich! @julienchastang uses xray in the following notebook: https://github.com/Unidata/unidata-python-workshop/blob/master/casestudy.ipynb I'll let @julienchastang comment about adding this notebook for this year, as he knows the schedule best. |
|
It is very cool--the problem is that we can only teach what we actually use. :) The schedule is pretty full already, and personally I'm 👎 on squishing anything else in there. Also, there's a whole issue of xray vs. NCSS, which are essentially doing the same thing; I'm inclined to favor the solution I work on versus anything else. :) |
|
Oh, okay, I didn't see that @julienchastang was mentioning xray. I still think it would be nice to show users that they can use lon/lat/time slicing in xray, which works with any OPeNDAP data or local NetCDF files, not just datasets served by NCSS. I know my daughter, who just learned python last year, independently discovered xray, started using it for working with netcdf files, and loved it. She particularly loved that it was so easy to save to netcdf ( |
|
@rsignell-usgs - maybe I misunderstood xray, but can it do geospatial slices on datasets that do not have explicit lat/lon variables? That is, can I slice in lat/lon space if I only have x, y, and a projection variable? |
|
@lesserwhirls, no, it can't do that. It uses the pandas data model for slicing along indices, so it only works for 1D lon,lat, time and depth coordinates. |
|
And to be clear, I'm not completely anti-xray. I just don't want to teach it when I don't have good guidance for: "When should I use NCSS and when should I use xray?" I will add xray to a list of possible future topics. |
|
@shoyer, have I represented xray correctly here? |
|
I didn't think so, but I wanted to check. The 1D limitation is kind of a On Thu, Jul 16, 2015 at 8:40 AM, Rich Signell notifications@github.com
|
|
@lesserwhirls, so NCSS handles that? |
|
NCSS does not work yet, but it's in the works. OPeNDAP works just fine. I'm not sure if there is an archive of the NCEP HRRR output that is On Thu, Jul 16, 2015 at 8:50 AM, Rich Signell notifications@github.com
|
|
BTW, this PR just adds the notebook so you can have it there as an xray example. You don't have to teach it. 😺 |
|
@lesserwhirls , just to make sure I understand, if the 2D lon,lat arrays were supplied in the dataset, NCSS would be able to extract using lon/lat bounding box. But because only the 1D projected coordinate arrays are supplied along with a map projection, NCSS can't yet handle subsetting HRRR (and datasets like it) yet using the lon/lat bounding box. Is that right? |
|
@rsignell-usgs I'll tag @JohnLCaron here to make sure I am understanding this correctly, but... I am pretty sure that NCSS will work on a dataset with 2D lat/lon arrays, and I know it can work with 1D lat/lon arrays OR datasets with (x, y, projection info) (and still subset in lat/lon space because enough information is supplied with x, y, and projection info). NCSS can handle subsetting individual model runs and the "best" virtual datasets, just not the new 2D Time Collection virtual datasets...yet... The issue is that NCSS, at this point, does not understand how to work with 2D times (i.e. model run time, valid forecast time). It does not have the semantics to differentiate run time vs forecast time. |
|
That said, if you do an NCSS request on a 2D Time dataset, you will get a return, but it will include all run times and valid times. Any spatial and variable subsets will still be applied successfully. |
|
Working on this now in v5. Still a ways to go. On Thu, Jul 16, 2015 at 9:31 AM, Sean Arms notifications@github.com wrote:
|
|
I just tried NCSS "grid as point" time-series extraction on the HRRR and got this URL: but it only returns a single point, with no time value. This is also expected (for now), right? |
|
For grid as point, yes, as NCSS does not know how to deal with the time coordinates for the 2D dataset. But, individual model runs will work just fine (including "latest" and "best". |
xray and NCSS do have overlap in terms of subsetting remote datasets, but the overall goals of the projects are pretty different and complementary IMO. For example, you could download a netCDF with NCSS and then analyze it with xray. Xray is really more about manipulating datasets than subsetting them, e..g, you wouldn't use NCSS for resampling a time-series or calculating anomalies.
No direct plans yet, but it's something we've been talking about (e.g., in pydata/xarray#475). |
|
Sorry for coming into this discussion late. Yes, the casestudy notebook uses xray, but for the only lame reason that I prefer xray formatted output over netcdf4-python. I will briefly mention xray's capabilities during my section, but won't go beyond that. I agree that we should have a section on xray next year (can probably supplant the case study). Unidata staff have a very small percentage of time allocated to Python so we just did not have the time to bring xray into the workshop this year, alas. |
Seems a shame not to mention Xray to this crowd.
Here's a simple Notebook example of reading, slicing and writing using the GFS 0.5 m res model results (the same model used in the "netcdf_read" python script that uses NetCDF4 directly:
http://nbviewer.ipython.org/gist/rsignell-usgs/047235496029529585cc