cf-coding

[kmuehlbauer]

• xref Saving and loading an array of strings changes datatype to object #7652
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst
• includes
  • implement coders
  • preserve boolean-dtype within encoding, adapt test
  • determine cf packed data type
  • transform numpy object-dtype strings (vlen) to numpy unicode strings

This should also fix:

• Closes nan values appearing when saving and loading from netCDF due to encoding #7691
• Closes float32 instead of float64 when decoding int16 with scale_factor netcdf var using xarray  #2304
  and possible other issues which have the float32/float64 issue

[kmuehlbauer]

I'll add to whats-new.rst if an updated version is merged.

[kmuehlbauer]

The one failed run might be spurious. Maybe a re-run of this would work.

[basnijholt]

Thanks a lot for the quick PR!

I can confirm that this fixes

• Saving and loading an array of strings changes datatype to object #7652 (comment) (bool -> int8)

But not int64 -> int32, and <U1 -> O

• Saving and loading an array of strings changes datatype to object #7652 (comment)
• Saving and loading an array of strings changes datatype to object #7652 (comment)

[kmuehlbauer]

@basnijholt Thanks for testing. I can't reproduce this. Here everything works as expected. But I've a slightly different environment (full netcdf4-stack, latest versions of everything).

The tests which check for dtype equality (vlen-case) do not raise in this PR for any backend and array container, so I'm assuming this should work as expected (at least for bool->int8, and <U1 -> O.

As stated over at #7652 I can't reproduce the int64->int32 conversion in any environments I tested so far. I'll have another look at your environment.

[kmuehlbauer]

I've added some explanations and concerns. I'd very much appreciate comments and guidance here.

Pinging @shoyer and @max-sixty as authors of relevant code.

[kmuehlbauer]

But not int64 -> int32, and <U1 -> O

@basnijholt I've explained the issues whereabouts over at your issue #7652 (comment).

[kmuehlbauer]

XRef: #2040 (comment)

Citing @shoyer from above comment:

The main reason why we don't do any special handling for object arrays currently in xarray is that our conventions coding/decoding system has no way of marking variable length string arrays. We should probably handle this by making a custom dtype like h5py that marks variables length strings using dtype metadata: http://docs.h5py.org/en/latest/special.html#variable-length-strings

Another interesting issue by @shoyer:
#2059

I'm really uncertain if using .astype here is the right way to go. Any comments appreciated.

[kmuehlbauer]

And yet another related issue: #1647

Currently both netcdf/h5netcdf are able to set a _FillValue for VLEN strings, but for the numpy-side "NaN" can only be handled with dtype=object. Maybe it's time to consolidate string handling in xarray. But that should be taken care of in a separate issue / feature branch.

[dcherian]

Any comments appreciated.

Let's discuss at next week's meeting. @kmuehlbauer can you make it? 9.30am MT Wed 29 Mar 2023

[kmuehlbauer]

make

That's 15 UTC, or 17 CEST (my local time). Should work for me. I'll try to collect all available information on that topic and the current status.

[basnijholt]

Just leaving a note here. I would expect that the datatype that was saved, is the datatype that is loaded. So preferably if I save a string array of e.g., type <U5, I expect it would still be <U5 when loaded, not suddenly HDF5 VLEN types.

Thanks again @kmuehlbauer for digging into this problem and all your work! 😄

[kmuehlbauer]

@dcherian @basnijholt

After the dev-meeting I've taken a step back and first implemented the coders as mentioned in @shoyer's ToDo.

I've fixed the one bool->int issue and it now derives the dtype for ScaleOffset coding from scale_factor add_offset.

I've improved some test with regard to the scale/offset issue.

I'll concentrate on the string fillvalue issues in a follow up PR.

[kmuehlbauer]

@dcherian @Illviljan Thanks for the first round of review. I've rebased everything on latest main. Now the code moving from conventions.py to coding.variable.py is correct. I've also removed the functions which have been converted to VariableCoders and adapted the tests.

To sum up this PR, it does:

• convert functions to VariableCoders along @shoyer's TODO:
  

  xarray/xarray/conventions.py

  Lines 298 to 302 in 1c81162

  	# TODO(shoyer): convert all of these to use coders, too:
  	var = maybe_encode_nonstring_dtype(var, name=name)
  	var = maybe_default_fill_value(var)
  	var = maybe_encode_bools(var)
  	var = ensure_dtype_not_object(var, name=name)

  
  

  xarray/xarray/conventions.py

  Lines 393 to 405 in 1c81162

  	# TODO(shoyer): convert everything below to use coders
  	if decode_endianness and not data.dtype.isnative:
  	# do this last, so it's only done if we didn't already unmask/scale
  	data = NativeEndiannessArray(data)
  	original_dtype = data.dtype
  	encoding.setdefault("dtype", original_dtype)
  	if "dtype" in attributes and attributes["dtype"] == "bool":
  	del attributes["dtype"]
  	data = BoolTypeArray(data)

• preserve boolean dtype within encoding:
  Saving and loading an array of strings changes datatype to object #7652 (comment)
• deterrmine cf packed dtype from scale_factor/add_offset
nan values appearing when saving and loading from netCDF due to encoding #7691, float32 instead of float64 when decoding int16 with scale_factor netcdf var using xarray  #2304

[kmuehlbauer]

@Illviljan I'm not able to figure out the typing if I want to use Data-types as functions to convert python numbers to array scalars. If you have any suggestion how to fix this, please let me know.

[dcherian]

@mankoff do you have time to take a look here please?

[mankoff]

Hi @dcherian . I'm not sure what to look for. I will link to my open-but-stale PR that tried to start addressing this/similar issues: #6812

Perhaps also relevant are my last few comments on #2304 (see #2304 (comment) ). The problem for me is that (1) the CF standards are ambiguously defined and (2) xarray needs to address the many use-cases where the CF standards are not followed (usually this means different data types).

[dcherian]

Thanks! Does this PR fix your original issue?

[kmuehlbauer]

Thanks @mankoff, I'll have a look at your extensive notes over there and try to come up with aomething.

[kmuehlbauer]

Still hunting for corner cases and issues inside encode_cf_variable/decode_cf_variable.

It looks like I already see some light again. Not sure, if this is the last iteration, but the testsuite is still running green with added and enhanced tests, which is not that bad.

Unfortunately #2304 is still an issue for now. I'll clarify that later with an added test.

[kmuehlbauer]

@dcherian Just a heads-up: I find this PR getting more and more involved at different parts of the machinery and hard to follow for reviewers. I'll split this up and start with the more or less undisputed changes.

[kmuehlbauer]

As explained I've created two PR (#7719 and #7720) for the "easy" changes from this PR. Would be great, if those could go in fast. Thanks!

[kmuehlbauer]

I've converted to draft for now, as I'm still evaluating solutions for the CF encoding/decoding.

[kmuehlbauer]

I'll close this one. Most things have been addressed in other PR's.