Harmonize FillValue and missing_value during encoding and decoding steps

[andersy005]

As pointed out in jbusecke/xMIP#5, xarray appears to be very strict during the encoding and decoding steps even when there are (harmless) discrepancies between missing_value and _FillValue. For instance, when dtypes of missing_value and _FillValue are different, xarray gives up:

In [74]: from xarray.coding import variables                                                    

In [75]: import numpy as np                                                                     

In [76]: import xarray as xr                                                                    

In [77]: original = xr.Variable( 
    ...:         ("x",), 
    ...:         [0.0, -1.0, 1.0], 
    ...:         encoding={"_FillValue": np.float32(1e20), "missing_value": np.float64(1e20)}, 
    ...:     )                                                                                  

In [78]: coder = variables.CFMaskCoder()                                                        

In [79]: encoded = coder.encode(original)                                                       
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-79-9fbc3632e28b> in <module>
----> 1 encoded = coder.encode(original)

/glade/work/abanihi/devel/pangeo/xarray/xarray/coding/variables.py in encode(self, variable, name)
    156             raise ValueError(
    157                 "Variable {!r} has multiple fill values {}. "
--> 158                 "Cannot encode data. ".format(name, [fv, mv])
    159             )
    160 

ValueError: Variable None has multiple fill values [1e+20, 1e+20]. Cannot encode data. 

• Closes #xxxx
• Tests added
• Passes black . && mypy . && flake8
• Fully documented, including whats-new.rst for all changes and api.rst for new API

[andersy005]

@dcherian, what is the right way to do the type casting in encode()?

I thought of trying something along these lines:

encoding["missing_value"] = encoding["missing_value"].astype(data.dtype)

However, I quickly realized that this breaks when encoding["missing_value"] is not a numpy object.

EDIT:

I will try using np.asarray():

encoding["missing_value"] = np.asarray(encoding["missing_value"]).astype(data.dtype)

[dcherian]

Thanks for taking this on, Anderson.

You'll have to convert back from the numpy array before exiting that function. It may be simpler if you avoid the cast and just keep the change from equivalent to allclose_or_equiv.

For reference, here's how xarray determines dtype for netCDF4

xarray/xarray/backends/netCDF4_.py

Lines 131 to 141 in 4e9240a

	def _nc4_dtype(var):
	if "dtype" in var.encoding:
	dtype = var.encoding.pop("dtype")
	_check_encoding_dtype_is_vlen_string(dtype)
	elif coding.strings.is_unicode_dtype(var.dtype):
	dtype = str
	elif var.dtype.kind in ["i", "u", "f", "c", "S"]:
	dtype = var.dtype
	else:
	raise ValueError(f"unsupported dtype for netCDF4 variable: {var.dtype}")
	return dtype

(I found this by working looking for the createVariable statement here)

xarray/xarray/backends/netCDF4_.py

Lines 439 to 485 in 4e9240a

	def prepare_variable(
	self, name, variable, check_encoding=False, unlimited_dims=None
	):
	datatype = _get_datatype(
	variable, self.format, raise_on_invalid_encoding=check_encoding
	)
	attrs = variable.attrs.copy()
	fill_value = attrs.pop("_FillValue", None)
	if datatype is str and fill_value is not None:
	raise NotImplementedError(
	"netCDF4 does not yet support setting a fill value for "
	"variable-length strings "
	"(https://github.com/Unidata/netcdf4-python/issues/730). "
	"Either remove '_FillValue' from encoding on variable %r "
	"or set {'dtype': 'S1'} in encoding to use the fixed width "
	"NC_CHAR type." % name
	)
	encoding = _extract_nc4_variable_encoding(
	variable, raise_on_invalid=check_encoding, unlimited_dims=unlimited_dims
	)
	if name in self.ds.variables:
	nc4_var = self.ds.variables[name]
	else:
	nc4_var = self.ds.createVariable(
	varname=name,
	datatype=datatype,
	dimensions=variable.dims,
	zlib=encoding.get("zlib", False),
	complevel=encoding.get("complevel", 4),
	shuffle=encoding.get("shuffle", True),
	fletcher32=encoding.get("fletcher32", False),
	contiguous=encoding.get("contiguous", False),
	chunksizes=encoding.get("chunksizes"),
	endian="native",
	least_significant_digit=encoding.get("least_significant_digit"),
	fill_value=fill_value,
	)
	nc4_var.setncatts(attrs)
	target = NetCDF4ArrayWrapper(name, self)
	return target, variable.data

[andersy005]

@dcherian,

You'll have to convert back from the numpy array before exiting that function.

I was able to address this using @shoyer's suggestion above.

[dcherian]

Some minor comments. Looks great!
Thanks @andersy005

[pep8speaks]

Hello @andersy005! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-11-12 19:10:24 UTC

[andersy005]

@dcherian & @shoyer, thank you both for your help!

In which section(bug fixes? enhancements?) in whats-new.rst should I document these changes?

[dcherian]

Let's do bug fixes

[max-sixty]

Thanks @andersy005 !