Differences between NUG/CF _FillValue and Zarr fill_value (and how Xarray handles them)

[ethanrd]

Question

Hi all,

I just wanted to point out an ongoing xarray (edit: Zarr) discussion (issue #5475) about NUG/CF _FillValue and Zarr fill_value. It appears that they mean (are used for) very different things but Xarray is currently treating them the same.

I haven't made it all the way through the conversation and I'm not sure when I will be able to get back to it. So if anyone else is interested in taking a look and commenting, that would be great.

Quick comparison (my understanding after a quick read):
Zarr fill_value is the value to use for undefined sections of an array, e.g., chunks that are not written. When the array is read, data from missing chunks is set to the value of fill_value. It does not necessarily imply missing data.

NUG/CF _FillValue is the value used to initialize variables when allocating memory or disk space before data was written. When _FillValue is found when reading a variable, it indicates that the data was never written for that index location in the variable. So it is always considered missing data.

[ChrisBarker-NOAA]

I see that they are technically different, but in practice, are they.

CF has missing_value, which is what should be used if you want to be explicit.

I know that despite that people treat _FillValue as missing values-- but what else can you do?

And what else could you do with zarr's fillvalue? Not set is not set, regardless of why.

Any idea what carry does if the three have different values?

[DocOtak]

It seems to me that zarr, or users, got itself into this ambitious situation by using fill_value in a situation where they actually wanted a default_value as a space saving/compression technique. Which is a fantastic idea in my opinion but probably should be formalized into its own independent concept (outside CF).

I think the one thing that CF should at least think about, is that _FillValue is netCDF implementation specific and I know there have been some efforts in distancing CF form the serialized formats that could represent it.

[cofinoa]

NUG/CF _FillValue is the value used to initialize variables when allocating memory or disk space before data was written. When _FillValue is found when reading a variable, it indicates that the data was never written for that index location in the variable. So it is always considered missing data.

For "chunked" netCDF variables, _FillValue is returned when entire chunks have no data written at all (i.e., like empty chunks in zarr).

It's important to note that _FillValue and missing_value serve different purposes. _FillValue represents data that was never written to disk, while missing_value represents data that was written but is flagged as missing. However, in certain cases, both values can be used to represent missing data in different contexts.

Moreover, missing data, could be encoded using more than on value missing_value, or indirectly with values out of the interval valid_range or [valid_min, valid_max].

My recommendation it's always to use an explicit value, different to _FillValue, for encoding missing data, and NUG and CF should be more explicit on this.

[larsbarring]

Yes, I absolutely agree with this

My recommendation it's always to use an explicit value, different to _FillValue, for encoding missing data, and NUG and CF should be more explicit on this.

And in particular with the last part that I boldfaced.

While I am currently not using zarr or xarray, I can imagine future problems if they decide (already have decided) to threat _FillValue and missing_value the same without functionality for distinguishing between them.

[cofinoa]

This issue is already manifesting, as highlighted in this GitHub issue. The core problem arises from xarray assuming that the missing_value attribute can only have a single value and treating missing_value as equivalent to _FillValue. This assumption leads to incorrect handling of missing data, as the two attributes serve distinct roles and should not be confused.

Further collateral issues have emerged, as noted in this related issue, due to the netCDF-C library (and NUG) exposing _FillValue as a "regular attribute" accessible to users, as seen here. In reality, _FillValue is a "special case" in the library, handled differently when attributes are defined, as described here.

The NUG does state:

Attribute names commencing with an underscore ('_') are reserved for use by the netCDF library.

However, this reservation applies specifically to the CDL syntax, and the documentation does not clearly emphasize this distinction.

This highlights the special nature of _FillValue, which functions more like properties such as chunking or deflation that are associated with the variable storage dispatch layer, rather than behaving as a general attribute.

[Probably this last part, it's completely out of scope of CF discuss]

[sethmcg]

One challenge here is that for a long time, there was a lot of software that recognized either _FillValue or missing_value but not both. So historically, it was good practice to define both and set them to the same value, because then your missing data would be interpreted as missing and not as legitimate data no matter what tools you were using.

[taylor13]

Yes, that's what we've done in CMIP for 20 years (and for the reason you state). missing=fill=same constant. Maybe a bit sloppy, but practical.

[JonathanGregory]

CF Appendix A describes _FillValue as "A value used to represent missing or undefined data" and missing_value as "A value or values used to represent missing or undefined data" i.e. the same, except that missing_value could be a vector. Section 2.5.1 on "Missing data, valid and actual range of data" doesn't draw any distinction of function between them, but recommends that if you want to use only one, _FillValue should be preferred.

[ChrisBarker-NOAA]

Thanks @dopplershift. I see this now in the CF docs:

"If only one missing value is needed for a variable then we recommend that this value be specified using the _FillValue attribute."

Though I don't see any other mention of it being a vector in the CF docs, but the NUG says "The missing_value attribute can be a scalar or vector containing values indicating missing data."

Perhaps we should strengthen/clarify that point, something along the lines of:

• If there is one missing value needed, it should. be the _FillValue attribute, and missing_value should not be present.

• If there is one value in missing value (not recommended), then it should be the same value as _FillValue

• If there are more than one value in missing_value, the the first should be the same as _FillValue

alternatively:

The missing values shall be determined by the union of the _FillValue and the missing_value vector.

• So if _FillValue and missing_value are different, then both are considered missing values.

All this gets more complicated with:

"Since version 2.4 the NUG defines missing data as all values outside of the valid_range, and specifies how the valid_range should be defined from the _FillValue (which has library specified default values) if it hasn’t been explicitly specified."

So what if valid_range and missing_value are both specified? the union of all of them??

And this: "valid_range should be defined from the _FillValue" is concerning .... though looking at the NUG, it seems reasonable. (though they don't mention NaN as a _FillValue -- presumably that would be valid range the full range of the float type)

[larsbarring]

I strongly argue that in the general case it is not helpful to combine _FillValue used for represent undefined data with missing data. Just think of a variable that is only defined over land (or in oceans, where there are no clouds, etc ....), i.e. where a mask is used. And for the areas where data exists there was quality control procedure that identifies one or several types of deficient data (data below/above valid range range, otherwise dubious data, and whatever). Now, you would want to know about the proportion of masked data in relation to the total domain/scene size, and also the proportion of problematic data covered by the mask. You might also want to know the proportion of data having each individual type of problem.

That is precisely why we in the general case need to make the distinction between _FillValue and missing_values. While this may not be required for specific use cases such as CMIP that @taylor13 refers to.

Thus I think that it is well motivated that CF improves the text on this to be come more clear and explicit.

[ChrisBarker-NOAA]

@larsbarring: I titall agree, and Im sure that's why missing_value was added in the first place.

However, it seems that ship has sailed -- i know this confused me when I first encountered it, and a LOT of people, and software, and now even the CF docs conflate the two.

However -- missing_value can be a vector -- and THAT is what one should use if you want to express multiple ways that data may be invalid, but in more than one way. So I think we can make use of that.

One way to do that (my preference) is to say that missing values are the set of _FillValue and the missing_value vector.

If they are the same, or _FillValue is the same as the first missing_value, then that's redundant information, but otherwise, no problem.

[larsbarring]

@ChrisBarker-NOAA I am happy that we agree!

Yes, the ship may have sailed but the question is what to do about it? Stand at the pier waving a (perhaps slightly wet) handkerchief, or try to muster a tug to assist the vessel back to a safe harbour from a bumpy journey through partly uncharted waters? (Sorry, couldn't resist the flowery metaphor :-)

If CF is not clear about the intended use of the machinery we provide , how can we ever expect data producers to know how use them, and even less how should software developers be able decode our intentions into into rules that can be formalised into code?

What we can do is to improve the relevant text of the CF convention to, as @cofinoa suggests be more explicit about this.

In the same vein, bouncing back the question to @ethanrd, even though it is not a matter for CF: would it be possible to update the NUG document to more clearly express that [often] it is necessary/relevant to make the distinction between undefined and missing data, for example here:

Generic applications often need to write a value to represent undefined or missing values. The fill value provides an appropriate value for this purpose because it is normally outside the valid range and therefore treated as missing when read by generic applications. It is legal (but not recommended) for the fill value to be within the valid range.

[davidhassell]

Chris notes:

"Since version 2.4 the NUG defines missing data as all values outside of the valid_range, and specifies how the valid_range should be defined from the _FillValue (which has library specified default values) if it hasn’t been explicitly specified."

This is important, since it means that there there is nearly always a range of missing values (i.e. those that are defined values which are flagged as missing), unless the _FillValue is set to, or defaults to, the largest/smallest representable number of that data type.

See the valid_range entry in https://docs.unidata.ucar.edu/nug/current/attribute_conventions.html where this is defined (although I'm not sure how to reconile the 3rd and 4th bullet points in that section).

The netCDF library _FillValue defaults, according to netCDF4-python, are not always the largest/smallest numbers:

>>> import netCDF4
>>> netCDF4.default_fillvals
{'S1': '\x00',
 'i1': -127,
 'u1': 255,
 'i2': -32767,
 'u2': 65535,
 'i4': -2147483647,
 'u4': 4294967295,
 'i8': -9223372036854775806,
 'u8': 18446744073709551614,
 'f4': 9.969209968386869e+36,
 'f8': 9.969209968386869e+36}

I also agree with Lars that "it is not helpful to combine _FillValue used for represent undefined data with missing data. "

[ChrisBarker-NOAA]

'f4': 9.969209968386869e+36,
'f8': 9.969209968386869e+36}

I haven't looked at the others, but f8 sure as heck looks like a bug.

and why not NaN? ???

I"ll try to go post on the project to ask ...

-CHB

[larsbarring]

Sorry, I don't recognize the bug. Could someone enlighten me?

[Armin-RS]

A "float" data type can handle numbers between 0 and +- 3.4028234663852886e+38

A "double" data type can handle numbers between 0 and +- 1.7976931348623157e+308

So it looks strange to me to
a) define NC_FILL_FLOAT as 100x smaller than FLOAT_MAX
b) use the same value for NC_FILL_DOUBLE, which for "double" data type is somewhere in the middle of the possible range of values

Why not use 3.4028234663852886e+38 and 1.7976931348623157e+308 for the "NC_FILL_xxxx" values ? Or a value very close to those maximum values ?

[ChrisBarker-NOAA]

I've just created an issue on the netCDF-c project:

Unidata/netcdf-c#3041

[larsbarring]

Ah, I see. But that is maybe not a bug, but perhaps a poor choice? But I am not sure about that either.

Instead I think it is be a pretty smart pick with the aim to get the largest integer that can be exactly represented in IEEE-754 float32, and will thus be exactly the same in float64. That is, one can do a roundtrip recasting and get exactly the same number. And this is because one can never know iwhether users are always careful about those things....

On the other hand, _FillValue = float32(1e20), which is used by CMIP, and others is not 10²⁰, the exact value is 100000002004087734272). That is float64(1e20) != float32(1e20).

[larsbarring]

And I should add that probably most geophysical applications use float64 to get increased precision rather than extending the range covered, but that is only what I guess.

[ChrisBarker-NOAA]

And I should add that probably most geophysical applications use float64 to get increased precision rather than extending the range covered, but that is only what I guess.

well, yes, very likely true -- but that does apply to the largest integer exactly represented :-) -- though yes, maybe numbers that large aren't common.

But anyway, I don't think that value is the largest integer that can be represented by float32 -- that would be, well the same as the largest number a float can represent -- and the largest of the range of integers that can be exactly represented is far smaller -- on order of 10^8 I think.

We'll see what the netcdf-C devs say.

I suppose the real lesson here is that you shouldn't use the defaults :-)

[ChrisBarker-NOAA]

(although I'm not sure how to reconcile the 3rd and 4th bullet points in that section).

which are:

If neither valid_min, valid_max nor valid_range is defined then generic applications should define a valid range as follows:

• If the data type is byte and _FillValue is not explicitly defined, then the valid range should include all possible values.
• Otherwise, the valid range should exclude the _FillValue (whether defined explicitly or by default) as follows.
• If the _FillValue is positive then it defines a valid maximum, otherwise it defines a valid minimum.
• For integer types, there should be a difference of 1 between the _FillValue and this valid minimum or maximum.
• For floating point types, the difference should be twice the minimum possible (1 in the least significant bit) to allow for rounding error.

I think the idea with 3 and 4 here is that a_value > minimum_value and a_value < maximum_value should always work reliably.

But yes, not for byte, where the _FillValue is 0, but they want 0 to be valid.

And this does point to NaN as being the best option for float types if you don't want to restrict the range.

[dopplershift]

If _FillValue is NaN, you can no longer do data[i][j] == _FillValue to check for missing data. Also, NaN can be multiple bit patterns, so you would then have to pick one.

[ChrisBarker-NOAA]

Such is the fun of NaN! yes, you'd need to do an is_nan check of some sort.

I'm of the opinion that that if you write code that deals with floats, you should be considering the special values. But yeah, that does make it a little harder to write generic code.

Anyway, I don't think CF makes recommendations about what values to use.

And I don't think netCDF is going to change its defaults.

Should CF discuss in any more detail what should be used?

NOTE: playing around with the Python netCDF4 lib -- if you create a variable without specifying the _FillValue, you get:

<class 'netCDF4._netCDF4.Variable'>
float64 fred(a_dim)
unlimited dimensions: 
current shape = (100,)
filling on, default _FillValue of 9.969209968386869e+36 used

and the resulting ncl is pretty simple:

netcdf junk {
dimensions:
	a_dim = 100 ;
variables:
	double fred(a_dim) ;

e.g. no _FillValue attribute.

and when you reload the file, you can tell that the default _FillValue is used (from the repr), but there's no easy way that I can find to see what the default fill value is.

That could be a deficiency of the Python wrapper, but this tells me that CF should perhaps require the _FillValue attribute to be set?

I also can't see an easy way to set it to the default value -- you can go look at the default_fillvals dict, and set it to that, but that's kind of awkward -- again, a missing feature in the Python lib, but still, I think it's harder to do than you'd think.

Maybe time for an issue?

[ChrisBarker-NOAA]

Replying to myself:

I put in an issue on the netCDF python lib -- looks like we're going to get both a way to get the fill value, even if there is no _FillValue attribute, and a way to easily set the _FillValue attribute to the default.

However, the C lib provides no handy way to do these things, and the lib explicitly says you don't need to set _FillValue if you wan tthe default, so there are, and will be for along time, files that use the default, and have no attribute set.

So, for CF:

a) I think we should add text that says that if no _FillValue (or missing_value) set, then the consumer should assume that the default was used (which could be touch wrong, it's possible to create an array with no fill value -- I assume that means garbage in any items that haven't been set).

b) I htink we should say that the _FillValue attribute should be set, even if it is the default -- that makes it a lot easier for clients to know what it is.

honestly, when thinking about all this, I've been thinking of how to write a robust function to figure out invalid values for an arbitrary variable -- it's very hard! Which goes against the aims of CF, I think :-( -- legacy is tough!