xradar and pyart fillvalue discrepancy when working with ODIM

[pavlikp]

• xradar version: 0.3.0
• Python version: 3.11.4
• Operating System: Windows 11

Description

I am trying to convert some RAINBOW RB5 data to ODIM for further processing using pyart (the RB5 volumes can't be imported directly due to the variable number of range bins between sweeps not being supported by the pyart package). Anyway, the export and import both work fine in general, however, there is a problem when filling the missing values using the fillvalue/undetect. Here is a plot obtained from the RB5 volume converted using xradar to ODIM, then loaded and plotted through pyart.

The red areas on the plot should be masked, however they show up as areas of high reflectivity.

What I Did

# convert file
datatree = xradar.io.backends.rainbow.open_rainbow_datatree(filepath_start + datetime_str + product + filepath_extension)
xradar.io.export.odim.to_odim(datatree, datetime_str + product + '.h5',source='WMO:11887')

# load new file
radar = pyart.aux_io.read_odim_h5(datetime_str + product + '.h5')

Plot showing the unmasked sweep data.

plt.imshow(radar.fields['reflectivity_horizontal']['data'])

My temporary solution

I have modified the lines around:
https://github.com/openradar/xradar/blob/5d6a3b86b94860aaf966b47eedff24b8b61ed566/xradar/io/export/odim.py#L113C29-L113C29

from

val = value.sortby(dim0).values
fillval = _fillvalue * scale_factor
fillval += add_offset
val = (val - add_offset) / scale_factor
val[np.isnan(val)] = fillval

to

val = value.sortby(dim0).values
val = (val - add_offset) / scale_factor
val[np.isnan(val)] = _fillval

as the pyart import seems to expect the exact value that is not transformed to create the mask.

After this change, the plot in pyart seems as expected, with the correct mask:

Is this a bug or am I misunderstanding the usage of fillval in the ODIM format?

[kmuehlbauer]

Sorry for the delay here.

Rainbow does not have a notion of undetect/_FillValue. It has a notion of nodata which is natively coded with zero.

But nevertheless this example revealed a bug in the order of applying the _FillValue. We need to switch the last two lines, otherwise a wrong value is encoded.

val = value.sortby(dim0).values
fillval = _fillvalue * scale_factor
fillval += add_offset
val[np.isnan(val)] = fillval
val = (val - add_offset) / scale_factor

In general it is a major pain point, that several radar data formats only have nodata. NetCDF4-based formats have the inherent _FillValue. ODIM knows _Undetect and nodata.

You can specify _FillValue and _Undetect in the .encoding-dict of the respective moment. If none of those is provided xradar choses a default value (maximum of datatype).

[kmuehlbauer]

Xref #103

[BerendWijers]

Hi all,

This 'bug' seems to be still apparent in '0.5.0', any idea if and when this will be patched?

• OS
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.4 LTS
  Release: 22.04
  Codename: jammy
• xRadar version
  0.5.0

Thanks,
Berend

[kmuehlbauer]

@BerendWijers Please check if #173 fixes your issue. Thanks!

[BerendWijers]

@kmuehlbauer it does! Thank you!

[kmuehlbauer]

@kmuehlbauer it does! Thank you!

Great, I'll try to finish #173 and let others have a look there.