Diagnosing and Fixing ephemeral Segmentation Faults / HDF Errors

[HCookie]

Hi all,
I am using xarray and dask at scale, and have been running into issues with occasional complete segmentation faults. As in, some times on the same data and code it will fail, and other times it succeeds.
Occasionally, it will report more helpful information, like NetCDF- HDF error, malloc, free pointer, but other times no such luck.

The same code succeeds every time when running without a local dask cluster, but when a dask cluster is present, failures occur.

# Pseudocode to get the point across
def succeed():
    xr_operation()

def fail():
    from dask.distributed import Client
    client = Client()
    xr_operation()

I am looking for any help / tips on diagnosing and identifying the cause of these errors. It clearly has something to do with dask, its worker setup and netcdf, but further then that I am not sure.

Example Error

  Exception ignored in: <function CachingFileManager.__del__ at 0x151ab7e20ea0>
  Traceback (most recent call last):
    File "HIDDEN/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 250, in __del__
      self.close(needs_lock=False)
    File "HIDDEN/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 234, in close
      file.close()
    File "src/netCDF4/_netCDF4.pyx", line 2618, in netCDF4._netCDF4.Dataset.close
    File "src/netCDF4/_netCDF4.pyx", line 2585, in netCDF4._netCDF4.Dataset._close
    File "src/netCDF4/_netCDF4.pyx", line 2113, in netCDF4._netCDF4._ensure_nc_success
  RuntimeError: NetCDF: HDF error
  Segmentation fault
  $ HIDDEN/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 48 leaked semaphore objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d

These errors are occurring when loading large > 50 GB datasets from disk, modifying the shape, and values, and saving again to disk.
All files are netcdf compliant, and are being loaded with xr.open_mfdataset and saved with .to_netcdf.

Result of `xr.show_versions()`

>>> xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.12.1 [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
python-bits: 64
OS: Linux
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: None
LOCALE: ('C', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.3-development

xarray: 2024.6.0
pandas: 2.2.0
numpy: 1.26.4
scipy: 1.12.0
netCDF4: 1.7.1.post1
pydap: None
h5netcdf: 1.3.0
h5py: 3.11.0
zarr: 2.18.2
cftime: 1.6.3
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.3.7
dask: 2024.7.1
distributed: 2024.7.1
matplotlib: 3.8.3
cartopy: 0.22.0
seaborn: 0.13.2
numbagg: 0.8.1
fsspec: 2024.6.1
cupy: 13.2.0
pint: None
sparse: 0.15.1
flox: 0.9.8
numpy_groupies: 0.11.1
setuptools: 69.1.0
pip: 24.1
conda: None
pytest: 8.0.1
mypy: 1.10.0
IPython: 8.21.0
sphinx: None

What I have tried

Change	Result
Write to local filesystem	netCDF- Can't open HDF5 attribute, then worker timeouts
.compute(scheduler='single-threaded')	Works each time (but defeats the point)
.compute(rerun_exceptions_locally=True)	Same occasional complete Seg fault

[keewis]

this is a long-standing issue by now (see e.g. #7079).

Note that this only happens with a threading scheduler, so you can use scheduler="processes" to avoid this while still computing in parallel (though with the overhead of process communication).

[HCookie]

Thanks for pointing to that issue, it is reassuring to see this is not a quirk of my setup.
Setting scheduler="processes" interferes with the spin up of a local distributed cluster, so I assume they are not intended to run together. However, without the cluster, and setting scheduler results in a lock inheritance error.

Lock Error Stack Trace

  File "HIDDEN/lib/python3.12/site-packages/xarray/core/dataset.py", line 2327, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/xarray/backends/api.py", line 1346, in to_netcdf
    writes = writer.sync(compute=compute)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/xarray/backends/common.py", line 267, in sync
    delayed_store = chunkmanager.store(
                    ^^^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py", line 249, in store
    return store(
           ^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/dask/array/core.py", line 1229, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "HIDDEN/lib/python3.12/site-packages/dask/base.py", line 403, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/dask/multiprocessing.py", line 234, in get
    result = get_async(
             ^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/dask/local.py", line 500, in get_async
    fire_tasks(chunksize)
  File "HIDDEN/lib/python3.12/site-packages/dask/local.py", line 482, in fire_tasks
    dumps((dsk[key], data)),
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1479, in dumps
    cp.dump(obj)
  File "HIDDEN/lib/python3.12/site-packages/cloudpickle/cloudpickle.py", line 1245, in dump
    return super().dump(obj)
           ^^^^^^^^^^^^^^^^^
  File "HIDDEN/lib/python3.12/multiprocessing/synchronize.py", line 101, in __getstate__
    context.assert_spawning(self)
  File "HIDDEN/lib/python3.12/multiprocessing/context.py", line 374, in assert_spawning
    raise RuntimeError(
RuntimeError: Lock objects should only be shared between processes through inheritance

[keewis]

for LocalCluster, you'd need to set processes=True:

cluster = LocalCluster(..., processes=True, ...)
client = Client(cluster)

(I don't think we've been testing the local scheduler, so it doesn't surprise me too much that using that doesn't work too well. I might be wrong, though)

[HCookie]

It does indeed look as if there is a bug when using dask.config.set(scheduler='processes').
Even after setting the cluster with processes=True, the lock issues occur.