Import hangs when matplotlib installed but no display available

[jaicher]

What happened: On a device with no display available, importing xarray without setting the matplotlib backend hangs on import of matplotlib.pyplot since #5794 was merged.

What you expected to happen: I expect to be able to run import xarray without needing to mess with environment variables or import matplotlib and change the default backend.

[Illviljan]

What do you mean with "no display available"? You haven't installed matplotlib?
The CI runs without mpl as well and works fine there.

[mathause]

Can you let us know your OS and xr.show_versions().

Could be a bug of the "MacOSX" backend: matplotlib/matplotlib#19197 & matplotlib/matplotlib#19268. (However, we test on macos in our CI.)

[jaicher]

I have matplotlib installed.

Here's the output of xr.show_versions() (after rolling back to 0.19):

Output of xr.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:59:51)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.4.72-microsoft-standard-WSL2
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.1
libnetcdf: 4.8.1

xarray: 0.19.0
pandas: 1.3.4
numpy: 1.21.4
scipy: 1.7.3
netCDF4: 1.5.8
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: 2.10.3
cftime: 1.5.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.11.2
distributed: 2021.11.2
matplotlib: 3.5.1
cartopy: None
seaborn: 0.11.2
numbagg: None
pint: None
setuptools: 59.4.0
pip: 21.3.1
conda: None
pytest: None
IPython: 7.30.1
sphinx: 4.3.1

By "no display available", I meant on a headless server (e.g. WSL2, compute node on slurm cluster).

Digging more into this, this is related to matplotlib/matplotlib#17396 (matplotlib >= 3.4):

• matplotlib tries to automatically select the plotting backend, especially whether or not there is a display to use.
• On Linux, matplotlib < 3.4, this was effectively done by checking if DISPLAY environment variable was set.
• For matplotlib >= 3.4, Improve headlessness detection for backend selection. matplotlib/matplotlib#17396 added additional check to call XOpenDisplay(NULL) to see if X11 is able to connect to the DISPLAY environment variable.
• This was to fix instances where DISPLAY was set but no longer valid. A common example is on slurm (or some other cluster managers) opening a compute node which by default copies all environment variables, including DISPLAY, to a new node.

I experience hanging because my WSL2 setup sets DISPLAY to reference an X11 server that isn't always running. Since the X11 server is on Windows, the DISPLAY refers to a remote address. Apparently, X11 doesn't give up when it isn't able to connect to a remote X11 server, and, so it hangs on that command.

That is for matplotlib >= 3.4. For matplotlib = 3.3 (which is still less than a year old) and xarray=0.20, I get:

• (RedHat on cluster, selects TkAgg backend): is able to import, but fails to plot afterwards
• (WSL2, attempts to select QtAgg backend): fails import (and exits Python!) with message:

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the
application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

Aborted

This will become a little bit less of an issue when we set the minimum version of matplotlib to 3.4. However, I still think we should revert PR #5794 (PR #6064). It doesn't fix any bug. Before matplotlib >= 3.4, the common-knowledge I had previously seen from writing scripts with headless-use of matplotlib was that we shouldn't import matplotlib.pyplot until we needed it. This was consistent with the previous imports in the plotting functions. Reverting will make it so that errors related to how the plotting environment is set up only take place when users explicitly want to make plots, rather than for all imports of xarray, many of which do not involve any plotting whatsoever.