libstdc++.so.6: version `GLIBCXX_3.4.30' not found

[avriiil]

I created a Coiled software environment using coiled-runtime and the following .yml:

name: xgboost-coiled
channels:
  - defaults
  - conda-forge
dependencies:
  - python==3.9.6
  - coiled-runtime
  - dask-ml==2022.5.27
  - xgboost==1.5.1

I can confirm that coiled-runtime==0.0.4 was installed

Trying to run a basic computation on the cluster fails with the following error:

ImportError: /opt/conda/envs/coiled/lib/python3.9/site-packages/snappy/../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/envs/coiled/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.800)

"Minimal" Code

import coiled
import dask.dataframe as dd
from distributed import Client

coiled.create_software_environment(
    account="coiled-examples",
    name="xgboost-coiled",
    conda="environment.yml",
)

cluster = coiled.Cluster(
    name="xgboost-2",
    software="coiled-examples/xgboost-coiled",
    n_workers=25,
    worker_memory='16Gib',
    shutdown_on_close=False,
)

client = Client(cluster)

data = dd.read_parquet(
    "s3://coiled-datasets/dea-opioid/arcos_washpost_comp.parquet", 
    compression="lz4",
    storage_options={"anon": True},
    columns=columns+categorical,
)

data ## runs

data.head() ## fails 

Full traceback below

Full traceback:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [6], in <cell line: 1>()
----> 1 data.head()

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/dask/dataframe/core.py:1225, in _Frame.head(self, n, npartitions, compute)
   1223 # No need to warn if we're already looking at all partitions
   1224 safe = npartitions != self.npartitions
-> 1225 return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/dask/dataframe/core.py:1259, in _Frame._head(self, n, npartitions, compute, safe)
   1254 result = new_dd_object(
   1255     graph, name, self._meta, [self.divisions[0], self.divisions[npartitions]]
   1256 )
   1258 if compute:
-> 1259     result = result.compute()
   1260 return result

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/dask/base.py:312, in DaskMethodsMixin.compute(self, **kwargs)
    288 def compute(self, **kwargs):
    289     """Compute this dask collection
    290 
    291     This turns a lazy Dask collection into its in-memory equivalent.
   (...)
    310     dask.base.compute
    311     """
--> 312     (result,) = compute(self, traverse=False, **kwargs)
    313     return result

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/dask/base.py:600, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    597     keys.append(x.__dask_keys__())
    598     postcomputes.append(x.__dask_postcompute__())
--> 600 results = schedule(dsk, keys, **kwargs)
    601 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/client.py:3000, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2998         should_rejoin = False
   2999 try:
-> 3000     results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   3001 finally:
   3002     for f in futures.values():

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/client.py:2174, in Client.gather(self, futures, errors, direct, asynchronous)
   2172 else:
   2173     local_worker = None
-> 2174 return self.sync(
   2175     self._gather,
   2176     futures,
   2177     errors=errors,
   2178     direct=direct,
   2179     local_worker=local_worker,
   2180     asynchronous=asynchronous,
   2181 )

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/utils.py:320, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    318     return future
    319 else:
--> 320     return sync(
    321         self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    322     )

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/utils.py:387, in sync(loop, func, callback_timeout, *args, **kwargs)
    385 if error:
    386     typ, exc, tb = error
--> 387     raise exc.with_traceback(tb)
    388 else:
    389     return result

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/utils.py:360, in sync.<locals>.f()
    358         future = asyncio.wait_for(future, callback_timeout)
    359     future = asyncio.ensure_future(future)
--> 360     result = yield future
    361 except Exception:
    362     error = sys.exc_info()

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/tornado/gen.py:762, in Runner.run(self)
    759 exc_info = None
    761 try:
--> 762     value = future.result()
    763 except Exception:
    764     exc_info = sys.exc_info()

File ~/mambaforge/envs/xgboost-coiled/lib/python3.9/site-packages/distributed/client.py:2037, in Client._gather(self, futures, errors, direct, local_worker)
   2035         exc = CancelledError(key)
   2036     else:
-> 2037         raise exception.with_traceback(traceback)
   2038     raise exc
   2039 if errors == "skip":

File /opt/conda/envs/coiled/lib/python3.9/site-packages/distributed/protocol/pickle.py:66, in loads()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/dask/dataframe/io/parquet/arrow.py:8, in <module>

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pyarrow/__init__.py:65, in <module>

ImportError: /opt/conda/envs/coiled/lib/python3.9/site-packages/snappy/../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/envs/coiled/lib/python3.9/site-packages/pyarrow/../../../libarrow.so.800)

[avriiil]

@ncclementi does this look familiar to you by any chance?

[ncclementi]

Noup I have not seen this, but I wonder if there is a dependency problem because the xgboost version is older, I'd suggest a couple of things first:

• I'd avoid using the defaults channel as first priority (you can remove it directly), is there any reason for this?
• Is there a reason why you are using an older xgboost? that version still has problems, xgboost most recent version is not updated in conda-forge (it's still blocked by Upgrade to 1.6.1 conda-forge/xgboost-feedstock#88) but you can try couple of things and see if this unblock you. In your yaml file do this instead and see what happens.

name: xgboost-coiled
channels:
  - conda-forge
dependencies:
  - python=3.9.6
  - coiled-runtime
  - dask-ml=2022.5.27
  - pip
  - pip:
      - xgboost==1.6.1

[avriiil]

Thanks @ncclementi - will try that out.

[avriiil]

Running into an error with xgboost and pip now 😔 Have opened an issue: dmlc/xgboost#4949 (comment)

[ntabris]

xgboost 1.6.1 on pypi has builds for x86 for various MacOS versions and arm build for MacOS 12, but not arm for MacOS 11. Since you don't fit either of those, pip is trying to build xgboost from source. That's not particularly an xgboost issue, you just don't have tools installed for build (in particular, cmake). Haven't tried but brew install gcc might help.

[avriiil]

Thanks @ntabris - I will try upgrading my MacOS tonight.

[ncclementi]

@rrpelgrim is this still an issue, were you able to upgrade your OS and give it a try?

[avriiil]

thanks for the ping @ncclementi - upgrading my OS did the trick. Closing.