Dask arrays without distributed?

[Hoeze]

Hi, I'd like to run the XGBRegressor with dask arrays (from xarray zarr dataset) on a Ray cluster.
However, when running:

model = xgboost.XGBRegressor()
model.fit(x_train.data, y_train.data)

I just get this error message:
TypeError: Not supported type for data.<class 'dask.array.core.Array'>

Is there a simple way to achieve this?

• I am using dask.config.set(scheduler=ray_dask_get) as scheduler for dask.
• I want to avoid setting up a dask.distributed cluster
• I want to avoid loading the whole dataset into memory

[hcho3]

No, currently we do not support training with a Dask array unless you set up a distributed cluster. See the example at https://xgboost.readthedocs.io/en/latest/tutorials/dask.html

[trivialfis]

You will have to load the dataset into memory if using xgboost.

[Hoeze]

Thanks for your answers and making this a feature request :)

[trivialfis]

We can probably integrate it with the internal IterativeDMatrix .

[trivialfis]

I want to avoid loading the whole dataset into memory

I looked into this issue again. For using sklearn interface, loading the whole dataset is required.

The only way to avoid loading the whole dataset is using external memory in XGBoost. For a quick start see https://github.com/dmlc/xgboost/blob/master/demo/guide-python/external_memory.py . I wrote a simple wrapper for dask array during the development of this new interface. The idea is to get the chunks of data by calling to_delayed, then feed a chunk of data into XGBoost during each iteration in the data loading process.

But the external memory is still experimental until #7214 can be merged. Closing this issue as the interface is here now.