Avoid default tokenization in Dask

[rjzamora]

I spent some time investigating a concurrent.futures._base.CancelledError error in test_gpu_with_dask.py::TestDistributedGPU::test_categorical and eventually bisected the cause to dask/dask#10883

That PR changed how Booster objects are tokenized by dask, and effectively moved from using a random uuid-based approach to a deterministic hash. I haven't had a chance to figure out exactly why the new deterministic hash causes problems, but this PR opts out of the new approach for now.

cc @trivialfis

Addresses part of #10379

[rjzamora]

All changes in this file are related to the recent deprecation of dask_cudf.from_dask_dataframe.

[trivialfis]

Thank you for the fix! We recommend users to scatter the booster before running prediction, this way we can reuse the same booster already on the workers for multiple prediction calls instead of transferring it repeatedly. Does the PR change that?

[rjzamora]

We recommend users to scatter the booster before running prediction, this way we can reuse the same booster already on the workers for multiple prediction calls instead of transferring it repeatedly. Does the PR change that?

This PR should produce the same behavior as dask<2024.2.1. You should still be able to scatter the model ahead of time and use the futures for multiple prediction calls. However, if you scatter the same model twice, dask won't recognize that the model hasn't changed, and so the same model will need to be transferred again (with different key names).

I'm not entirely sure what is causing the "cancelled" error for dask>=2024.2.1, but it seems likely that dask's attempt to tokenize Booster objects deterministically is a bit "off". It seems like different/retrained models can be incorrectly hashed to the same token. For example, if a model is retrained, you would want dask to use new keys to represent the corresponding futures. In dask>=2024.2.1, this does not seem to be the case (In test_empty_partition, I added a break point and found that both bst["booster"] and bst_empty["booster"] tokenize to the same value).

It may also be the case that deterministic tokenization is working "fine" for Booster objects, but xgboost happens to be using dask in a way that only works if a newly scattered model is always treated as unique.

[trivialfis]

Thank you for the detailed explanation!

[crusaderky]

Two consecutive calls to tokenize() will return two different values, and this can cause problems. I would advise caching the output: dask/dask#11179 (comment)

[rjzamora]

Thanks @crusaderky ! The purpose of this PR was to effectively roll back behavior to "match" tokenization before 10883. However, I agree that it makes sense to cache the value.