[dask] Add scheduler address to dask config. #7581
Conversation
trivialfis
force-pushed
the
scheduler-addr
branch
from
56d5ff5 to
b913c2f
Compare
trivialfis
force-pushed
the
scheduler-addr
branch
from
76ed1cd to
cff0733
Compare
trivialfis
force-pushed
the
scheduler-addr
branch
from
7de4a15 to
c51eabe
Compare
| # 2. Guessed by xgboost `get_host_ip` function | ||
| # 3. From dask scheduler | ||
| # We try 1 and 3 if 1 is available, otherwise 2 and 3. | ||
| valid_config = ["scheduler_address"] |
There was a problem hiding this comment.
When extending the Dask config system we typically create a new YAML with default values in and extend the Dask config with it.
See how we do it in dask-ctl.
There was a problem hiding this comment.
Thank you for sharing, let me look into that.
| for k in dconfig: | ||
| if k not in valid_config: | ||
| raise ValueError(f"Unknown configuration: {k}") | ||
| host_ip: Optional[str] = dconfig.get("scheduler_address", None) |
There was a problem hiding this comment.
I'm curious why this dconfig object is being passed around instead of accessing dask.config.get("xgboost.scheduler_address") directly.
There was a problem hiding this comment.
I couldn't access it on the worker side for some reason (key not found). I read the doc from https://docs.dask.org/en/stable/configuration.html , it's not quite clear to me how to make sure everything is synced.
There was a problem hiding this comment.
I also read this section https://docs.dask.org/en/stable/configuration.html#downstream-libraries . In XGBoost dask is loaded lazily so having dask in init might be less desirable due to the size of dependency. So I went on and read this:
However, downstream libraries may choose alternative solutions, such as isolating their configuration within their library, rather than using the global dask.config system. All functions in the dask.config module also work with parameters, and do not need to mutate global state.
So far using this parameter passing approach seems to be the easiest way to get 1 parameter in.
There was a problem hiding this comment.
That makes sense. Generally the Dask config should be available on the scheduler/workers but that can be deployment dependent.
- Add user configuration. - Bring back to the logic of using scheduler address from dask. This was removed when we were trying to support GKE, now we bring it back and let xgboost try it if direct guess or host IP from user config failed.
[backport] [dask] Add scheduler address to dask config. (dmlc#7581)
See added doc for use case.