Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xgboost 2.0.1 is breaking on rootless docker. #10250

Closed
prafullat opened this issue May 3, 2024 · 10 comments · Fixed by #10623
Closed

xgboost 2.0.1 is breaking on rootless docker. #10250

prafullat opened this issue May 3, 2024 · 10 comments · Fixed by #10623

Comments

@prafullat
Copy link

On rootless docker, path cgroup is readable only by root user.
image

As such, if any non-root users even tries to import xgboost, it fails with permission error.
Permission denied [/sys/fs/cgroup/cpu.max]

Can you please check and avoid this error is this path is not accessible?
We are not able to use xgboost in the rootless docker container.

@hcho3
Copy link
Collaborator

hcho3 commented May 3, 2024

@prafullat Can you post the full stack trace for the permission error?

@trivialfis
Copy link
Member

Will try to reproduce once I get back to work. At the moment my guess is this line to be blamed

auto has_v2 = fs::exists(bandwidth_path);

@trivialfis
Copy link
Member

trivialfis commented May 6, 2024

I have tried to launch an image with and without gosu, running in rootless mode of docker, but couldn't reproduce the issue. @hcho3 might have better insight here.

$ docker info | grep "Root"
WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
 Docker Root Dir: /home/ubuntu/.local/share/docker
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support

@trivialfis
Copy link
Member

Feel free to reopen if there is further information. Will try to fix it if there's a reproducer.

@zhangzzk
Copy link

I am having the exact issue with a non-root user importing the package:

>>> import xgboost terminate called after throwing an instance of 'std::filesystem::__cxx11::filesystem_error' what(): filesystem error: status: Permission denied [/sys/fs/cgroup/cpu.max] Aborted (core dumped)

@trivialfis trivialfis reopened this Jul 17, 2024
@trivialfis
Copy link
Member

Could you please share how you installed docker (or any other environment)?

@zhangzzk
Copy link

Thank you for the quick reply.
Sorry I am a very beginner at this. I am a non-root user on a shared computing cluster and I am not the one setting up the environment. Could you perhaps give more specific instructions on the information I can provide?

@trivialfis
Copy link
Member

Could you perhaps give more specific instructions on the information I can provide?

Unfortunately, I want these instructions as well. ;-( Maybe you can share the result of docker info so that I can compare it with mine? @hcho3 might be able to provide more insights into the system setup.

@zhangzzk
Copy link

I think I am not running on a docker so docker info is not a valid command.

@trivialfis
Copy link
Member

I will open a PR to catch all potential exceptions in that function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants