Xgboost compiles successfully but some interfaces cannot be called

[5201314wq]

I have added xgboost in my bazel project with rules_python.
After I build my project, there be a xgboost directory in pypi__xgboost_0_80 of my project's runfiles.and in xgboost, i can find rabit.py， But when I call xgb.train() in my code, I get an error as follows：
File "//bazel-bin//test.runfiles/pypi__xgboost_0_80/xgboost/training.py", line 52, in _train_internal
assert(rabit.get_world_size() != 1 or version == 0)
AttributeError: 'module' object has no attribute 'get_world_size'

I have also tried other versions of xgboost and bazel 0.13.1 and 0.17.1, but it doesn't work.
when i use ‘pip install xgboost==0.80’, everything works fine.

[cmgreen210]

Were you able to solve this? I'm running into the same problem.

[brandjon]

Missing elements of a module are a sign of the wrong module being imported.

Looking at dmlc/xgboost, if that's the right repository, I see that the problematic module name is imported as from . import rabit. There's a rabit.py in that directory. But there's also a rabit/ directory at the root of the repository, and the pip rules put the repository root on PYTHONPATH. So I believe the module is conflicting with that directory as a namespace package.

I'm not super familiar with namespace packages, but from googling, this looks potentially relevant.

Looking at the top-level of the repo, it also appears like maybe it's the subdirectory python-package that should be put on PYTHONPATH rather than the repo root. I don't see anything about customizing the imports of the py_library that the pip rules generate -- are you customizing the path somehow?

[cmgreen210]

@brandjon thank you for your very prompt reply! Indeed, the rabit file and directory are the issue. Luckily, I've have a way to patch pip packages and remove (or rename) the rabit directory (which I don't believe is necessary for the library to function). Thanks!

[brandjon]

Glad that explains it. Filed #157 for the feature request to address this in the future, closing this issue.

[scele]

I don't think #157 will solve this. I assume pypi__xgboost_0_80 here was extracted from a wheel, the layout of the source code repo where the wheel was built is irrelevant. If you look at the wheel (download e.g. here), it contains both rabit.py and rabit subdirectory (and no python-package subdir at all). So the question seems to be, how come pip-installing the module will resolve to rabit.py, but when used via bazel it will resolve to the rabit subdir?!

@brandjon if you agree, would you mind re-opening this issue?

[brandjon]

Right. Thanks for pointing that out.

[groodt]

I've ran into this issue too. After much head-scratching, it appears to be caused by the behaviour where empty __init__.py files are added into Python packages when the wheels are unpacked and spread.

You can disable this behaviour with: --incompatible_default_to_explicit_init_py which will eventually become the default from my understanding.

This only partially fixes the problem. For xgboost 0.90 you also need purelib support, which is missing from rules_python, but available in other rules. #92

[thundergolfer]

@5201314wq it's been a long long while, but if you try the pip_install repo rule with xgboost==1.3.3 things should work for you 🙂.