Rewrite setup.py.

[trivialfis]

This is a rewritten setup.py for a more user (and maintainer) friendly experience of managing Python packages. Currently marked as WIP as I haven't test it on other platforms.

• Remove setup_pip.py

• Remove soft links.

• Remove shell script.

• Remove makefile script.

• Define customized commands.

• Update documents.

[hcho3]

@trivialfis Yes, refactor of setup.py was long overdue. Can you make a list of goals you’d like to achieve with this refactor? Also brief summary of what’s going on inside setup.py would be nice.

[trivialfis]

@hcho3

For goals

• Make the following commands work:

    From internet:
    - pip install xgboost
    - pip install --no-binary :all: xgboost

    From source tree `xgboost/python-package`:
    - python setup.py install
    - python setup.py build
    - python setup.py sdist       && pip install <sdist-name>
    - python setup.py bdist_wheel && pip install <wheel-name>
    - pip install . -e
    - python setup.py develop   # same as above

• Make the script self-contain, instead of relying on some shell scripts with directory links.

• Make source distribution portable. I expect source distribution working on all 3 supported platforms.

For what's inside setup.py

There are multiple commands with setuptools:

• build_ext: Build C/C++ extension.
• build: Build the Python package (with C++).
• install: install the library, compile the source code by calling build_ext. This is the only place where we can accept user provided parameters.
• develop: Install the links to original code, for the ease of rapid testing.
• sdist: Create a source distribution.
• bdist: Create a binary distribution.
• bdist_wheel: Create a binary distribution with wheel format.

Each of these commands is represented as a Python module and a class with the same name. Hence we can subclass selected commands and customize it.

pip adds another layer on top of setuptools, with copying files to a temporary directory. Also internally it uses subprocess.call to invoke above commands. So quite difficult to debug ...

[trivialfis]

@hcho3 How do I log into Jenkins and find the libxgboost.so manually? I don't see any build step before building doc so may I assume the libxgboost.so is cached somewhere?

[trivialfis]

I tried to use setuptools with sphinx build, but some packages uses exec function internally, resulting in some weird errors. So for building the document, we will have to install all dependencies ourselves.

[trivialfis]

How do I log into Jenkins and find the libxgboost.so manually? I don't see any build step before building doc so may I assume the libxgboost.so is cached somewhere?

Never mind. Found the root cause.

[trivialfis]

@RAMitchell @hcho3 Ready for review. I will resolve the conflicts along with review comments.

[RAMitchell]

LGTM, please test your doc changes locally.

[RAMitchell]

Is cmake 3.12 strictly necessary?

[hcho3]

Since #5146, yes. It was done to support compiling XGBoost with OpenMP on Mac OSX.

[trivialfis]

@hcho3 changed it to 3.12 for Apple Clang in CMakeLists.txt

[trivialfis]

Since when it takes 1 and a half hour for appveyor to finish the tests .. ? ;-(