Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script for building manylinux wheels #189

Merged
merged 7 commits into from
Mar 9, 2023
Merged

Script for building manylinux wheels #189

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
23bc8db
Script for building manylinux wheels
nrontsis Feb 21, 2023
dd9c772
Rename build_many_linux_wheels.sh to build_manylinux_wheels.sh
nrontsis Feb 21, 2023
8a4ad3f
Update README.rst
nrontsis Feb 21, 2023
35116bf
Correct typos
nrontsis Feb 21, 2023
b75a005
Correct typo (mumps version)
nrontsis Feb 21, 2023
d6b4300
Add licenses for all libraries we bundle
nrontsis Feb 23, 2023
a579ca7
Rephrase
nrontsis Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,23 @@ Other `installation options`_ are present in the documentation.

.. _installation options: https://github.com/mechmotum/cyipopt/blob/master/docs/source/install.rst


Building `manylinux` wheels
===========================

manylinux wheels can be built for a tagged version (GIT_TAG below) of cyipopt via docker by running (while in the root of this repo)::

docker run -v $(pwd):/wheels --rm --platform=linux/amd64 quay.io/pypa/manylinux_2_28_x86_64 bash /wheels/build_manylinux_wheels.sh GIT_TAG

for linux/amd64 and::

docker run -v $(pwd):/wheels --rm --platform=linux/aarch64 quay.io/pypa/manylinux_2_28_aarch64 bash /wheels/build_manylinux_wheels.sh GIT_TAG

for linux/aarch64 platforms. Built wheels appear at the folder the command was executed from.

.. warning::
Docker supports emulating non-native platforms to e.g. produce ARM binaries from an AMD64 host. However this can be quite slow (~1h for our case).

License
=======

Expand Down
45 changes: 45 additions & 0 deletions build_manylinux_wheels.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/bin/bash
# Builds manylinux cyipopt wheels with Ipopt 3.14.11 based on MUMPS 5.5.1, and OpenBLAS 0.3.15
set -eu # Stop script if a line fails
TAG=${1}
echo "Building cyipopt with tag $TAG"

# Install efficient BLAS & LAPACK library
yum install -y openblas-devel-0.3.15-4.el8

pushd /tmp

# MUMPS (Linear solver used by Ipopt)
git clone https://github.com/coin-or-tools/ThirdParty-Mumps --depth=1 --branch releases/3.0.4
pushd ThirdParty-Mumps
sh get.Mumps
./configure --with-lapack="-L/usr/include/openblas -lopenblas"
make
make install
popd


# Ipopt (The solver itself)
git clone https://github.com/coin-or/Ipopt --depth=1 --branch releases/3.14.11
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you just yum install ipopt? Seems like either 3.13 or 3.14 are available: https://packages.fedoraproject.org/pkgs/coin-or-Ipopt/coin-or-Ipopt/

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you say "but I would rather rely on building ipopt and mumps from source, so that we have recent versions".

pushd Ipopt
./configure --with-lapack="-L/usr/include/openblas -lopenblas"
make
make install
popd

# build cyipopt for many python versions
git clone https://github.com/mechmotum/cyipopt --depth=1 --branch $TAG
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean by the release tarballs is to download https://files.pythonhosted.org/packages/4c/34/180b14dc521ae6a393c98d74d59afc79a7edc19a18721b69571843bfc495/cyipopt-1.2.0.tar.gz and extract it instead of checkout out a commit in the git repo.

pushd cyipopt
mkdir dist
for PYVERSION in "cp311-cp311" "cp310-cp310" "cp39-cp39" "cp38-cp38" "pp39-pypy39_pp73" "pp38-pypy38_pp73" ; do
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also support for python 3.7 in this manylinux docker image, but building on python 3.7 was failing with:

  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [42 lines of output]
      /tmp/cyipopt/setup.py:26: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.
        dist.Distribution().fetch_build_eggs(SETUP_REQUIRES)
      WARNING: The wheel package is not available.
      /opt/_internal/cpython-3.7.16/bin/python: No module named pip
      Traceback (most recent call last):
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 97, in _fetch_build_egg_no_warn
          subprocess.check_call(cmd)
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/subprocess.py", line 363, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/opt/_internal/cpython-3.7.16/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmppqmsycvd', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 26, in <module>
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 894, in fetch_build_eggs
          return _fetch_build_eggs(self, requires)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 42, in _fetch_build_eggs
          replace_conflicting=True,
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 816, in resolve
          req, best, replace_conflicting, env, installer, required_by, to_activate
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 852, in _resolve_dist
          req, ws, installer, replace_conflicting=replace_conflicting
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1123, in best_match
          return self.obtain(req, installer)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in obtain
          return installer(requirement)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 99, in _fetch_build_egg_no_warn
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/opt/_internal/cpython-3.7.16/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmppqmsycvd', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: /opt/_internal/cpython-3.7.16/bin/python -m pip install --upgrade pip

/opt/python/$PYVERSION/bin/pip wheel --no-deps --wheel-dir=./dist .
done
for wheel in dist/cyipopt*; do
auditwheel repair $wheel # Inject solver's shared libraries to wheel
done
for wheel in wheelhouse/*.whl; do
cp -rf $wheel /wheels/ # Copy repaired wheel to shared volume
done
popd

popd