Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script for building manylinux wheels #189

Merged
merged 7 commits into from
Mar 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,23 @@ Other `installation options`_ are present in the documentation.

.. _installation options: https://github.com/mechmotum/cyipopt/blob/master/docs/source/install.rst


Building `manylinux` wheels
===========================

manylinux wheels can be built for a tagged version (GIT_TAG below) of cyipopt via docker by running (while in the root of this repo)::

docker run -v $(pwd):/wheels --rm --platform=linux/amd64 quay.io/pypa/manylinux_2_28_x86_64 bash /wheels/build_manylinux_wheels.sh GIT_TAG

for linux/amd64 and::

docker run -v $(pwd):/wheels --rm --platform=linux/aarch64 quay.io/pypa/manylinux_2_28_aarch64 bash /wheels/build_manylinux_wheels.sh GIT_TAG

for linux/aarch64 platforms. Built wheels appear at the folder the command was executed from.

.. warning::
Docker supports emulating non-native platforms to e.g. produce ARM binaries from an AMD64 host. However this can be quite slow (~1h for our case).

License
=======

Expand Down
49 changes: 49 additions & 0 deletions build_manylinux_wheels.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash
# Builds manylinux cyipopt wheels with Ipopt 3.14.11 based on MUMPS 5.5.1, and OpenBLAS 0.3.15
set -eu # Stop script if a line fails
TAG=${1}
echo "Building cyipopt with tag $TAG"

# Install efficient BLAS & LAPACK library
yum install -y openblas-devel-0.3.15-4.el8

pushd /tmp

# MUMPS (Linear solver used by Ipopt)
git clone https://github.com/coin-or-tools/ThirdParty-Mumps --depth=1 --branch releases/3.0.4
pushd ThirdParty-Mumps
sh get.Mumps
./configure --with-lapack="-L/usr/include/openblas -lopenblas"
make
make install
popd


# Ipopt (The solver itself)
git clone https://github.com/coin-or/Ipopt --depth=1 --branch releases/3.14.11
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't you just yum install ipopt? Seems like either 3.13 or 3.14 are available: https://packages.fedoraproject.org/pkgs/coin-or-Ipopt/coin-or-Ipopt/

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you say "but I would rather rely on building ipopt and mumps from source, so that we have recent versions".

pushd Ipopt
./configure --with-lapack="-L/usr/include/openblas -lopenblas"
make
make install
popd

# build cyipopt for many python versions
git clone https://github.com/mechmotum/cyipopt --depth=1 --branch $TAG
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean by the release tarballs is to download https://files.pythonhosted.org/packages/4c/34/180b14dc521ae6a393c98d74d59afc79a7edc19a18721b69571843bfc495/cyipopt-1.2.0.tar.gz and extract it instead of checkout out a commit in the git repo.

pushd cyipopt
echo "------------------------------" >> LICENSE
echo "This binary distribution of cyipopt also bundles the following software" >> LICENSE
for bundled_license in licenses_manylinux_bundled_libraries/*.txt; do
cat $bundled_license >> LICENSE
done
for PYVERSION in "cp311-cp311" "cp310-cp310" "cp39-cp39" "cp38-cp38" "pp39-pypy39_pp73" "pp38-pypy38_pp73" ; do
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also support for python 3.7 in this manylinux docker image, but building on python 3.7 was failing with:

  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [42 lines of output]
      /tmp/cyipopt/setup.py:26: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.
        dist.Distribution().fetch_build_eggs(SETUP_REQUIRES)
      WARNING: The wheel package is not available.
      /opt/_internal/cpython-3.7.16/bin/python: No module named pip
      Traceback (most recent call last):
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 97, in _fetch_build_egg_no_warn
          subprocess.check_call(cmd)
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/subprocess.py", line 363, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/opt/_internal/cpython-3.7.16/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmppqmsycvd', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
      
      The above exception was the direct cause of the following exception:
      
      Traceback (most recent call last):
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/_internal/cpython-3.7.16/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 26, in <module>
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 894, in fetch_build_eggs
          return _fetch_build_eggs(self, requires)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 42, in _fetch_build_eggs
          replace_conflicting=True,
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 816, in resolve
          req, best, replace_conflicting, env, installer, required_by, to_activate
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 852, in _resolve_dist
          req, ws, installer, replace_conflicting=replace_conflicting
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1123, in best_match
          return self.obtain(req, installer)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1135, in obtain
          return installer(requirement)
        File "/tmp/pip-build-env-gdt5mhcy/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 99, in _fetch_build_egg_no_warn
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/opt/_internal/cpython-3.7.16/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmppqmsycvd', '--quiet', 'numpy>=1.15']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[notice] A new release of pip is available: 23.0 -> 23.0.1
[notice] To update, run: /opt/_internal/cpython-3.7.16/bin/python -m pip install --upgrade pip

/opt/python/$PYVERSION/bin/pip wheel --no-deps --wheel-dir=./dist .
done
for wheel in dist/cyipopt*; do
auditwheel repair $wheel # Inject solver's shared libraries to wheel
done
for wheel in wheelhouse/*.whl; do
cp -rf $wheel /wheels/ # Copy repaired wheel to shared volume
done
popd

popd
1 change: 1 addition & 0 deletions licenses_manylinux_bundled_libraries/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This folder contains licenses for all the libraries that are bundled in the `manylinux` wheels of `cyipopt`. These linceses are appended to `cyipopt`'s `LICENSE` file when building these wheels in `build_manylinux_wheel.sh`.
81 changes: 81 additions & 0 deletions licenses_manylinux_bundled_libraries/gfortran.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#############
start of gfortran license
applicable for bundled libraries: .libs/libgfortran*.so
license text obtained from https://github.com/gcc-mirror/gcc/blob/748086b7b2201112aff2dea9d037af1fc92567ff/COPYING.RUNTIME
#############

GCC RUNTIME LIBRARY EXCEPTION

Version 3.1, 31 March 2009

Copyright (C) 2009 Free Software Foundation, Inc. <http://fsf.org/>

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

This GCC Runtime Library Exception ("Exception") is an additional
permission under section 7 of the GNU General Public License, version
3 ("GPLv3"). It applies to a given file (the "Runtime Library") that
bears a notice placed by the copyright holder of the file stating that
the file is governed by GPLv3 along with this Exception.

When you use GCC to compile a program, GCC may combine portions of
certain GCC header files and runtime libraries with the compiled
program. The purpose of this Exception is to allow compilation of
non-GPL (including proprietary) programs to use, in this way, the
header files and runtime libraries covered by this Exception.

0. Definitions.

A file is an "Independent Module" if it either requires the Runtime
Library for execution after a Compilation Process, or makes use of an
interface provided by the Runtime Library, but is not otherwise based
on the Runtime Library.

"GCC" means a version of the GNU Compiler Collection, with or without
modifications, governed by version 3 (or a specified later version) of
the GNU General Public License (GPL) with the option of using any
subsequent versions published by the FSF.

"GPL-compatible Software" is software whose conditions of propagation,
modification and use would permit combination with GCC in accord with
the license of GCC.

"Target Code" refers to output from any compiler for a real or virtual
target processor architecture, in executable form or suitable for
input to an assembler, loader, linker and/or execution
phase. Notwithstanding that, Target Code does not include data in any
format that is used as a compiler intermediate representation, or used
for producing a compiler intermediate representation.

The "Compilation Process" transforms code entirely represented in
non-intermediate languages designed for human-written code, and/or in
Java Virtual Machine byte code, into Target Code. Thus, for example,
use of source code generators and preprocessors need not be considered
part of the Compilation Process, since the Compilation Process can be
understood as starting with the output of the generators or
preprocessors.

A Compilation Process is "Eligible" if it is done using GCC, alone or
with other GPL-compatible software, or if it is done without using any
work based on GCC. For example, using non-GPL-compatible Software to
optimize any GCC intermediate representations would not qualify as an
Eligible Compilation Process.

1. Grant of Additional Permission.

You have permission to propagate a work of Target Code formed by
combining the Runtime Library with Independent Modules, even if such
propagation would otherwise violate the terms of GPLv3, provided that
all Target Code was generated by Eligible Compilation Processes. You
may then convey such a combination under terms of your choice,
consistent with the licensing of the Independent Modules.

2. No Weakening of GCC Copyleft.

The availability of this Exception does not imply any general
presumption that third-party software is unaffected by the copyleft
requirements of the license of GCC.

#############
end of gfortran license
Loading