Build wheels automatically in CI environment #168
Conversation
|
The build process for linux (and I assume osx and windows once I start debugging that) is blocked by #172 because cibuildwheel uses venvs to created isolated python environments. If we can't build the sdist into a wheel inside a venv than it's going to fail every time. |
mtreinish
force-pushed
the
automate-wheel-build
branch
from
035bb87 to
a365a25
Compare
mtreinish
force-pushed
the
automate-wheel-build
branch
from
a365a25 to
714b640
Compare
|
Well 1/3 of the way there. The osx wheels built successfully: https://travis-ci.com/Qiskit/qiskit-aer/jobs/229330750 |
|
@mtreinish are you still working on this or should I close this PR? |
|
@chriseclectic I am still working on it, higher priority work has come up since my last update here. I'm planning to circle back to it eventually. |
|
So the current state of this is that windows builds are failing on It looks like to move forward on here we'll have to change the linux builds to not link against libpython and for windows figure out why |
mtreinish
force-pushed
the
automate-wheel-build
branch
from
d426f3d to
d6018e0
Compare
|
Ok, now that we've merged pybin11 support and that we took advantage of it to change the way we were linking the extensions, we shall have no problems with libpython anymore. I'll resume this PR asap and fix the remaining issues. |
mtreinish
force-pushed
the
automate-wheel-build
branch
2 times, most recently
from
589a883 to
2bcc5c2
Compare
|
Finally, the linux builds are working now. OSX is working as well except for python 3.8: https://travis-ci.com/Qiskit/qiskit-aer/jobs/273238792#L1023 which looks like it's related to what was fixed in: Qiskit/qiskit@af390a8 Windows also still needs work it's not finding openblas right now: |
mtreinish
force-pushed
the
automate-wheel-build
branch
3 times, most recently
from
bee2c36 to
96b84ce
Compare
Right now when it's time to push a relase of qiskit-aer we have to manually build wheels for all our supported environments because we included compiled code in the project. This involves setting up a proper environment with portable librariers and compilers in Linux, Windows, and Mac OSX with all 3 versions of python we support. Then manually building the 9 wheels necessary to publish on pypi. There is no reason this manual step needs to exist and we can just leverage CI to do our wheel building when it's time to do a release. This commit adds initial support for building our wheels in travis (for linux and osx) and appveyor (for windows). It leverages the cibuildwheel project to automate the wheel building. This will ensure we setup the proper environment for all 3 python versions on all 3 environments (for example pulling in the manylinux docker container on linux).
mtreinish
force-pushed
the
automate-wheel-build
branch
from
1fa5bb8 to
f8279e4
Compare
|
After merging the openpulse branch onto master things are failing on linux and windows now. @atilag it looks like the linux failure is because things are attempting to link against libpython again: https://travis-ci.com/Qiskit/qiskit-aer/jobs/280091283#L615 we need to fix this. |
|
Yep, Cython code from Open Pulse sim branch was still being linked against libpython, but I have already fix it on |
Now that we've finally debugged the wheel build process on all python versions across all 3 supported OSes it's time to prepare this job for actually automating wheel publishing at release. This commit does 2 things first it makes all the new jobs conditional so they only run on new tags (ie a release). The second thing it does is uncomment the twine upload section. Now after each job finishes it's wheel build it will upload those files to pypi automatically.
mtreinish
changed the title
|
Ok, everything here is finally working now that #550 merged. All the wheel builds succeeded on ea96598 so this is finally ready to go. I've removed the WIP from the PR title and made all the jobs only run on tags/releases. After this merges I will go into the travis and azure UI and set the password for twine as a secret so that for releases CI will automatically build and upload wheels. If anyone is curious azure enables us to upload artifacts, the windows wheels that were built can be found here: https://dev.azure.com/qiskit-ci/qiskit-aer/_build/results?buildId=7712&view=artifacts&type=publishedArtifacts (it's the zip file labeled |
In the recently merged Qiskit#168 a small oversight was made in the last revision to that PR. The name of the stage was updated from wheels to deploy, but the stage listed for the individual jobs was never updated. This commit corrects the oversight and uses a consistent stage name for the new jobs.
In the recently merged #168 a small oversight was made in the last revision to that PR. The name of the stage was updated from wheels to deploy, but the stage listed for the individual jobs was never updated. This commit corrects the oversight and uses a consistent stage name for the new jobs.
Summary
Right now when it's time to push a relase of qiskit-aer we have to
manually build wheels for all our supported environments because we
included compiled code in the project. This involves setting up a proper
environment with portable librariers and compilers in Linux, Windows,
and Mac OSX with all 4 versions of python we support. Then manually
building the 12 wheels necessary to publish on pypi. There is no reason
this manual step needs to exist and we can just leverage CI to do our
wheel building when it's time to do a release.
This commit adds initial support for building our wheels in travis
(for linux and osx) and azure pipelines (for windows). It leverages the
cibuildwheel project to automate the wheel building on linux and osx. This
will ensure we setup the proper environment for all 4 python versions on
both environments (for example pulling in the manylinux docker container on
linux). For windows it leverages conda to setup a fresh environment for each
python version instead of using cibuildwheel because of specific configuration
needed there.
Details and comments
TODO: