ARROW-10224: [Python] Add support for Python 3.9 except macOS wheel and Windows wheel #8386
Conversation
terencehonles
force-pushed
the
arrow-10224
branch
from
0747e3d to
da52256
Compare
| @@ -16,7 +16,7 @@ class ApacheArrow < Formula | |||
| depends_on "numpy" | |||
| depends_on "openssl@1.1" | |||
| depends_on "protobuf" | |||
| depends_on "python@3.8" | |||
| depends_on "python@3.9" | |||
There was a problem hiding this comment.
I'm assuming homebrew should be on the latest Python? It does look like it's available now https://formulae.brew.sh/formula/python@3.9
| @@ -110,6 +110,6 @@ set ARROW_TEST_DATA=%ARROW_SRC%\testing\data | |||
| %PYTHON_INTERPRETER% -c "import pyarrow.dataset" || exit /B | |||
|
|
|||
| @rem run the python tests, but disable the cython because there is a linking | |||
| @rem issue on python 3.8 | |||
| @rem issue on python 3.9 | |||
| set PYARROW_TEST_CYTHON=OFF | |||
There was a problem hiding this comment.
I don't know if this still applies, but I figured it probably would
There was a problem hiding this comment.
Sounds good to me, thanks for kicking things off. I had also create voltrondata-labs/ursabot#205 to adjust buildbot since I realized 3.9 wasn't being built just by me creating this PR.
|
It looks like I "missed" the github workflows, but that might need to be done after the wheels are built and published as I haven't looked at what exactly are doing, and I kinda doubt they are building from source. I can open that up as a separate PR if that is helpful. |
|
@github-actions crossbow submit -g conda -g wheel |
|
Revision: da52256 Submitted crossbow builds: ursa-labs/crossbow @ actions-616
|
terencehonles
force-pushed
the
arrow-10224
branch
from
da52256 to
59496cc
Compare
|
It looks like cmake was already part of the quay.io/pypa/manylinux2014_x86_64:2020-10-06-b2ca7a1 and |
|
@github-actions crossbow submit -g conda -g wheel |
|
Revision: 59496cc Submitted crossbow builds: ursa-labs/crossbow @ actions-617
|
|
Revision: 59496cc Submitted crossbow builds: ursa-labs/crossbow @ actions-618
|
|
(We need to wait a while until our comments are processed.) |
| @@ -31,7 +31,7 @@ addons: | |||
| - git | |||
| - openssl@1.1 | |||
| - protobuf | |||
| - python@3.8 | |||
| - python@3.9 | |||
| - thrift | |||
| - wget | |||
| update: false | |||
There was a problem hiding this comment.
I think it's possible Travis doesn't have Python 3.9 in it's homebrew cache, but I would expect a different error than it succeeds and then /usr/local/opt/python@3.8/libexec/bin/pip is just not found...
Nevermind! Apparently it's just a "warning" 😒 :
Warning: 'python@3.9' formula is unreadable: No available formula with the name "python@3.9"
https://travis-ci.org/github/ursa-labs/crossbow/builds/733873746#L145
terencehonles
force-pushed
the
arrow-10224
branch
from
59496cc to
9ee37df
Compare
|
I reverted some of the OS X changes. I'm not sure I actually needed to change Python 3.8 there since it looks like that travis file is templated out to all OS X wheels and it is still using Python 3.8 in all of them. That's either a bug or that there's something else that's using the |
|
Note that Numpy doesn't provide pre-compiled wheels for 3.9 currently. |
|
Also, conda-forge doesn't seem to be ready either. |
Yeah, that's fine. I'm not really in a rush, I just wanted to help to get things in place that we could move forward as soon as all of the dependencies are met (and to identify where things are and what is still missing) It does build, and I was able to install everything in the project I am working on and run the tests (and discover other things that need to be updated), so that's really mainly what I was looking for. I figured I'd do it in public on a PR and use the built wheels instead of just building/testing locally and not sharing my work 🙂 |
Should be ready now. |
|
@github-actions crossbow submit -g conda |
|
Revision: 9ee37df Submitted crossbow builds: ursa-labs/crossbow @ actions-651
|
terencehonles
force-pushed
the
arrow-10224
branch
from
9ee37df to
32582a2
Compare
This comment has been minimized.
This comment has been minimized.
terencehonles
force-pushed
the
arrow-10224
branch
from
32582a2 to
5f695d4
Compare
|
I pinned semver to 2.10.2 since it looks like there is a breaking change for |
This comment has been minimized.
This comment has been minimized.
|
Revision: 173d45e Submitted crossbow builds: ursa-labs/crossbow @ actions-783
|
|
@kou When will the binaries be available in pypi? Thanks! |
|
We'll release them in 2021-01. |
|
@kou You're likely well aware, but if looking for e.g. $ curl -s 'https://api.github.com/repos/ursa-labs/crossbow/releases' | jq -r '.[]["assets"] | .[]["browser_download_url"]' | grep manylinux2014-cp39
https://github.com/ursa-labs/crossbow/releases/download/nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev412-cp39-cp39-manylinux2014_x86_64.whl
https://github.com/ursa-labs/crossbow/releases/download/nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev412.tar.gzLooks like the api has a limit of 100 results per call, which doesn't go back far enough. Instead of dealing with pagination, it seems like $ gh release list --repo=ursa-labs/crossbow --limit=1000 | grep manylinux2014-c
p39
nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39 2020-12-18T07:26:20Z
nightly-2020-12-17-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-17-0-azure-wheel-manylinux2014-cp39 2020-12-17T07:33:01Z
actions-807-azure-wheel-manylinux2014-cp39 actions-807-azure-wheel-manylinux2014-cp39 2020-12-17T01:23:13Z
actions-804-azure-wheel-manylinux2014-cp39 actions-804-azure-wheel-manylinux2014-cp39 2020-12-16T23:00:37Z
actions-799-azure-wheel-manylinux2014-cp39 actions-799-azure-wheel-manylinux2014-cp39 2020-12-16T08:54:37Z
nightly-2020-12-16-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-16-0-azure-wheel-manylinux2014-cp39 2020-12-16T06:58:23Z
actions-794-azure-wheel-manylinux2014-cp39 actions-794-azure-wheel-manylinux2014-cp39 2020-12-15T21:35:42Z
nightly-2020-12-15-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-15-0-azure-wheel-manylinux2014-cp39 2020-12-15T06:22:33Z
nightly-2020-12-14-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-14-0-azure-wheel-manylinux2014-cp39 2020-12-14T07:19:20Z
actions-783-azure-wheel-manylinux2014-cp39 actions-783-azure-wheel-manylinux2014-cp39 2020-12-12T22:57:41Z
actions-772-azure-wheel-manylinux2014-cp39 actions-772-azure-wheel-manylinux2014-cp39 2020-12-11T07:19:59Z
actions-767-azure-wheel-manylinux2014-cp39 actions-767-azure-wheel-manylinux2014-cp39 2020-12-10T20:02:41Z
actions-759-azure-wheel-manylinux2014-cp39 actions-759-azure-wheel-manylinux2014-cp39 2020-12-10T04:27:52Z
actions-758-azure-wheel-manylinux2014-cp39 actions-758-azure-wheel-manylinux2014-cp39 2020-12-09T23:30:31ZFrom the name of the release you can construct the URL: And from there you can find: From here you can install directly with pip: $ python -m pip install https://github.com/ursa-labs/crossbow/releases/download/actions-783-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev363-cp39-cp39-manylinux2014_x86_64.whl |
|
Please be aware that's like running a nightly release, but if you are fine with that you can install a crossbow wheel. However, you probably want to just follow the instructions here https://arrow.apache.org/docs/python/install.html#installing-nightly-packages TL;DR:
|
|
Thanks for the suggestion!
Unfortunately my distro has already moved to 3.9 (including all my cuda and DL libraries etc) so I need a 39 compatible wheel or I can't use it. Nightly it is for now! This way I can make sure the release I'm installing is at least passing CI.
|
|
Using the command mentioned by @terencehonles doesn't work for me. This line still looking for pyarrow from the default indexes. |
|
Adding |
|
@kou Thanks for your help. I think it may because the link |
|
We're still working on it at #8915. |
@liyucheng09 yes that's correct, if you're impatient and OK with using a crossbow build as @n8henrie was attempting you can use https://github.com/ursa-labs/crossbow/branches/all?query=actions-686 which built both mavericks and high sierra as you can see in #8386 (comment) |
|
@terencehonles Thanks a lot. I installed pyarrorw sucessfully. I was just curious that when I type I thought that figure it out may help me to understand deeper about python package installing process. |
I'm assuming you mean trying to install the most recently released pyarrow before you installed via crossbow? That's because pyarrow has a source distribution and you're trying to build from source. Pip will not have installed numpy at install time and is working to make isolated installs the default, if you have numpy already installed the version you have may be an issue, but I think it's more likely it's too old not too new. You can try using the flag pyarrow is probably not the easiest to learn about Python's build process on, but you should read their documentation if you would like to actually be able to build it from source. |
… wheels Picking back up from #8386 in case I can handle this or someone else wants to work off of it. Closes #8915 from terencehonles/arrow-10904 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Missed in apache#8386.
…nd Windows wheel Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments. H/T: @kou, the documentation and apache#5685 for helping me get this off the ground. Closes apache#8386 from terencehonles/arrow-10224 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
… wheels Picking back up from apache#8386 in case I can handle this or someone else wants to work off of it. Closes apache#8915 from terencehonles/arrow-10904 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Missed in apache#8386. Closes apache#10286 from adamchainz/patch-1 Authored-by: Adam Johnson <me@adamj.eu> Signed-off-by: David Li <li.davidm96@gmail.com>
Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments.
H/T: @kou, the documentation and #5685 for helping me get this off the ground.