python 3.10 support

[leifwalsh]

llvmlite currently explicitly doesn't support python 3.10. What needs to be done to lift this restriction?

I'm mostly creating this issue so I have something to watch so I can know when it's ready for 3.10, but if there's something you all need help with I'm happy to try.

[esc]

@leifwalsh I tried to run llvmlite in a Python 3.10 (beta?) docker container from here: https://hub.docker.com/_/python?tab=description -- IIRC the only thing that needs to be done right now is to remove the restriction on the Python version in setup.py and it will compile and the test suite will run (or there is a minimal amount of trivial patching, can't quite remember).

The guard is here: https://github.com/numba/llvmlite/blob/master/setup.py#L36

If you are inclined, it would be great for you to confirm my statements above independently. If we can be sure, that no changes are needed in llvmlite for Python 3.10 that would be awesome! Let me know if you need any more pointers or if you hit any snags!

[gmarkall]

I think we'd also want to modify the CI setup to include 3.10?

Would we also drop support for 3.6, given that Numba no longer supports it?

[esc]

I think we'd also want to modify the CI setup to include 3.10?

Yes, but I think we will need to wait until a Python 3.10 becomes available as a conda package because all our CI uses that.

Would we also drop support for 3.6, given that Numba no longer supports it?

Yes, that was an oversight, it should have been removed for v0.37.0 RC1 -- but was missed. It doesn't hurt to include them for now, but yes, 3.6 has to go.

[bcbnz]

I've just tested 4ba6f53 with LLVM 11.1.0 and a freshly built copy of Python 3.10.0b3 (on Arch Linux) and did not have to change anything other than the version guard.

I guess the conda support is conda-forge/python-feedstock#463

[esc]

I've just tested 4ba6f53 with LLVM 11.1.0 and a freshly built copy of Python 3.10.0b3 (on Arch Linux) and did not have to change anything other than the version guard.

Great, thanks for confirming!

I guess the conda support is conda-forge/python-feedstock#463

The conda-forge support, yes.

[leifwalsh]

3.10 final is out so I think this can move forward whenever your CI is ready to run it, and you're ready to cut a fresh release

[esc]

@leifwalsh thank you for the ping. We depend on a Python 3.10 conda package for our internal build system, so it will be some more time until we can conclude this.

[seibert]

One thing to note is that while llvmlite would work with Python 3.10 now, Numba does not yet due to bytecode changes. You'll want to track this PR to know when Numba is ready for Python 3.10 (at which point both Numba and llvmlite can be released together): numba/numba#7381

[leifwalsh]

Ooh, thanks, I do want to track that.

[2sn]

for me, llvmlite does not compile from git with pip. If it is just a version guard that needs updating, could release an intermediate version (0.37.1) that fixes this?

~>pip3 install git+https://github.com/numba/llvmlite.git
Collecting git+https://github.com/numba/llvmlite.git
  Cloning https://github.com/numba/llvmlite.git to /tmp/pip-req-build-i_30n3s_
  Running command git clone -q https://github.com/numba/llvmlite.git /tmp/pip-req-build-i_30n3s_
  Resolved https://github.com/numba/llvmlite.git to commit 257f77f7417e5fbeea3aa3f7eaba176aec482fa9
    ERROR: Command errored out with exit status 1:
     command: /home/alex/Python/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-i_30n3s_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-i_30n3s_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-kffafi7k
         cwd: /tmp/pip-req-build-i_30n3s_/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-i_30n3s_/setup.py", line 55, in <module>
        _guard_py_ver()
      File "/tmp/pip-req-build-i_30n3s_/setup.py", line 52, in _guard_py_ver
        raise RuntimeError(msg.format(cur_py, min_py, max_py))
    RuntimeError: Cannot install on Python version 3.10.0; only versions >=3.7,<3.10 are supported.
    ----------------------------------------
WARNING: Discarding git+https://github.com/numba/llvmlite.git. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[esc]

for me, llvmlite does not compile from git with pip. If it is just a version guard that needs updating, could release an intermediate version (0.37.1) that fixes this?

~>pip3 install git+https://github.com/numba/llvmlite.git
Collecting git+https://github.com/numba/llvmlite.git
  Cloning https://github.com/numba/llvmlite.git to /tmp/pip-req-build-i_30n3s_
  Running command git clone -q https://github.com/numba/llvmlite.git /tmp/pip-req-build-i_30n3s_
  Resolved https://github.com/numba/llvmlite.git to commit 257f77f7417e5fbeea3aa3f7eaba176aec482fa9
    ERROR: Command errored out with exit status 1:
     command: /home/alex/Python/bin/python3.10 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-i_30n3s_/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-i_30n3s_/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-kffafi7k
         cwd: /tmp/pip-req-build-i_30n3s_/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-i_30n3s_/setup.py", line 55, in <module>
        _guard_py_ver()
      File "/tmp/pip-req-build-i_30n3s_/setup.py", line 52, in _guard_py_ver
        raise RuntimeError(msg.format(cur_py, min_py, max_py))
    RuntimeError: Cannot install on Python version 3.10.0; only versions >=3.7,<3.10 are supported.
    ----------------------------------------
WARNING: Discarding git+https://github.com/numba/llvmlite.git. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Can you try: pip install git+https://github.com/numba/llvmlite.git@refs/pull/769/merge that should allow you to circumvent the version guard.

[2sn]

Thanks.
On Fedora 34, after installing llvm development

dnf install -y llvm11-devel

the install works

LLVM_CONFIG=/usr/bin/llvm-config-11-64 pip install git+https://github.com/numba/llvmlite.git@refs/pull/769/merge
Collecting git+https://github.com/numba/llvmlite.git@refs/pull/769/merge
  Cloning https://github.com/numba/llvmlite.git (to revision refs/pull/769/merge) to /tmp/pip-req-build-r19nzj88
  Running command git clone -q https://github.com/numba/llvmlite.git /tmp/pip-req-build-r19nzj88
  WARNING: Did not find branch or tag 'refs/pull/769/merge', assuming revision or ref.
  Running command git fetch -q https://github.com/numba/llvmlite.git refs/pull/769/merge
  Running command git checkout -q 376bea85de272c5f45f52bf3ecffbfb8870ac6e4
  Resolved https://github.com/numba/llvmlite.git to commit 376bea85de272c5f45f52bf3ecffbfb8870ac6e4
Building wheels for collected packages: llvmlite
  Building wheel for llvmlite (setup.py) ... done
  Created wheel for llvmlite: filename=llvmlite-0.38.0.dev0+28.g376bea8-cp310-cp310-linux_x86_64.whl size=2355671 sha256=ff056cae98b843095985923344bf5bafffedf35a86058efaa75b062a1a0bbf00
  Stored in directory: /tmp/pip-ephem-wheel-cache-okch4loc/wheels/ad/44/cb/b4dda368d857c5f5baccf51ce6e16aed4d1f23ab707654d11c
Successfully built llvmlite
Installing collected packages: llvmlite
Successfully installed llvmlite-0.38.0.dev0+28.g376bea8

Thanks for your fast help!

Now need to wait for numba port ...

[sanjayankur31]

On Fedora 34, after installing llvm development

Just FYI: we included llvmlite in Fedora recently, so you can just use dnf install python3-llvmlite on F34+:

https://src.fedoraproject.org/rpms/python-llvmlite

(For python3.10, we only needed to remove the version guard as noted above, and it built fine will all tests passing.)

[esc]

Thanks. On Fedora 34, after installing llvm development

dnf install -y llvm11-devel

the install works

LLVM_CONFIG=/usr/bin/llvm-config-11-64 pip install git+https://github.com/numba/llvmlite.git@refs/pull/769/merge
Collecting git+https://github.com/numba/llvmlite.git@refs/pull/769/merge
  Cloning https://github.com/numba/llvmlite.git (to revision refs/pull/769/merge) to /tmp/pip-req-build-r19nzj88
  Running command git clone -q https://github.com/numba/llvmlite.git /tmp/pip-req-build-r19nzj88
  WARNING: Did not find branch or tag 'refs/pull/769/merge', assuming revision or ref.
  Running command git fetch -q https://github.com/numba/llvmlite.git refs/pull/769/merge
  Running command git checkout -q 376bea85de272c5f45f52bf3ecffbfb8870ac6e4
  Resolved https://github.com/numba/llvmlite.git to commit 376bea85de272c5f45f52bf3ecffbfb8870ac6e4
Building wheels for collected packages: llvmlite
  Building wheel for llvmlite (setup.py) ... done
  Created wheel for llvmlite: filename=llvmlite-0.38.0.dev0+28.g376bea8-cp310-cp310-linux_x86_64.whl size=2355671 sha256=ff056cae98b843095985923344bf5bafffedf35a86058efaa75b062a1a0bbf00
  Stored in directory: /tmp/pip-ephem-wheel-cache-okch4loc/wheels/ad/44/cb/b4dda368d857c5f5baccf51ce6e16aed4d1f23ab707654d11c
Successfully built llvmlite
Installing collected packages: llvmlite
Successfully installed llvmlite-0.38.0.dev0+28.g376bea8

Thanks for your fast help!

Now need to wait for numba port ...

Thank you for trying and confirming this. But please, be aware of: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#what-to-be-aware-of-when-using-a-system-provided-llvm-package

[esc]

On Fedora 34, after installing llvm development

Just FYI: we included llvmlite in Fedora recently, so you can just use dnf install python3-llvmlite on F34+:

https://src.fedoraproject.org/rpms/python-llvmlite

(For python3.10, we only needed to remove the version guard as noted above, and it built fine will all tests pas

On Fedora 34, after installing llvm development

Just FYI: we included llvmlite in Fedora recently, so you can just use dnf install python3-llvmlite on F34+:

https://src.fedoraproject.org/rpms/python-llvmlite

(For python3.10, we only needed to remove the version guard as noted above, and it built fine will all tests passing.)

@sanjayankur31 thank you for reporting back about this. You may need an additional patch, there is a PR with all the relevant patches at: #769

[sanjayankur31]

@esc thanks, I'll pull the patches in when I can find the time

[2sn]

@esc thanks. Since I use my own Python compiled-from-scratch the dnf version might not work for me.

[bmerry]

Should this be closed now that 0.38 supports Python 3.10?

[esc]

Should this be closed now that 0.38 supports Python 3.10?

Yes, thank you for the ping, 0.38 included support for 3.10 so I will close this now.