Python 3.10 #769

esc · 2021-09-02T12:09:05Z

As title

esc · 2021-10-07T14:12:09Z

/azp run

azure-pipelines · 2021-10-07T14:13:11Z

Azure Pipelines successfully started running 1 pipeline(s).

esc · 2021-10-07T14:24:13Z

@sklam and I agreed that a 0.37.1 may be in the cards for next week.

esc · 2021-10-11T18:12:44Z

/azp run

azure-pipelines · 2021-10-11T18:12:52Z

Azure Pipelines successfully started running 1 pipeline(s).

sklam · 2021-10-11T22:37:04Z

llvmlite/binding/ffi.py

@@ -183,7 +183,8 @@ def __call__(self, *args, **kwargs):
 for _lib_path in _lib_paths:
    try:
        lib = ctypes.CDLL(_lib_path)
-    except OSError:
+    except OSError as e:
+        print(e)


Is this leftover from debugging?

yes probably, but we may to leave it it. This solves the issue, where errors of the form libllvmlite.so could not be loaded masquerade the real error, which makes these kinds of errors hard to debug. If the lib can not be loaded, we can't continue anyway, so maybe a lone print here is the right thing to do?

I'd prefer buffering the messages and report it later when it does raise OSError at

llvmlite/llvmlite/binding/ffi.py

Line 192 in 334c000

raise OSError("Could not load shared object file: {}".format(_lib_name))

Right now, if the last path is successful, the earlier error are still being printed.

ok, that makes sense

Fixed in: 8cc3c39

Here is a sample failure:

Python 3.9.5 (default, May 18 2021, 12:31:01) [Clang 10.0.0 ] :: Anaconda, Inc. on darwin Type "help", "copyright", "credits" or "license" for more information. i>>> import llvmlite >>> import llvmlite.binding Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/vhaenel/git/llvmlite/llvmlite/binding/__init__.py", line 4, in <module> from .dylib import * File "/Users/vhaenel/git/llvmlite/llvmlite/binding/dylib.py", line 3, in <module> from llvmlite.binding import ffi File "/Users/vhaenel/git/llvmlite/llvmlite/binding/ffi.py", line 195, in <module> raise OSError(msg) OSError: Could not load shared object file: libllvmlite.dylib Errors were: [OSError('dlopen(/Users/vhaenel/git/llvmlite/llvmlite/binding/libllvmlite.dylib, 6): image not found'), OSError('dlopen(libllvmlite.dylib, 6): image not found'), OSError('dlopen(./libllvmlite.dylib, 6): image not found'), OSError('dlopen(/Users/vhaenel/git/llvmlite/llvmlite/binding/libllvmlite.dylib, 6): image not found')]

This will avoid printing errors unnecessarily.

sklam

I've manually tested the failed to find libllvmlite error and it looks good.

sklam · 2021-10-13T16:51:52Z

I've manually tested this on OSX on py3.10. ✅

As title

esc · 2021-10-13T17:50:41Z

The current CI failures:

pygments -> setuptools
python=3.10 -> pip -> setuptools
sphinx -> setuptoolsThe following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.27=0
  - python=3.10 -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Are because the packages related to documentation are not available yet via the defaults channel on anaconda.org.

esc · 2021-10-14T07:05:51Z

So, what should we do. I see three options:

a) Don't try to build the docs for 3.10
b) pip install the doc dependencies for 3.10 into the existing conda environment
c) Wait until the documentation packages become available for 3.10

We also still need to make changes to the anaconda internal build farm in order to build the conad-packages for 3.10 and the wheels too.

esc · 2021-10-20T20:48:31Z

Build Farm ID: llvmlite_115.

esc · 2021-10-25T12:34:00Z

/azp run

azure-pipelines · 2021-10-25T12:34:13Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2021-11-15T13:22:13Z

Azure Pipelines successfully started running 1 pipeline(s).

esc · 2021-11-16T15:48:09Z

#789 is now provided to circumvent the CI issues present on this PR.

esc · 2021-11-18T10:27:57Z

/azp run

azure-pipelines · 2021-11-18T10:28:08Z

Azure Pipelines successfully started running 1 pipeline(s).

esc · 2021-11-18T11:03:10Z

Oh wow, it went green for the first time, yay!

esc · 2021-11-18T11:46:18Z

Now running on the Anconda internal build farm as: llvmlite_119.

esc · 2021-11-18T21:37:04Z

This passed on the anaconda internal build farm:

jakirkham · 2021-11-18T23:29:06Z

Matti tracked the issue Marcel noted above to the finalization line here (622):

llvmlite/llvmlite/tests/test_binding.py

Lines 621 to 625 in d77dc1b

    
           def get_sum(self, ee, func_name="sum"): 
        
               ee.finalize_object() 
        
               cfptr = ee.get_function_address(func_name) 
        
               self.assertTrue(cfptr) 
        
               return CFUNCTYPE(c_int, c_int, c_int)(cfptr)

More details in comment ( conda-forge/llvmlite-feedstock#58 (comment) )

esc · 2021-11-19T05:15:28Z

Matti tracked the issue Marcel noted above to the finalization line here (622):

llvmlite/llvmlite/tests/test_binding.py

Lines 621 to 625 in d77dc1b

def get_sum(self, ee, func_name="sum"):

ee.finalize_object()

cfptr = ee.get_function_address(func_name)

self.assertTrue(cfptr)

return CFUNCTYPE(c_int, c_int, c_int)(cfptr)

More details in comment ( conda-forge/llvmlite-feedstock#58 (comment) )

@jakirkham thank you for posting this. It will probably make sense to debug this during the RC period when we have also built our binary artifacts.

mbargull · 2021-11-19T07:38:30Z

See conda-forge/llvmlite-feedstock#58 (comment) : Building works fine in conda-forge's setup but testing with conda-forge::python=3.10 fails.

mbargull · 2021-11-19T07:41:56Z

See conda-forge/llvmlite-feedstock#58 (comment) : Building works fine in conda-forge's setup but testing with conda-forge::python=3.10 fails.

@esc , this could/would also mean that you can probably test it in an environment created with your build artifact and conda-forge::python=3.10.

mbargull · 2021-11-19T13:38:36Z

@esc, as an update: the segfault on osx-64 for conda-forge stems from the CPython 3.10 plus libffi 3.4 combination.
@jakirkham found and upstream issue for this: https://bugs.python.org/issue44556
I opened conda-forge/python-feedstock#523 for us to handle this.

TL;DR looks like llvmlite=0.37.0 works fine with python=3.10 overall and the one known encountered issue doesn't stem from llvmlite's code base :).

esc · 2021-11-19T13:40:13Z

@esc, as an update: the segfault on osx-64 for conda-forge stems from the CPython 3.10 plus libffi 3.4 combination. @jakirkham found and upstream issue for this: https://bugs.python.org/issue44556 I opened conda-forge/python-feedstock#523 for us to handle this.

TL;DR looks like llvmlite=0.37.0 works fine with python=3.10 overall and the one known encountered issue doesn't stem from llvmlite's code base :).

Great finds! Thank you for keeping us in the loop. It is much appreciated!

esc · 2021-11-20T12:20:09Z

@mbargull would you recommend we pin libffi in our recpie?

jakirkham · 2021-11-20T23:30:22Z

No the issue was in how CPython was doing the build itself. We fixed that in conda-forge ( conda-forge/python-feedstock#525 ). Apparently the Python devs also fixed this, but it didn't make it into 3.10.0 ( https://bugs.python.org/issue45350 ).

mbargull · 2021-11-22T12:19:50Z

Yep, what John wrote! All good from our side now. I'll open an issue on Anaconda's recipe repo (\edit: AnacondaRecipes/python-feedstock#50 ) too so we don't run into the same issue for the defaults channel if defaults::python=3.10.0 gets rebuild with libffi=3.4.

esc · 2021-11-22T20:40:14Z

@sklam @stuartarchibald @seibert I think this one is ready, needs a final review, I believe.

esc · 2021-11-22T20:46:55Z

Yep, what John wrote! All good from our side now. I'll open an issue on Anaconda's recipe repo (\edit: AnacondaRecipes/python-feedstock#50 ) too so we don't run into the same issue for the defaults channel if defaults::python=3.10.0 gets rebuild with libffi=3.4.

@mbargull Thank you, much appreciated!! 🙏

gmarkall

It looks like all the issues brought up in the discussion are now resolved, so this looks good to me.

esc · 2021-11-23T10:14:23Z

Thank you all for the reviews!

esc added 3 commits August 31, 2021 15:24

bump supported Pyton version

34a85c2

As title

update trove classifiers

87c86ae

As title

print the OSError instead of swallowing it

334c000

As title

esc added the 2 - In Progress label Sep 29, 2021

esc mentioned this pull request Oct 7, 2021

python 3.10 support #740

Closed

esc added 3 - Ready for Review and removed 2 - In Progress labels Oct 7, 2021

sklam reviewed Oct 11, 2021

View reviewed changes

sklam added 4 - Waiting on author and removed 3 - Ready for Review labels Oct 11, 2021

buffer all errors, and echo them at the end

8cc3c39

This will avoid printing errors unnecessarily.

esc added this to the 0.37.1 milestone Oct 13, 2021

esc mentioned this pull request Oct 13, 2021

llvmlite 0.37.1 #777

Closed

15 tasks

sklam reviewed Oct 13, 2021

View reviewed changes

setup using Python 3.10 on public CI too

408c7ba

As title

esc mentioned this pull request Oct 20, 2021

Test python3.10 and compiler pin #782

Closed

esc added Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm 4 - Waiting on reviewer and removed 4 - Waiting on author labels Oct 20, 2021

esc mentioned this pull request Nov 16, 2021

don't build docs on every platform, only once #789

Merged

stuartarchibald modified the milestones: 0.37.1, Version 0.38.0 RC Nov 17, 2021

esc added BuildFarm Passed For PRs that have been through the buildfarm and passed and removed Pending BuildFarm For PRs that have been reviewed but pending a push through our buildfarm labels Nov 18, 2021

gmarkall approved these changes Nov 23, 2021

View reviewed changes

esc added 5 - Ready to merge and removed 4 - Waiting on reviewer labels Nov 23, 2021

esc merged commit 0cde753 into numba:master Nov 23, 2021

esc deleted the python3.10 branch January 28, 2022 15:39

jamesobutler mentioned this pull request Dec 5, 2022

Bump minimum supported Python version to 3.8 #892

Merged

Python 3.10 #769

Python 3.10 #769

Conversation

esc commented Sep 2, 2021

esc commented Oct 7, 2021

azure-pipelines bot commented Oct 7, 2021

esc commented Oct 7, 2021

esc commented Oct 11, 2021

azure-pipelines bot commented Oct 11, 2021

sklam Oct 11, 2021

Choose a reason for hiding this comment

esc Oct 12, 2021

Choose a reason for hiding this comment

sklam Oct 12, 2021 • edited Loading

Choose a reason for hiding this comment

esc Oct 13, 2021

Choose a reason for hiding this comment

esc Oct 13, 2021

Choose a reason for hiding this comment

sklam left a comment • edited Loading

Choose a reason for hiding this comment

sklam commented Oct 13, 2021

esc commented Oct 13, 2021

esc commented Oct 14, 2021

esc commented Oct 20, 2021

esc commented Oct 25, 2021

azure-pipelines bot commented Oct 25, 2021

azure-pipelines bot commented Nov 15, 2021

esc commented Nov 16, 2021

esc commented Nov 18, 2021

azure-pipelines bot commented Nov 18, 2021

esc commented Nov 18, 2021

esc commented Nov 18, 2021

esc commented Nov 18, 2021

jakirkham commented Nov 18, 2021

esc commented Nov 19, 2021

mbargull commented Nov 19, 2021

mbargull commented Nov 19, 2021

mbargull commented Nov 19, 2021

esc commented Nov 19, 2021

esc commented Nov 20, 2021

jakirkham commented Nov 20, 2021

mbargull commented Nov 22, 2021 • edited Loading

esc commented Nov 22, 2021

esc commented Nov 22, 2021 • edited Loading

gmarkall left a comment

Choose a reason for hiding this comment

esc commented Nov 23, 2021

sklam Oct 12, 2021 •

edited

Loading

sklam left a comment •

edited

Loading

mbargull commented Nov 22, 2021 •

edited

Loading

esc commented Nov 22, 2021 •

edited

Loading