Cannot initialize block support

[ronaldoussoren]

Original report by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

I will create a simple test case if needed, but I wondered if anyone had seen these before and might have any clues as to how I can fix them?

For now I only have the output from something a bit more involved, which is using CoreBlutooth on a background thread using the 'dispatch' workaround (Issue #215)

#!python

python3 setup.py test

on this branch: https://github.com/TheCellule/python-bleson/tree/ci_tweaks

The errors below don't appear when I run the code standalone as a 'normal' script.
But, when run as a single test or test suite via 'python3 setup.py test' I see:

For a single test:

#!python

/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/site-packages/PyObjCTools: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)

For a test suite where multiple tests run (but not concurrently) in the same process (probably the key there):

#!python

======================================================================
ERROR: test_all_examples (test_examples.TestExamples)
----------------------------------------------------------------------
objc.internal_error: Cannot initialize block support

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/tests/test_examples.py", line 40, in test_all_examples
    runpy.run_path(script)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/tests/../examples/basic_advertiser.py", line 12, in <module>
    adapter = get_provider().get_adapter()
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/__init__.py", line 17, in get_provider
    from bleson.providers.macos.macos_provider import MacOSProvider
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/macos/macos_provider.py", line 2, in <module>
    from .macos_adapter import CoreBluetoothAdapter
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/macos/macos_adapter.py", line 5, in <module>
    from Foundation import *
  File "/usr/local/lib/python3.6/site-packages/Foundation/__init__.py", line 8, in <module>
    import objc
  File "/usr/local/lib/python3.6/site-packages/objc/__init__.py", line 18, in <module>
    _update()
  File "/usr/local/lib/python3.6/site-packages/objc/__init__.py", line 15, in _update
    import objc._objc as _objc
SystemError: <built-in method replace of bytes object at 0x10bea1c88> returned a result with an error set

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

I haven't seen this error before, and PyObjC error message ("Cannot initialise block support") is an error that shouldn't happen. For some reason adding a PyObjC method to the Objective-C class for blocks doesn't work.

One thing to look into: is PyObjC loaded multiple times? Two possible reasons for this:

1. PyObjC was vendored somewhere in the tree and therefore imported under two names

2. The tests load and unload PyObjC's python packages (for example by clearing sys.modules between tests)

Unloading PyObjC's C extensions is not supported because PyObjC makes changes in the Objective-C runtime and those cannot be reverted.

[ronaldoussoren]

Original comment by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

On the theory it is some kind of 'unittest' loader/importer issue I lucked on a workaround.

Running the methods below inside and outside a virtual env, with two different Python3.x and with pip-9.0.1, setuptools-37.0.0, wheel-0.30.0 :

1 Homebrew

#!python

Python 3.6.3 (default, Oct  4 2017, 06:09:15) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin

2 Python.org

#!python

Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

Method 1

This works on both:

#!python

 python3 -m unittest discover  'tests'

Method 2

This fails on both:

#!python

python3 setup.py test

Methods 2 runs the same tests as method 1 but programatically launched in
[https://github.com/TheCellule/python-bleson/blob/master/setup.py#L20
](Link URL)

Note:

Only when using method 2 and only with python 3.5 does the afore mentioned warning appear:

#!python

/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/importlib/_bootstrap_external.py:412: ImportWarning: Not importing directory /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/PyObjCTools: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

I've tested this with an (oldish) python.org 3.5 install on macOS 10.12 and cannot reproduce the problem.

I do get the ImportWarning, that's probably due to the discover command not honouring pth files (PyObjCTools is a namespace package, and pip therefore doesn't install an init.py file). The warning should be harmless, if otherwise you'd get ImportErrors later on due to not being able to load modules in the PyObjCTools package.

This is pretty annoying, I don't know why you get an error and debugging is rather hard when I cannot reproduce the problem.

[ronaldoussoren]

Original comment by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

Continuing with the hunch... whilst creating simpler setup.py scripts I found that the line test_suite='setup.test_suite' is the candidate for the cause of the load/unload (or double import), it causes setup.py to be imported (by itself) again.

Obviously simpler tests in a test suite that can reproducibly fail for all would be ideal.

But how best to mitigate it? and where, in user code or pyobjc?

Test Cases:#

Test 1

Fails (as expected, it's just a trimmed down version of the original setup.py):

test1_setup.py

#!python

import unittest
from setuptools import setup, find_packages

def test_suite():
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests')
    return test_suite

setup(
    name='bleson',
    version='0.0.1',
    packages= [],
    test_suite='setup.test_suite')

Test 2

Works (bare bone unittest discovery, I assume it to be functionally identical to running the command line -m unitest discovery ..., which works) :

test2_setup.py

#!python

import unittest
test_loader = unittest.TestLoader()
test_suite = test_loader.discover('tests')
unittest.TextTestRunner().run(test_suite)

Test 3

python3 setup.py test working case,

Because of the 'self import' in test 1 the test suite has been moved to mytests.py to avoid (re)importing setup.py:

test3_setup.py:

#!python

import unittest
from setuptools import setup, find_packages

setup(
    name='bleson',
    version='0.0.1',
    packages= [],
    test_suite='mytests.test_suite')

mytests.py

#!python

import unittest

def test_suite():
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests')
    return test_suite

[ronaldoussoren]

Original comment by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

I've create a bare bones test case which only references setuptools, unittest, and PyObjC code in 2 files, here: https://github.com/TheCellule/python-bleson/tree/master/sandpit/pyobjc_import_test

setup.py

#!python

import unittest
from setuptools import setup, find_packages

def test_suite():
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests')
    return test_suite

setup(
    name='testpkg',
    version='0.0.1',
    packages= [],
    test_suite='setup.test_suite')

test_case1.py

#!python

#!/usr/bin/env python3
import unittest
import CoreBluetooth    # Comment out to fix 'objc.internal_error: Cannot initialize block support'

class TestRoles(unittest.TestCase):
    pass 

To remove ambiguity:

Environment:

macos 10.12.6

Pythons tested with:

#!python

Python 3.6.3 (default, Oct  4 2017, 06:09:15) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin

(homebrew)

#!python

Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin

(python.org)

Name: pyobjc-core
Version: 4.0.1

Name: pyobjc-framework-CoreBluetooth
Version: 4.0.1

[ronaldoussoren]

Original comment by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

Of course, if I had used the overloaded feature of setup()'s test_suite parameter and just put the folder name, i.e. test_suite='tests', I would not have opened this can.

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Thanks for the barebones test case, I can now reproduce the issue (python.org 3.6.3 with pyobjc 4.0.1 on macOS 10.3.1)

Now I just have to find the cause of this error...

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

What's pretty interesting is the following output of an instrumented build:

running build_ext
objc._objc module initializer called

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
objc._objc module initializer called
test_case1 (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: test_case1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_case1
objc.internal_error: Cannot initialize block support

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/Users/ronald/tmp/python-bleson/sandpit/pyobjc_import_test/tests/test_case1.py", line 3, in <module>
    import CoreBluetooth    # Comment out to fix 'objc.internal_error: Cannot initialize block support'
  File "/Users/ronald/tmp/python-bleson/sandpit/testenv/lib/python3.6/site-packages/CoreBluetooth/__init__.py", line 8, in <module>
    import objc
  File "/Users/ronald/Projects/pyobjc-hg/pyobjc/pyobjc-core/Lib/objc/__init__.py", line 18, in <module>
    _update()
  File "/Users/ronald/Projects/pyobjc-hg/pyobjc/pyobjc-core/Lib/objc/__init__.py", line 15, in _update
    import objc._objc as _objc
SystemError: <built-in method replace of bytes object at 0x1108f6b98> returned a result with an error set


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)
Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>

Note how there are two lines with "Ran ... tests in ... seconds", and two lines with "objc._objc module initializer called".

And this points to the source of the problem:

#!python

setup(
    ...
    test_suite='setup.test_suite'
    ...
)

This causes setuptools to perform "import setup", which runs setup.py again (hence the two tests runs). I guess the test runner/discoverer in setuptools cleans up sys.modules after (or before) a test run, which results in two imports of objc and the error message you got.

There are two possible fixes to this problem:

1. Use __main__.test_suite instead of setup.test_suite

2. Move the call to setup() into an "if __name__ == '__main__'"block.

The first option is IMHO the cleanest solution.

[ronaldoussoren]

Original comment by WayneK (Bitbucket: WayneK, GitHub: WayneK).

---

A third option, as mentioned in one of my previous comments, would be to specify the folder instead, i.e. test_suite='tests'

But I think all 3 of the options don't follow the principle of 'the path of least surprise' and how many as yet unknown, or knowable, other scenarios outside of this setuptools case need this kind of work around?

IMHO using test_suite='setup.tests' should not need working around in any existing or new enduser developed code; preventing the loading of the native components more than once should be done in a single place, inside the module.

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

This is not a bug in PyObjC, to make the cause of the error clearer, this is basically what happens:

#!python

import unittest
from setuptools import setup, find_packages
from setuptools.command import test

def test_suite():
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests')
    return test_suite

class my_test (test.test):
    def run(self):
        test.test.run(self)
        test.test.run(self)

setup(
    name='testpkg',
    version='0.0.1',
    packages= [],
    test_suite='__main__.test_suite',
    cmdclass=dict(test=my_test))

The custom "test" command runs the setuptools command twice. This runs the testsuite twice, and causes the problem you can into.

That causes problems because setuptools resets sys.modules after running tests (that is, saves a copy of sys.modules before running the testsuite and resets sys.modules to that copy afterwards).

Because sys.modules is reset the second run of the test suite doesn't know that PyObjC (or any other module you use) is already loaded and tries to load them again and that results in problems.

I could probably detect this, but avoiding the error you're getting makes the code base more complex for an obscure error.

I'm about to commit a changeset that improves the error message, but that's all I'll do.

P.S. Note that your current setup.py file runs the entire testsuite twice

[ronaldoussoren]

Original comment by Ronald Oussoren (Bitbucket: ronaldoussoren, GitHub: ronaldoussoren).

---

Issue #218: Explicitly raise ImportError when trying to reload objc._objc

Without this changeset reloading causes an exception about not being able
to initialize block support, which is confusing. The new exception at least
makes it clear that something fishy is going on.

Fixes #218