Cannot initialize block support

Issue #218 resolved
Wayne Keenan
created an issue

I will create a simple test case if needed, but I wondered if anyone had seen these before and might have any clues as to how I can fix them?

For now I only have the output from something a bit more involved, which is using CoreBlutooth on a background thread using the 'dispatch' workaround (Issue #215)

python3 setup.py test

on this branch: https://github.com/TheCellule/python-bleson/tree/ci_tweaks

The errors below don't appear when I run the code standalone as a 'normal' script. But, when run as a single test or test suite via 'python3 setup.py test' I see:

For a single test:

/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/importlib/_bootstrap_external.py:426: ImportWarning: Not importing directory /usr/local/lib/python3.6/site-packages/PyObjCTools: missing __init__
  _warnings.warn(msg.format(portions[0]), ImportWarning)

For a test suite where multiple tests run (but not concurrently) in the same process (probably the key there):

======================================================================
ERROR: test_all_examples (test_examples.TestExamples)
----------------------------------------------------------------------
objc.internal_error: Cannot initialize block support

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/tests/test_examples.py", line 40, in test_all_examples
    runpy.run_path(script)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/tests/../examples/basic_advertiser.py", line 12, in <module>
    adapter = get_provider().get_adapter()
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/__init__.py", line 17, in get_provider
    from bleson.providers.macos.macos_provider import MacOSProvider
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/macos/macos_provider.py", line 2, in <module>
    from .macos_adapter import CoreBluetoothAdapter
  File "/Users/wayne/local.projects/TheCellule/python-bleson-api-sandpit/bleson/providers/macos/macos_adapter.py", line 5, in <module>
    from Foundation import *
  File "/usr/local/lib/python3.6/site-packages/Foundation/__init__.py", line 8, in <module>
    import objc
  File "/usr/local/lib/python3.6/site-packages/objc/__init__.py", line 18, in <module>
    _update()
  File "/usr/local/lib/python3.6/site-packages/objc/__init__.py", line 15, in _update
    import objc._objc as _objc
SystemError: <built-in method replace of bytes object at 0x10bea1c88> returned a result with an error set

Comments (12)

  1. Ronald Oussoren repo owner

    I haven't seen this error before, and PyObjC error message ("Cannot initialise block support") is an error that shouldn't happen. For some reason adding a PyObjC method to the Objective-C class for blocks doesn't work.

    One thing to look into: is PyObjC loaded multiple times? Two possible reasons for this:

    1. PyObjC was vendored somewhere in the tree and therefore imported under two names

    2. The tests load and unload PyObjC's python packages (for example by clearing sys.modules between tests)

    Unloading PyObjC's C extensions is not supported because PyObjC makes changes in the Objective-C runtime and those cannot be reverted.

  2. Wayne Keenan reporter

    On the theory it is some kind of 'unittest' loader/importer issue I lucked on a workaround.

    Running the methods below inside and outside a virtual env, with two different Python3.x and with pip-9.0.1, setuptools-37.0.0, wheel-0.30.0 :

    1 Homebrew

    Python 3.6.3 (default, Oct  4 2017, 06:09:15) 
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
    

    2 Python.org

    Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
    

    Method 1

    This works on both:

     python3 -m unittest discover  'tests'
    

    Method 2

    This fails on both:

    python3 setup.py test
    

    Methods 2 runs the same tests as method 1 but programatically launched in
    https://github.com/TheCellule/python-bleson/blob/master/setup.py#L20

    Note:

    Only when using method 2 and only with python 3.5 does the afore mentioned warning appear:

    /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/importlib/_bootstrap_external.py:412: ImportWarning: Not importing directory /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/PyObjCTools: missing __init__
      _warnings.warn(msg.format(portions[0]), ImportWarning)
    
  3. Ronald Oussoren repo owner

    I've tested this with an (oldish) python.org 3.5 install on macOS 10.12 and cannot reproduce the problem.

    I do get the ImportWarning, that's probably due to the discover command not honouring pth files (PyObjCTools is a namespace package, and pip therefore doesn't install an init.py file). The warning should be harmless, if otherwise you'd get ImportErrors later on due to not being able to load modules in the PyObjCTools package.

    This is pretty annoying, I don't know why you get an error and debugging is rather hard when I cannot reproduce the problem.

  4. Wayne Keenan reporter

    Continuing with the hunch... whilst creating simpler setup.py scripts I found that the line test_suite='setup.test_suite' is the candidate for the cause of the load/unload (or double import), it causes setup.py to be imported (by itself) again.

    Obviously simpler tests in a test suite that can reproducibly fail for all would be ideal.

    But how best to mitigate it? and where, in user code or pyobjc?

    Test Cases:

    Test 1

    Fails (as expected, it's just a trimmed down version of the original setup.py):

    test1_setup.py

    import unittest
    from setuptools import setup, find_packages
    
    def test_suite():
        test_loader = unittest.TestLoader()
        test_suite = test_loader.discover('tests')
        return test_suite
    
    setup(
        name='bleson',
        version='0.0.1',
        packages= [],
        test_suite='setup.test_suite')
    

    Test 2

    Works (bare bone unittest discovery, I assume it to be functionally identical to running the command line -m unitest discovery ..., which works) :

    test2_setup.py

    import unittest
    test_loader = unittest.TestLoader()
    test_suite = test_loader.discover('tests')
    unittest.TextTestRunner().run(test_suite)
    

    Test 3

    python3 setup.py test working case,

    Because of the 'self import' in test 1 the test suite has been moved to mytests.py to avoid (re)importing setup.py:

    test3_setup.py:

    import unittest
    from setuptools import setup, find_packages
    
    setup(
        name='bleson',
        version='0.0.1',
        packages= [],
        test_suite='mytests.test_suite')
    

    mytests.py

    import unittest
    
    def test_suite():
        test_loader = unittest.TestLoader()
        test_suite = test_loader.discover('tests')
        return test_suite
    
  5. Wayne Keenan reporter

    I've create a bare bones test case which only references setuptools, unittest, and PyObjC code in 2 files, here: https://github.com/TheCellule/python-bleson/tree/master/sandpit/pyobjc_import_test

    setup.py

    import unittest
    from setuptools import setup, find_packages
    
    def test_suite():
        test_loader = unittest.TestLoader()
        test_suite = test_loader.discover('tests')
        return test_suite
    
    setup(
        name='testpkg',
        version='0.0.1',
        packages= [],
        test_suite='setup.test_suite')
    

    test_case1.py

    #!/usr/bin/env python3
    import unittest
    import CoreBluetooth    # Comment out to fix 'objc.internal_error: Cannot initialize block support'
    
    class TestRoles(unittest.TestCase):
        pass 
    

    To remove ambiguity:

    Environment:

    macos 10.12.6

    Pythons tested with:

    Python 3.6.3 (default, Oct  4 2017, 06:09:15) 
    [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
    

    (homebrew)

    Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
    [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
    

    (python.org)

    Name: pyobjc-core Version: 4.0.1

    Name: pyobjc-framework-CoreBluetooth Version: 4.0.1

  6. Wayne Keenan reporter

    Of course, if I had used the overloaded feature of setup()'s test_suite parameter and just put the folder name, i.e. test_suite='tests', I would not have opened this can.

  7. Ronald Oussoren repo owner

    Thanks for the barebones test case, I can now reproduce the issue (python.org 3.6.3 with pyobjc 4.0.1 on macOS 10.3.1)

    Now I just have to find the cause of this error...

  8. Ronald Oussoren repo owner

    What's pretty interesting is the following output of an instrumented build:

    running build_ext
    objc._objc module initializer called
    
    ----------------------------------------------------------------------
    Ran 0 tests in 0.000s
    
    OK
    objc._objc module initializer called
    test_case1 (unittest.loader._FailedTest) ... ERROR
    
    ======================================================================
    ERROR: test_case1 (unittest.loader._FailedTest)
    ----------------------------------------------------------------------
    ImportError: Failed to import test module: test_case1
    objc.internal_error: Cannot initialize block support
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
        module = self._get_module_from_name(name)
      File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
        __import__(name)
      File "/Users/ronald/tmp/python-bleson/sandpit/pyobjc_import_test/tests/test_case1.py", line 3, in <module>
        import CoreBluetooth    # Comment out to fix 'objc.internal_error: Cannot initialize block support'
      File "/Users/ronald/tmp/python-bleson/sandpit/testenv/lib/python3.6/site-packages/CoreBluetooth/__init__.py", line 8, in <module>
        import objc
      File "/Users/ronald/Projects/pyobjc-hg/pyobjc/pyobjc-core/Lib/objc/__init__.py", line 18, in <module>
        _update()
      File "/Users/ronald/Projects/pyobjc-hg/pyobjc/pyobjc-core/Lib/objc/__init__.py", line 15, in _update
        import objc._objc as _objc
    SystemError: <built-in method replace of bytes object at 0x1108f6b98> returned a result with an error set
    
    
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s
    
    FAILED (errors=1)
    Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
    error: Test failed: <unittest.runner.TextTestResult run=1 errors=1 failures=0>
    

    Note how there are two lines with "Ran ... tests in ... seconds", and two lines with "objc._objc module initializer called".

    And this points to the source of the problem:

    setup(
        ...
        test_suite='setup.test_suite'
        ...
    )
    

    This causes setuptools to perform "import setup", which runs setup.py again (hence the two tests runs). I guess the test runner/discoverer in setuptools cleans up sys.modules after (or before) a test run, which results in two imports of objc and the error message you got.

    There are two possible fixes to this problem:

    1) Use __main__.test_suite instead of setup.test_suite

    2) Move the call to setup() into an "if __name__ == '__main__'"block.

    The first option is IMHO the cleanest solution.

  9. Wayne Keenan reporter

    A third option, as mentioned in one of my previous comments, would be to specify the folder instead, i.e. test_suite='tests'

    But I think all 3 of the options don't follow the principle of 'the path of least surprise' and how many as yet unknown, or knowable, other scenarios outside of this setuptools case need this kind of work around?

    IMHO using test_suite='setup.tests' should not need working around in any existing or new enduser developed code; preventing the loading of the native components more than once should be done in a single place, inside the module.

  10. Ronald Oussoren repo owner

    This is not a bug in PyObjC, to make the cause of the error clearer, this is basically what happens:

    import unittest
    from setuptools import setup, find_packages
    from setuptools.command import test
    
    def test_suite():
        test_loader = unittest.TestLoader()
        test_suite = test_loader.discover('tests')
        return test_suite
    
    class my_test (test.test):
        def run(self):
            test.test.run(self)
            test.test.run(self)
    
    setup(
        name='testpkg',
        version='0.0.1',
        packages= [],
        test_suite='__main__.test_suite',
        cmdclass=dict(test=my_test))
    

    The custom "test" command runs the setuptools command twice. This runs the testsuite twice, and causes the problem you can into.

    That causes problems because setuptools resets sys.modules after running tests (that is, saves a copy of sys.modules before running the testsuite and resets sys.modules to that copy afterwards).

    Because sys.modules is reset the second run of the test suite doesn't know that PyObjC (or any other module you use) is already loaded and tries to load them again and that results in problems.

    I could probably detect this, but avoiding the error you're getting makes the code base more complex for an obscure error.

    I'm about to commit a changeset that improves the error message, but that's all I'll do.

    P.S. Note that your current setup.py file runs the entire testsuite twice

  11. Ronald Oussoren repo owner

    Issue #218: Explicitly raise ImportError when trying to reload objc._objc

    Without this changeset reloading causes an exception about not being able to initialize block support, which is confusing. The new exception at least makes it clear that something fishy is going on.

    Fixes #218

    → <<cset 79b0d3f88fb9>>

  12. Log in to comment