cffi could support Py_LIMITED_API

Issue #268 resolved
Daniel Holth
created an issue

Python 3 has a stable ABI. https://www.python.org/dev/peps/pep-0384/. It could be interesting for cffi to have a mode to take advantage of it. Then one could distribute a single cffi module for several Python 3 versions.

Comments (13)

  1. Armin Rigo

    Did you try? Because of the potential mess involved, I won't go there without someone checking the basics first. I'm afraid we're depending on specific CPython versions at some places, because of an unstability of details from the official API; these details are probably not part of the stable ABI at all, but I don't know (search for VERSION in misc_thread_common.h).

  2. Armin Rigo

    I tried and there is a long list of missing functions. This is "on hold" until someone motivated enough shows up, because I personally don't have a usage for this.

  3. Armin Rigo
    • changed status to open

    Sorry, I just realized another interpretation of what you said: maybe it is not about _cffi_backend, but about the C extension modules produced by set_source() / compile(). Maybe you're saying that it would be useful if these modules were using only the limited API.

    This is easy because these modules use only a very limited subset of the API anyway, at least for the non-embedding mode. I've fixed the only place where they use a "non-limited" API in 1629330be7e6.

    So it seems to be done, but distutils still produces names like foo.cpython-35m-x86_64-linux-gnu.so. I guess we can manually rename them, but isn't there some more official way?

  4. Armin Rigo

    I see that distutils is half-deprecated, but I can't depend on another third-party library inside CFFI. If there is no standard solution, then I guess I can just document the presence of the Py_LIMITED_API flag. That means it is up to individual CFFI module authors to use other tools to compile the CFFI-produced C files. I'd still like to know if just renaming the final .so is enough or not, and what it should be renamed to.

  5. Daniel Holth reporter

    Yes renaming it is sufficient after compiling with correct flags. It's probably easy to override the get-filename from distutils.

    What should it be renamed to? It should have a suffix from imp.get_suffixes() with abi3 in it.

    >>> imp.get_suffixes()
    [('.cpython-35m-darwin.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('.py', 'r', 1), ('.pyc', 'rb', 2)]
    

    pip doesn't yet know about a wheel tag to express 'abi3 with cpython x or later'. Right now it only knows about 'abi3 with the current version of cpython'.

  6. Daniel Holth reporter
    In [3]: from distutils.sysconfig import get_config_var
    In [4]: get_config_var('EXT_SUFFIX')
    Out[4]: '.cpython-35m-darwin.so'
    
    distutils.command.build_ext:
    
        def get_ext_filename(self, ext_name):
            r"""Convert the name of an extension (eg. "foo.bar") into the name
            of the file from which it will be loaded (eg. "foo/bar.so", or
            "foo\bar.pyd").
            """
            from distutils.sysconfig import get_config_var
            ext_path = ext_name.split('.')
            ext_suffix = get_config_var('EXT_SUFFIX')
            return os.path.join(*ext_path) + ext_suffix
    
  7. Armin Rigo

    Is there any documentation or standard example of compiling a CPython extension module that contains #define Py_LIMITED_API 1 into the expected file name? Or are you saying that there is nothing better than: "we need to patch distutils by following the instructions given above"?

  8. Daniel Holth reporter

    Haven't seen any. At least you can pass defines to distutils without patching. Also, py hex version -DPy_LIMITED_API=0x03030000 is better than 1, you get the limited API from that version 3.3 onward, it has more functions.

    Unfortunately the PEP itself says that no support will be added to distutils for compiling these extensions "due to the distutils code freeze". I doubt that there is a better way than patching / extending distutils by overriding the build_ext command, but, changing command classes is the way to extend distutils.

  9. Armin Rigo

    I'm -0.5 about adding a feature to CFFI that has nothing to do with CFFI. If there is no officially-approved way neither in distutils nor in another widely-used .so-maker for CPython, then I think I'll not do more than emit the #define, and mention in the documentation that people can look around for 3rd-party ways to compile this C code.

  10. Daniel Holth reporter

    Thank you for taking such a thorough look into the issue. Unfortunately as far as I can tell despite being a feature since Python 3.2 Py_LIMITED_API is a bit of an uncharted territory, but with any luck it can become less so, and reduce the burden of distributing cffi extensions for Windows. It is completely satisfying for cffi to stick to generating the code, and for some other software to worry about the matter of compiling and naming the artifact.

  11. Log in to comment