cffi could support Py_LIMITED_API

Python 3 has a stable ABI. https://www.python.org/dev/peps/pep-0384/. It could be interesting for cffi to have a mode to take advantage of it. Then one could distribute a single cffi module for several Python 3 versions.

Did you try? Because of the potential mess involved, I won't go there without someone checking the basics first. I'm afraid we're depending on specific CPython versions at some places, because of an unstability of details from the official API; these details are probably not part of the stable ABI at all, but I don't know (search for VERSION in misc_thread_common.h).

I tried and there is a long list of missing functions. This is "on hold" until someone motivated enough shows up, because I personally don't have a usage for this.

Sorry, I just realized another interpretation of what you said: maybe it is not about _cffi_backend, but about the C extension modules produced by set_source() / compile(). Maybe you're saying that it would be useful if these modules were using only the limited API.

This is easy because these modules use only a very limited subset of the API anyway, at least for the non-embedding mode. I've fixed the only place where they use a "non-limited" API in 1629330be7e6.

So it seems to be done, but distutils still produces names like foo.cpython-35m-x86_64-linux-gnu.so. I guess we can manually rename them, but isn't there some more official way?

I've been doing it this way. It's time to stop using distutils to compile extensions. https://bitbucket.org/dholth/cryptacular/src/tip/SConstruct

This one is not as pretty and doesn't set the flag for ABI3, but it does use cffi, and only recompiles or regenerates the C code when its source has changed. https://bitbucket.org/dholth/pysdl2-cffi/src/tip/SConstruct?#SConstruct-91

(SCons LoadableModule and SharedLibrary are equivalent after overriding all the CFLAGS with values from distutils)

I see that distutils is half-deprecated, but I can't depend on another third-party library inside CFFI. If there is no standard solution, then I guess I can just document the presence of the Py_LIMITED_API flag. That means it is up to individual CFFI module authors to use other tools to compile the CFFI-produced C files. I'd still like to know if just renaming the final .so is enough or not, and what it should be renamed to.

Yes renaming it is sufficient after compiling with correct flags. It's probably easy to override the get-filename from distutils.

What should it be renamed to? It should have a suffix from imp.get_suffixes() with abi3 in it.

>>> imp.get_suffixes()
[('.cpython-35m-darwin.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('.py', 'r', 1), ('.pyc', 'rb', 2)]

pip doesn't yet know about a wheel tag to express 'abi3 with cpython x or later'. Right now it only knows about 'abi3 with the current version of cpython'.

In [3]: from distutils.sysconfig import get_config_var
In [4]: get_config_var('EXT_SUFFIX')
Out[4]: '.cpython-35m-darwin.so'

distutils.command.build_ext:

    def get_ext_filename(self, ext_name):
        r"""Convert the name of an extension (eg. "foo.bar") into the name
        of the file from which it will be loaded (eg. "foo/bar.so", or
        "foo\bar.pyd").
        """
        from distutils.sysconfig import get_config_var
        ext_path = ext_name.split('.')
        ext_suffix = get_config_var('EXT_SUFFIX')
        return os.path.join(*ext_path) + ext_suffix

Is there any documentation or standard example of compiling a CPython extension module that contains #define Py_LIMITED_API 1 into the expected file name? Or are you saying that there is nothing better than: "we need to patch distutils by following the instructions given above"?

Haven't seen any. At least you can pass defines to distutils without patching. Also, py hex version -DPy_LIMITED_API=0x03030000 is better than 1, you get the limited API from that version 3.3 onward, it has more functions.

Unfortunately the PEP itself says that no support will be added to distutils for compiling these extensions "due to the distutils code freeze". I doubt that there is a better way than patching / extending distutils by overriding the build_ext command, but, changing command classes is the way to extend distutils.

I'm -0.5 about adding a feature to CFFI that has nothing to do with CFFI. If there is no officially-approved way neither in distutils nor in another widely-used .so-maker for CPython, then I think I'll not do more than emit the #define, and mention in the documentation that people can look around for 3rd-party ways to compile this C code.

Documentation written in 8f867f5a869f. Feel free to reopen if you feel that this is not satisfying.

Thank you for taking such a thorough look into the issue. Unfortunately as far as I can tell despite being a feature since Python 3.2 Py_LIMITED_API is a bit of an uncharted territory, but with any luck it can become less so, and reduce the burden of distributing cffi extensions for Windows. It is completely satisfying for cffi to stick to generating the code, and for some other software to worry about the matter of compiling and naming the artifact.