issue with Py_LIMITED_API on Windows

Issue #350 resolved
Cosimo Lupo
created an issue

When building a cffi extension module on Windows in API mode (i.e. set_source), instead of linking with the PYTHON3.DLL associated with the stable Py_LIMITED_API, the cffi extension module gets linked with the version-specific PYTHON35.DLL or PYTHON36.DLL.

Therefore, the module fails to be imported on a different CPython 3 minor version, even if the file name has a generic *.pyd suffix instead of the full *.cp36-win_amd64.pyd suffix. The generic filename is because cffi forces the use of setuptools' Extension py_limited_api=True option. However it is incorrect, because if one inspects the *.pyd file with Dependency Walker, one can see that the versioned PYTHON36.DLL is being linked.

The reason for this is that the name of the python DLL import library that is used when linking an extension module compiled with MSVC is not passed by distutils to the linker command line, but it is implicitly defined inside the pyconfig.h header via a #pragma directive, and this is based on the current value of Py_LIMITED_API:

https://github.com/python/cpython/blob/3460198f6ba40a839f105c381f07179aba1e8c61/PC/pyconfig.h#L288-L289

The _cffi_include.h header attempts to #define Py_LIMITED_API but only after including the pyconfig.h header. By the time it includes the Python.h, it is too late, since the pyconfig.h has already been included once and it is guarded by a #ifndef Py_CONFIG_H at the top, so the second time it's included by Python.h nothing happens (and the wrong python DLL gets linked).

The only workaround I could find to force the cffi extension module to be linked with PYTHON3.DLL instead of PYTHON36.DLL is to explicitly pass a define_macros=[("Py_LIMITED_API", None)] to the ffibuilder.set_source call, which will then forward the preprocessor definition to setuptools/distutils when the latter build up the linker command line.

When doing that, the _cffi_include.h's #if branch is skipped because Py_LIMITED_API was already defined.

It's a pity that one has to set that with define_macros in order to make it work on Windows. On Linux or macOS it doesn't seem necessary to do that.

Comments (12)

  1. Armin Rigo

    Ok, so we have a chicken-and-egg problem. We can't define Py_LIMITED_API if pyconfig.h defines Py_DEBUG or Py_TRACE_REFS or Py_REF_DEBUG, otherwise other things explode. But now you say we must define Py_LIMITED_API before including pyconfig.h. It looks very much like Py_LIMITED_API was never really tested or used and is thus broken in various ways.

    I suppose there is no clean way to solve the issue, but maybe we can on Windows reverse the logic: we can guess that Py_DEBUG will be defined if and only if _DEBUG is defined, and the other two macros won't be defined; then we set Py_LIMITED_API, do the include of pyconfig.py, and finally sanity-check what we just guessed.

  2. Cosimo Lupo reporter

    Thanks! I’ll check as soon as I get to the office. One thing I’m wondering is if checking WIN32 may be too broad. The issue was with the MSVC compiled CPython, but WIN32 is defined for MinGW GCC as well (I think). But I guess that’s not officially supported..

  3. Cosimo Lupo reporter

    I just tried with cpython 3.5 and 3.6 (official binaries from python.org), and cffi from latest master. It's working without needing to add define_macros=[("Py_LIMITED_API", None)]

    Thanks a lot!!! When do you think this will be released?

  4. Armin Rigo

    If I revert to the previous situation, the file is called foo.pyd but it is linking to python36.dll. Should I do that, or should I more clearly not define Py_LIMITED_API at all on Windows and have the file be called foo.cp36-win_amd64.pyd?

  5. Log in to comment