pypy binary is linked to too much stuff, which breaks manylinux wheels

Issue #2617 new
Nathaniel Smith
created an issue

With pypy 5.8, ldd pypy shows that it's linked to libbz2, libcrypto, libffi, libncurses, ... Likewise for pypy3 5.8 and ldd pypy3.5.

You would not think so, but it turns out that this is a big problem for distributing wheels.

The issue is that the way ELF works, any libraries that show up in ldd $TOPLEVELBINARY effectively get LD_PRELOADed into any extension modules that you load later. So, for example, if some wheel distributes its own version of openssl, then any symbols that show up in both their copy of openssl and pypy's copy of openssl will get shadowed and hello segfaults.

The cryptography project recently ran into this with uwsgi:

Fortunately this has not been a big deal so far because, uh... nobody distributes pypy wheels. But in the future maybe this is something that should be supported :-). And while in theory it would be nice if this could be fixed on the wheel side, this is not trivial.

The obvious solution would be to switch things around so that the top-level pypy executable does dlopen("", RTLD_LOCAL) to start the interpreter, instead of linking against it with -lpypy-c. Then the symbols from and everything it links to would be confined to an ELF local namespace, and would stop polluting the namespace of random extension modules.

However... there is a problem, which is that cpyext extension modules need some way to get at the C API symbols, and I assume cffi extension modules need access to some pypy symbols as well.

This is... tricky, given how rpython wants to mush everything together into one giant .so, and ELF makes it difficult to only expose some symbols from a binary like this. Some options:

  • when using libcrypto or whatever from rpython, use dlopen("libcrypto", RTLD_LOCAL) instead of -lcrypto. I guess this could be done systematically in rffi?
  • provide a special libcpyext that uses dlopen to fetch the symbols from and then manually re-exports them?

Comments (11)

  1. Armin Rigo

    For PyPy, we could imagine a solution where we use dlopen("", RTLD_LOCAL), and then not using the linker at all for cpyext and cffi modules. This would avoid this whole class of problem.

    Right now, a cpyext module is compiled with a large number of #define PyFoo PyPyFoo, and the names PyPyFoo are symbols exported by the main The solution that completely avoids relying on the platform-specific linker would be to compile them instead with #define PyFoo (_pypy_cpyext.PyFoo) and change the initialization code so that the (cpyext-module-local) structure _pypy_cpyext is filled with function pointers when the extension module is loaded. The same can be done with cffi's few functions.

    It's a bit of work and requires a full breakage of cpyext's binary API (which is fine I guess). The upside is that we don't rely on the linker to find the symbols from the that imported the extension module, as opposed to not finding them at all (because of RTLD_LOCAL) or even finding the wrong ones (if there are several loaded in the same process).

    For backward compatibility with the old way to embed PyPy, we'd retain a few exported symbols on like pypy_execute_source.

    I think that what I'm suggesting would always just work, but I may be missing something...

  2. Nathaniel Smith reporter

    That sounds plausible to me.

    Note that on Windows this issue doesn't come up, because symbols are always resolved as (dll name, symbol name) pairs (unless you have dll name collisions, but there are ways to avoid that). But your suggestion should also work fine.

    On MacOS... it's complicated. The default and recommended way to do things is to use "two-level namespace" lookup which is like Windows, except it uses (path to shared library, symbol name), so you actually have to know -- at compile time! -- where the shared library will be placed on the end user's disk (either absolute, or relative to the extension module). Alternatively, you can use the old "single-level namespace" lookup, which is like ELF, except without RTLD_LOCAL – it really is just a single big soup of symbols for the whole process. I believe that what people usually do in practice on CPython is to use single-level lookup to find the CPython C API symbols (because you certainly don't know where the Python interpreter will be installed), and two-level lookup for vendored libraries. (Or at least that's how it's supposed to work?) Your proposal + always using two-level namespaces seems like it might even be an improvement on this, but since it's different from what CPython does you might have to hack at distutils a bit to get it to play along.

    One thing you have to be careful of is ABI compatibility – if you add a new symbol to cpyext and do nothing else, then with traditional linking that's backwards compatible. But with the _pypy_cpyext trick, the struct needs to get bigger, so either you need to bump your ABI compatibility version, or else you need to somehow detect that an extension was built against an older version of cpyext, and only fill in as much of the struct as it knows about. (This is what numpy does.)

    On the other hand, this also potentially allows you quite a bit of freedom when it comes to evolving the cpyext ABI and controlling extension module ABI breakage -- for example, you could have different incompatible versions of the same PyWhatever symbol, and use metadata from the extension module to figure out which version it wants.

  3. Nathaniel Smith reporter

    Right, the connection to the cryptography issue isn't that uwsgi+pypy could somehow avoid this by changing pypy -- it's uwsgi causing the problem there. The connection is that right now, even the plain pypy executable causes the same problem, even without uwsgi getting involved :-).

  4. mattip

    @Armin Rigo when you say

    The solution that completely avoids relying on the platform-specific linker would be to compile them instead with #define PyFoo (_pypy_cpyext.PyFoo) and change the initialization code so that the (cpyext-module-local) structure _pypy_cpyext is filled with function pointers when the extension module is loaded

    the "initialization code" would then be like numpy's _import_array() that each c-extension module that uses numpy (pandas, matplotlib) is meant to call? Or is this the general PyPy startup code run in app_main ?

  5. Nathaniel Smith reporter

    I think the idea is that it would be like import_array, except that the
    interpreter calls it automatically instead of requiring each module to do
    it explicitly. How to actually implement this is a bit trickier though :-).

    It could be checked and lazily initialized in every cpyext API call.
    There's some overhead here, but maybe it's not too bad on modern CPUs.

    With some cleverness, it might be possible to convince each cpyext module
    to export a special symbol, that pypy dlsym's and fills in itself. I don't
    think this can be done using only standard C though; you could have your
    Python.h declare a public variable, but then if a single extension is built
    out of multiple files you'd get collisions between each file's copy of the
    variable. Some platforms have ways to declare that redundant definitions
    should be merged; maybe that could be made to work.

    Or given that CPython has already done the work of figuring out how to
    export its symbols into extension module namespaces, maybe the thing to do
    is to continue to copy them. But if you want to avoid exporting too much,
    and to gain the ability to support multiple versions of the cpyext ABI
    simultaneously, then you could combine the two approaches: have the one
    symbol you export from pypy be pypy_cpyext_abi_v1, and then later you could
    add a pypy_cpyext_abi_v2, etc.

  6. Nathaniel Smith reporter

    It's something that should be sorted out whenever you want people to start putting wheels up on PyPI, but I don't see why it should be a blocker for PyPy6 in particular.

  7. Nathaniel Smith reporter

    I should clarify though: despite the title, this issue doesn't literally break all manylinux wheels, just the ones that have symbol collisions with any of the libraries that are linked to the main pypy binary. (Usually because they use that library themselves.) So it may not affect scipy.

  8. Log in to comment