Option to link _cffi_backend into generated module.c

Issue #210 resolved
C Anthony Risinger
created an issue

Possible alternative to cffi-runtime package -- could this work? At runtime, maybe the builtin's used (PyPy), or maybe the "statically" linked _cffi_backend in module.so is used (CPython).

Comments (9)

  1. Armin Rigo

    There is a lot of confusion this way: if a Python program loads two modules that each statically link _cffi_backend, then we end up with two independent CData and CType types. This is more-or-less fine if the two modules are well-separated. However, we'd run into endless fun if e.g. one module produces a <cdata 'void *'> and we try to pass it to the other module ("TypeError: 'cdata' object expected, got a 'cdata' object")...

  2. C Anthony Risinger reporter

    I was thinking there'd be a little indirection involved -- both PyPy and the module.so would have internal copies (static storage maybe) of whatever _cffi_backend implements (I admit I've not delved into the details), and the first once initialized would register a Python-level _cffi_backend module, and create hooks there pointing to it's implementation. In PyPy's case, since _cffi_backend already exists, when the embedded version booted in module.so, it would be a no-op (it's implementation would not be used, incoming calls would be forwarded to real impl, if necessary).

  3. C Anthony Risinger reporter

    The context for this request stems this ticket:

    https://github.com/unbit/uwsgi/pull/852#issuecomment-115370199

    I sort of unofficially maintain the pyuwsgi plugin because I think I might be the only one using it :) This plugin sort of "completes" the relationship between Python and uWGSI. Normally uWSGI starts up then initializes Python, whereas pyuwsgi allows uWSGI to be built as a proper Pyton extension, and initialized/configured directly from a host Python process.

    This allows simpler testing of apps and uWSGI itself, makes uWSGI integrate more naturally in Python ecosystem, and makes the (previously "virtual" only) uwsgi python module always importable.

    So I'm trying to find a way to create a _uwsgi.so that is both PyPy and CPython compatible, then import that from a new uwsgi.py module using cffi -- once uWSGI starts it simply loads the existing uwsgi module from sys.modules and adds stuff to that. This would enable all Pythons to import uwsgi and optionally start it up.

  4. Armin Rigo

    It seems to me that the present issue is only about including the content of the _cffi_backend module inside the CFFI-generated module, instead of requiring it to be separately installed. With enough hacks, indeed, it can be made exactly equivalent to loading a separate _cffi_backend module. But I mostly don't see the point because listing "cffi" in the dependencies is not hard; and I certainly don't understand how this relates to that other ticket...

  5. C Anthony Risinger reporter

    You are right, adding cffi to deps is not a hard thing per se... it's more political really? I'm not sure? People don't like gaining new deps, for whatever reasons. A compiled module's embedded _cffi_backend only comes into play if:

    import _cffi_backend

    fails. When it does (the first time any cffi based extension intializes), then the extension should call a local function with the purpose of "init_cffi_backend", ie. it will create a _cffi_backend module, populate it with the embedded definitions, and place into sys.modules so all future cffi extensions use it.

    If the user wants control over the loaded version (since it would depend on which cffi extension loaded first), they can simply install the cffi pypi package like usual.

    Benefits I see are:

    • Eliminate setup.py logic for older cffi versions(?)
    • Makes module.c 100% self-contained
    • Makes cffi a build-time only option, if desired. The generated C extension is ready-to-go as-is for PyPy or CPython.
    • Allows cffi adoption to spread "behind the scenes" without affecting user-facing things like deps
    • Makes the PyPy and CPython variants work the same with respect to _cffi_backend (neither require cffi package)
    • Would continue to work if CPython adds cffi to stdlib
  6. Armin Rigo

    The main drawback is that it would add versioning nightmares in large projects, if there are several such modules that are imported. Depending on the import order, it might install an older version of _cffi_backend, which then another module would try and fail to use because it requires a more recent version. Then you swap some imports and it works fine. With an explicit dependency, the setup.py system installs the current version of _cffi_backend and all modules should work.

    I think the deps affected are already behind the scenes. For example, if you installed PyOpenSSL recently, you might not have noticed it but it depends on CFFI.

  7. Armin Rigo

    For the "if CPython adds cffi to stdlib" part, the problem is the same as for any other package. It can be resolved like we did in PyPy, which is to put a cffi.egg-info directory inside the stdlib.

  8. C Anthony Risinger reporter

    My only motivation here was to allow me to generate a _uwsgi.so without requiring adding runtime deps to uWSGI itself. I also thought it might be a cool selling point for some folks (say, to ship a network-free all-in-one module).

    Closing for lack of interest.

  9. Log in to comment