static callbacks

Create issue
Issue #232 resolved
Glyph created an issue

This is an alternate way to address #231 and some issues that stem from it and the fix.

The solution proposed on #231 to avoid depending on writable+executable pages uses a trick within libffi to map the same page twice; once writably, once executably. It seems likely to me that SELinux, or perhaps some more restrictive environment, will eventually cotton on to this trick and disallow it somehow. That should certainly be implemented as an improvement to the modernity of the usage of libffi.

While you can't fully replicate the functionality of ffi.callback without this functionality - it's literally impossible to make a function pointer without writing to an executable page - it might be desirable to have some libraries work entirely without relying on writing to executable pages, at least on CPython. I am pretty sure that this trick won't work on mobile devices, for example, because you just can't make writable stuff executable at all. (And that is CPython's main advantage over PyPy these days; the ability to operate in this mobile-friendly mode.)

If you were writing code in C, the way you generally handle "closures" is that somebody passes you a void* and you figure out what function to execute from there. CFFI already has some affordances in this area, in the form of ffi.new_handle / ffi.from_handle. It could also have a static_callback decorator so that you could say for example

    cffi_callback int python_callback(int how_many, int *values, void *whatever);

and then do something like this, at global scope, to "fill in the blank" left by the cffi_callback declaration:

@lib.static_callback("python_callback")
def pcb(how_many, values, whatever):
    whatever = ffi.from_handle(whatever)
    return int(whatever())

(maybe it would be @ffi.static_callback, I'm not sure how these things are classified)

Implementation-wise, this would produce a static python_callback function in the cffi output which would be a fully implemented C function, not a function pointer, and then a Python object pointer (PyObject* or whatever the PyPy equivalent is) that points at the python-level function. So much what CPython's internal APIs do for registering callbacks: stuff them away in whatever C struct is already designed to hold state for callbacks.

Comments (7)

  1. Armin Rigo

    Maybe this solution looks less magical: in the set_source(), you would implement a function like this:

        static int foobar(int x) {
            return CFFI_CALL_PYTHON(int baz(int), x);
        }
    
  2. Glyph reporter

    @arigo that looks fine to me, although it actually looks more magical from my perspective :). Would I be doing @ffi.static_callback("foobar") or @ffi.static_callback("baz") in that case? If baz, it doesn't seem very clear that that is actually a global top-level name in the set_source C namespace...

  3. Armin Rigo

    Yes, I'm still trying to figure out a good way that is also easily implementable. The current version looks like this: you'd say in the set_source():

    static CFFI_CALL_PYTHON(foobar, "int(int)");
    

    at the correct place for the declaration (so after the types you use are declared, but before you're using foobar). It would expand to a function int foobar(int) { ... }, which can be static like above. And then from Python you do @ffi.call_python("foobar"). The decorated function turns into a cdata of type int(*)(int), like with ffi.callback(), which you can pass around --- or you can pass around &foobar from C directly.

  4. Armin Rigo

    ...or, just take exactly what you suggested, which gives the same result except that you can't use &foobar in the C code; you have to pass it from Python. It is cleaner than dropping a magic macro line in the middle of set_source()... and maybe if you really want to, you can still obtain foobar from the C code by writing a prototype for that function. Ok, roundabout but I'm now convinced that your suggestion was the best all along :-)

  5. Log in to comment