Accessing PyMOAB from Cython

[Guilherme Caminha]

I'm unable to cimport moab packages from Cython. I believe that the only cause is some include files that are not installed (such as TagInfo.hpp) but are included in the moab.pxd file. I'm not sure though if there is some option to install all the include files.

[Patrick Shriwise]

This an issue we've faced before when pymoab was a separate project from MOAB. TagInfo.hpp (and possibly other headers like it) contain access to deeper core functionality of MOAB. My understanding is that giving other applications access to those headers by installing them can leave the database vulnerable to unexpected behavior. Does that sound about right @vijaysm?

[Vijay M]

@pshriwise Yes this is correct. For example, installing TagInfo.hpp would require we also install SequenceManager.hpp, which handles the guts of the memory management in MOAB. This was one of the reasons we integrated PyMOAB to be part of the MOAB repository.

Let me think some more about this and see if we can refactor just the required interface for Tag so that it can be publicly installed.

@Estanho Are there other headers that are causing an issue as well ?

[Guilherme Caminha]

@vijaysm I believe so. When I removed the extern reference to TagInfo.hpp from the Cython header file (and all methods that had a Tag argument) it compiled fine.

[Patrick Shriwise]

Out of curiosity @Estanho, why would you like to cimport pymoab in another cython module instead of as a python package?

[Guilherme Caminha]

@pshriwise I'm optimizing some loops from my code. It would be interesting to access pymoab internals (such as the Core pointer) and call some methods directly from there, for example, and also avoid some Python object wrapping on tighter loops through moab entities. Another thing that could be interesting would be to declare pymoab methods as cpdefs instead of defs, so that they are callable from Python but also have a light wrapper for Cython, avoiding Python overhead. I will see if that makes any difference when testing.

[Patrick Shriwise]

@Estanho that all sounds good to me. I'm interested to hear how it goes.

[Guilherme Caminha]

@pshriwise Sure!

By the way @pshriwise @vijaysm I believe I have found a solution for that. Cython allows for class and type declaration. So, instead of changing MOAB's internals, I simply removed the mention to the TagInfo.hpp header from the moab.pxd file and placed the TagInfo class declaration there, under the cdef extern from 'moab/Types.hpp' namespace "moab" block:

    cdef cppclass TagInfo:
        pass

    ctypedef TagInfo* Tag;

This way, the specific TagInfo.hpp reference can be placed somewhere else, for example in the specific Tag header, without stopping someone from declaring a variable of type Tag and cimporting pymoab. This could be done for other headers that are not installed, if any is added in the future.

[Vijay M]

Interesting @Estanho. So is that like forward declaration in C++ ?

Which file will include the Tag header then ? The user code based on installation (where TagInfo.hpp is unavailable) or the PyMOAB code during compilation for symbol resolution (where it is available) ?

[Guilherme Caminha]

@vijaysm Exactly, it's just like forward declaration. The Types.hpp file does the same there. I believe the file Tag.pxd could include the TagInfo.hpp, however that would stop someone from cimporting Tag then. I'm not sure but maybe it could go on a isolated header which shouldn't be cimported, since the TagInfo.hpp doesn't have any user level API, I think.

[Vijay M]

Did we propagate this change back to remove the dependency on TagInfo.hpp ? If not, we should.

[Patrick Shriwise]

I don't think so. I'll try to add it soon.

One hang-up I see is that the pymoab Tag object calls some TagInfo methods directly (maybe it shouldn't?).

[Vijay M]

Yeah that shouldn't be happening. We should be using methods from the Tag public interface only. Is there any restriction or functionality that isn't currently exposed by Tag ?

[Patrick Shriwise]

The only thing I can think of is that it allows us to get that information without tying a MOAB (or pymoab) core instance to the Tag object to get information about it (name, type, length, etc.).

[Vijay M]

Hmm, is that a required feature ? I feel this is more of a helper functionality. And if it reduces dependency on private headers, I am all for removing such an interface. My 2 cents.

[Patrick Shriwise]

It's part of the interface now, but it could be done by tying a MOAB instance to the Tag class instead. It's just more straightforward to call those methods directly.

In the long run, it's probably better to go through a core interface to get that information anyhow. I'll update that before the next release.

[Patrick Shriwise]

I was messing around with this a little bit today. It isn't clear to me how placing the reference to TagInfo.hpp in Tag.pxd is going to avoid the cimport of that same information in core.pyx or any other module that relies on the ability to return a PyMOAB tag object.

It feels like we'd just be putting it in a different spot... Am I missing something here?

[Patrick Shriwise]

@Estanho, any further guidance on this?