stand-alone python installation

[matthew.broadway]

I recently ran into an issue where trying to import gtsam in python gave:

ImportError: libgtsam.so.4: cannot open shared object file: No such file or directory

I eventually found that this was because the shared library at gtsam/build/install/gtsam/libgtsam.so* was being used when loading into python. I had expected the gtsam.so in gtsam/build/cython/gtsam to be stand-alone and therefore thought that doing make install with -DCMAKE_INSTALL_PREFIX=./install didn't affect anything system-wide. However after deleting gtsam/build/install/gtsam I was no longer able to load the python wrapper (with the above error).

The instructions I have given elsewhere to use -DCMAKE_INSTALL_PREFIX=./install was based on the assumption that no system-wide changes to the PATH or anything else was taking place.

would there be a way to have a 'portable' install where everything that the python wrapper needs can be kept in cython/gtsam. that way it would be easier to keep track of what versions of GTSAM (built against what libraries, eg MKL vs no MKL) are being used at any given moment.

[Frank Dellaert]

I thought all these issues were resolved with setup.py ?

[matthew.broadway]

setup.py bundles up the gtsam.so but I didn't realize that there were other shared libraries that were needed. the install instructions only called for pointing the python path at the cython wrapper before, but actually libgtsam.so.4 also has to be in the path. I will try and see what happens if I manually copy libgtsam.so.4 over as well

[matthew.broadway]

to reproduce I did the following on a brand new Ubuntu 18.04 VM:

1. download gtsam && cd gtsam

2. mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=./install -DGTSAM_INSTALL_CYTHON_TOOLBOX=ON -DGTSAM_BUILD_EXAMPLES_ALWAYS=OFF -DGTSAM_BUILD_TIMING_ALWAYS=OFF ..

3. make

4. cd cython && python -c 'import gtsam' # works

5. cd .. && mv gtsam gtsam_moved

6. cd cython && python -c 'import gtsam' # import error

so without even running make install something has happened to the path so that python knows where to look for libgtsam. LD_LIBRARY_PATH is empty but doing the following works:

GT=/home/matthew/gtsam/build/gtsam-moved/ LD_LIBRARY_PATH="$GT:$GT/3rdparty/metis/libmetis:$GT/" python -c 'import gtsam'

copying all the required .so files to the cython/gtsam directory, it works if LD_LIBRARY_PATH is set to point to it before running python, but setting it from within python with os.environ doesn't work

copying libgtsam.so.4 and libmetis.so into cython/gtsam and adding the following to the top of gtsam/__init__.py allows for a completely portable installation:

from ctypes import cdll
import sys
import os

def lib_path(x):
    return os.path.join(os.path.abspath(os.path.dirname(__file__)), x)

cdll.LoadLibrary(lib_path('libmetis.so'))
cdll.LoadLibrary(lib_path('libgtsam.so.4'))

so this could be a configurable option from cmake? (probably want to bundle ceres the same way but I was just experimenting with what is absolutely required just to import the library)

[Chris Beall]

Iirc we build our dynamic libs with absolute rpaths, which is why libraries seem to point to original locations when you copy them.

The preferred way to fix this (instead of setting LD_LIBRARY_PATH) would be to build the cython gtsam.so with a relative rpath to libgtsam.so, thereby creating a truly relocatable package.

Recommended reading: https://en.wikipedia.org/wiki/Rpath https://amir.rachum.com/blog/2016/09/17/shared-libraries/#rpath-and-runpath https://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_3.html

[Frank Dellaert]

Leaving you guys to recommend something ;-) A question I have is: what does it take to put gtsam 4 on pipy so we can create docke instances that just "pip install gtsam4" ? Seems the answer here should be informed by this.

[Chris Beall]

Ah, if you want to distribute, then surely we should start with a static build. We also had to do a static build to distribute the Matlab toolbox!

Building statically would also be convenient way to sidestep the dynamic linking issues entirely when dealing with python. The flag is BUILD_SHARED_LIBS=OFF. I just tried static cython build on Ubuntu 16.04, but got this error:

[ 95%] Building CXX object cython/CMakeFiles/cythonize_gtsam.dir/gtsam/gtsam.cpp.o
[ 95%] Linking CXX shared module gtsam/gtsam.so
/usr/bin/ld: ../gtsam/libgtsam.a(Matrix.cpp.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
../gtsam/libgtsam.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
cython/CMakeFiles/cythonize_gtsam.dir/build.make:117: recipe for target 'cython/gtsam/gtsam.so' failed
make[2]: *** [cython/gtsam/gtsam.so] Error 1
CMakeFiles/Makefile2:24955: recipe for target 'cython/CMakeFiles/cythonize_gtsam.dir/all' failed
make[1]: *** [cython/CMakeFiles/cythonize_gtsam.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

Not sure why it's referring to libgtsam.so. Might be something hardcoded somewhere. Happy to help sort that out, but probably won't have time until the weekend.

[Frank Dellaert]

Excellent idea about sidestepping. José might be helpful as well to get cmake conquered. Or Varun. Yeah, also want to create docker instances that have just python gtsam4, all pre-built.

[Varun Agrawal]

The trick would be to have gtsam.so include everything and not rely on libgtsam.so. This would decouple the two and allow us to distribute a python based GTSAM with no problem.

It would be a bit of a challenge to get python and setup.py to figure all this out, but it is not impossible (e.g. in point is Pytorch).

[Frank Dellaert]

Cmake should figure this out, no? As Chris mentioned, our cmake files already support static libraries.

PS I don’t know what gtsam.so is, you probably mean gtsam.a ?

[Chris Beall]

To answer Frank's last question, in the dynamic case: For some reason the cythonized lib is called gtsam.so (to avoid name clash?) and it dynamically links libgtsam.so. Interestingly, it refers to the libgtsam.so in my build folder, and not the installed one. See below:

cbeall@workstation-1091:~/gtsam_prefix$ ls  -alh lib/libgtsam.so
lrwxrwxrwx 1 cbeall cbeall 13 Mar 20 19:18 lib/libgtsam.so -> libgtsam.so.4
cbeall@workstation-1091:~/gtsam_prefix$ ls  -alh cython/gtsam/gtsam.so 
-rw-r--r-- 1 cbeall cbeall 8.3M Mar 20 19:13 cython/gtsam/gtsam.so
cbeall@workstation-1091:~/gtsam_prefix$ ldd cython/gtsam/gtsam.so 
    linux-vdso.so.1 =>  (0x00007ffd3cb1d000)
    libgtsam.so.4 => /home/cbeall/git/gtsam/build/gtsam/libgtsam.so.4 (0x00007fb919aeb000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb9198ce000)
    libboost_system.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0 (0x00007fb9196ca000)
    libtbb.so.2 => /usr/lib/x86_64-linux-gnu/libtbb.so.2 (0x00007fb91948d000)
    libtbbmalloc.so.2 => /usr/lib/x86_64-linux-gnu/libtbbmalloc.so.2 (0x00007fb91924f000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fb918ecd000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb918bc4000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb9189ae000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb9185e4000)
    libboost_serialization.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.58.0 (0x00007fb918383000)
    libboost_filesystem.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0 (0x00007fb91816b000)
    libboost_timer.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_timer.so.1.58.0 (0x00007fb917f66000)
    libmetis.so => /home/cbeall/git/gtsam/build/gtsam/3rdparty/metis/libmetis/libmetis.so (0x00007fb917cf5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fb91a994000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb917af1000)
    libboost_chrono.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.58.0 (0x00007fb9178e9000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb9176e1000)

In any case, we should focus on making this work with static build, but dynamic build should be workable, too. Looks like we have some things to fix for both scenarios.

[Frank Dellaert]

gtsam 5 feature request