libR not found on OS X with Homebrew R installation

Issue #194 resolved
Tom Loredo
created an issue

RPy2 fails to build and install (via pip) on OS X (10.8.5) against the current Homebrew R installation (and using brew's python); it cannot find libR:

ld: library not found for -lR clang: error: linker command failed with exit code 1 (use -v to see invocation)

As best as I can determine, this is due to problems with both Homebrew and RPy2.

Homebrew builds R as a framework; by default the framework build process does not produce a shared library and Homebrew does not set the required flag. Also, Homebrew does not expose an option to build the library.

According to this MacPorts post:

https://www.biostars.org/p/2859/

the R-Project binary build of R for OS X does build libR, so this should probably the default in Homebrew, or at least an exposed option. I've patched my brew r.rb formula to include the "--enable-R-shlib" flag, which builds libR.dylib within the R framework. But RPy2's setup.py still would not discover libR. By further patching the formula to link libR under brew's prefix (/usr/local/lib by default), RPy2 builds successfully. But this happened despite not linking the headers, which seemed fishy to me.

Pip's output suggests RPy2 is not consistently discovering the library location:

Configuration for R as a library:
  include_dirs: ('/usr/local/Cellar/r/3.1.0/R.framework/Resources/include',)
  libraries: ('lzma', 'iconv', 'icucore', 'm')
  library_dirs: ()
  extra_link_args: ('-F/usr/local/Cellar/r/3.1.0/R.framework/..', '-framework R')
 # OSX-specific (included in extra_link_args)
  framework_dirs: ('-F/usr/local/Cellar/r/3.1.0/R.framework/..',)
  frameworks: ('-framework R',)

It finds the headers within the framework, but library_dirs is empty; I think it should point to within the framework.

I think RPy2 gets the headers from this R query:

"/usr/local/Cellar/r/3.1.0/R.framework/Resources/bin/R" CMD config --cppflags

which returns:

-I/usr/local/Cellar/r/3.1.0/R.framework/Resources/include

Perhaps it is expecting to get the library path from the linker query:

"/usr/local/Cellar/r/3.1.0/R.framework/Resources/bin/R" CMD config --ldflags

But this returns:

-F/usr/local/Cellar/r/3.1.0/R.framework/.. -framework R -llzma -licucore -lm -liconv

which does not point to libR, presumably because the R runtime does not use the shared lib.

In setup.py, getRinterface_ext starts with:

def getRinterface_ext():
    #r_libs = [os.path.join(RHOME, 'lib'), os.path.join(RHOME, 'modules')]
    r_libs = []

If RHOME were set to the value of "R RHOME" and the commented line were kept, I suspect the build would work. As it stands, I think getRinterface_ext will always have an empty r_libs. I presume the commented line is commented for a reason, so I haven't tried to fix this myself.

I will submit an issue to Homebrew about patching the R formula once I'm clear what the proper fix is on both the RPy2 and Homebrew sides.

Comments (36)

  1. Laurent Gautier

    Homebrew builds R as a framework; by default the framework build process does not produce a shared library and Homebrew does not set the required flag. Also, Homebrew does not expose an option to build the library.

    I thought that it was the case. Could homebrew this be a temporary oversight in homebrew ?

  2. Laurent Gautier

    R CMD config --cppflags and R CMD config --ldflags are used to extract all info, as the promise from the help seems to be that this is all I needed (see below). This is true until proven wrong, do not hesitate to fork and propose improvements through a pull request.

    $ R CMD config
    Usage: R CMD config [options] [VAR]
    
    Get the value of a basic R configure variable VAR which must be among
    those listed in the 'Variables' section below, or the header and
    library flags necessary for linking against R.
    
    Options:
      -h, --help            print short help message and exit
      -v, --version         print version info and exit
          --cppflags        print pre-processor flags required to compile
                a C/C++ file using R as a library
          --ldflags         print linker flags needed for linking a front-end
                            against the R library
          --no-user-files  ignore customization files under ~/.R
          --no-site-files  ignore site customization files under R_HOME/etc
    
  3. Laurent Gautier
    def getRinterface_ext():
        #r_libs = [os.path.join(RHOME, 'lib'), os.path.join(RHOME, 'modules')]
        r_libs = []
    

    This was commented out with eab2bfdb8c05 (in 2009), presumably because it was breaking with some configurations.

    The idea was to add command-line options. Here:

    python setup.py build --r-home-lib=<path to Rlib.so>
    python setup.py install
    

    I have no idea this is still working (no unit tests for this).

  4. Tom Loredo reporter

    Laurent, I've looked further into this. The compilation command pip is reporting that fails is:

    cc -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.8-x86_64-2.7/./rpy/rinterface/_rinterface.o -L/usr/local/Cellar/r/3.1.0/R.framework/Resources/modules -lR -llzma -liconv -licucore -lm -o build/lib.macosx-10.8-x86_64-2.7/rpy2/rinterface/_rinterface.so
    

    Note the library search path is:

    -L/usr/local/Cellar/r/3.1.0/R.framework/Resources/modules
    

    Recall the "R CMD config --ldflags" output is:

    -F/usr/local/Cellar/r/3.1.0/R.framework/.. -framework R -llzma -licucore -lm -liconv
    

    So it looks like setup.py is adding "Resources/modules" to the -F framework path (e.g., line 142 of setup.py). However, the libR.dylib is placed in the R.framework/Libraries directory (a link to Versions/Current/Resources/lib). The Resources/modules directory holds several .so libs, but not libR.

    This could be fixed by changing, e.g., line 142 in setup.py:

                self.library_dirs.extend([os.path.join(r_home, 'modules'), ])
    

    Changing "modules" to "lib" should work.

    However, it's not clear to me that R is following standard conventions in this regard. Do you know of documentation specifying where such a shared library should live? All I've located so far is this section of the clang docs:

    http://clang.llvm.org/docs/Modules.html#module-declaration

    This suggests the library should be at the top level of the framework, and neither in a modules or Libraries subdirectory.

    That said, old versions of RPy apparently looked in the location R currently uses:

    http://ehc.ac/p/rpy/mailman/message/3383960/

  5. Tom Loredo reporter

    Evidently OS X distinguishes between shared libraries (.dylib files) and loadable modules (.bundle or .so files):

    http://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm

    You can link against shared libraries, but not modules; modules are meant to be dynamically loaded at runtime.

    Python's libpython shared library lives here in Homebrew:

    /usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib
    

    i.e., it is in a "lib" directory, not "modules", as is true for libR.

    Thus it seems RPy2 really should be looking in lib, not modules. As noted previously, it did previously look in lib. Can you recall why it was switched to modules and identify a case where that works for linking against libR.dylib?

  6. Laurent Gautier

    Can you recall why it was switched to modules and identify a case where that works for linking against libR.dylib?

    Unfortunately I don't, and I currently do not have a Mac to develop rpy2 on to try out things. This is hinting toward the need for definitions specific to OS X.

    If you fork in order to propose a patch, please do it against the 2.4.0-dev (I can backport to 2.3.x):

    hg update version_2.4.x
    
  7. Liang-Bo Wang

    Hi I was able to compile and pass all unit test by supplying correct R lib path.

    env LDFLAGS="-L/usr/local/Cellar/r/3.0.3/R.framework/Versions/3.0/Resources/lib -L/usr/local/opt/openblas/lib" \
        python setup.py build
    python setup.py install
    python -m 'rpy2.tests'
    # Ran 374 tests in 4.141s
    # OK (skipped=2, expected failures=1)
    

    Enviroment:

    • OSX 10.9.2
    • Brewed Python version: 3.4.0
    • Brewed R version: 3.0.3
      • Through cmd brew install r --with-openblas
    • rpy2 version: 5d459ff40929, branch: version_2.4.x
  8. Laurent Gautier

    I have cleaned a bit setup.py (commit f692ab7934e5) and line 142

     self.library_dirs.extend([os.path.join(r_home, 'modules'), ])
    

    is now gone.

    I am also reading from the discussion on the R bug tracker that the issue might be with the way Homebrew is building R. Is it the case ? Can this issue be closed ?

  9. Jake Swanson

    I just came to comment and thank Liang Bo Wang and say that his command works for R 3.1 as well (just changing the directories to match R 3.1).

    env LDFLAGS="-L/usr/local/Cellar/r/3.1.0/R.framework/Versions/3.1/Resources/lib -L/usr/local/opt/openblas/lib" \
        python setup.py build
    python setup.py install
    python -m 'rpy2.tests'
    
  10. Laurent Gautier

    Is anyone on the watchers' list for this issue following closely the homebrew project / the formula for R on homebrew and able to comment on whether the info in the link above is correct ?

  11. Liang-Bo Wang

    I am not that familiar with how to link OSX framework, but I tried downloading the R source and compiled Rtest. Followed the instruction given by upstream issue and it works

    # in R-3.1.1 source tests/Embedding
    clang -c Rtest.c embeddedRCall.c `R CMD config --cppflags`
    clang -o Rtest Rtest.o embeddedRCall.o `R CMD config --ldflags`
    env R_HOME=`R RHOME` ./Rtest --silent
    # [1] 1 2 3 4 5 6 7 8 9 10
    

    Maybe you need to link to the R framework given by R CMD config --ldflags.

  12. Laurent Gautier

    I dived in setup.py and it was quite a mess (accumulation of myself initially struggling with the Python packaging systems, them evolving, and the way R communicating information about how to compile extension evolving as well).

    This is now largely rewritten and I would appreciate if the homebrew users (and others) could try:

    pip install --allow-external=rpy2 https://bitbucket.org/lgautier/rpy2/downloads/rpy2-2.5.0dev.tar.gz
    
  13. Liang-Bo Wang

    I can compile and pass most of the tests while I need to modify the setup.py a bit. First since args.L can be None when linking to framework on brewed R, skipping empty one is needed,

    if args.L:
        library_dirs.extend(args.L)
    

    On the other hand, using R framework implies not linking to -lR, so in both extension I needed to comment out the line linking to R library.

    rinterface_ext = Extension(
        # libraries = ['R', ]
    )
    

    But on other platforms that is the common way to link, so maybe some extra detections for platform or input by R CMD config should be added.

    Most of the tests failed on my machines seems not related with this issue. I get some AttributeError: 'numpy.ndarray' object has no attribute 'typeof' errors.

  14. Laurent Gautier

    @Liang-Bo Wang : I see. Thanks for trying out and pinpointing the issue on OS X.

    I also need setup.py to work on non OS X / non-framworks systems as well, so I made the change a little more general (see e8cba96).

    I uploaded a build at the same location and the command to try out it still:

    pip install --allow-external=rpy2 https://bitbucket.org/lgautier/rpy2/downloads/rpy2-2.5.0dev.tar.gz
    

    I'll close this as soon as I hear that it is working.

  15. Liang-Bo Wang

    @Laurent Gautier I still get a -lR when compiling

    clang -bundle -undefined dynamic_lookup -L/usr/local/lib -L/usr/local/opt/sqlite/lib build/temp.macosx-10.9-x86_64-3.4/./rpy/rinterface/_rinterface.o -lR -o build/lib.macosx-10.9-x86_64-3.4/rpy2/rinterface/_rinterface.so -F/usr/local/Cellar/r/3.1.1/R.framework/.. -framework R
    
    ld: library not found for -lR
    

    I then replaced libraries = ['R', ] with libraries = libraries to pass the previous parsed args.l and I managed to pass tests on OSX.

    While on Debian(jessie) there is another problem. I found that not all -lxxx is captured by argparse. Here is my result,

    $ R CMD config --ldflags
    -Wl,--export-dynamic -fopenmp  -L/usr/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm
    $ python setup.py install
    # print out the result of argparse
    Namespace(I=['/usr/share/R/include'], L=None, l=None) []
    Namespace(I=None, L=['/usr/lib/R/lib'], l=['m']) ['-Wl,--export-dynamic', '-fopenmp']
    # what libraries and library_dirs have
    ['m'] ['/usr/lib/R/lib']
    

    So on Debain rpy2 tests failed because not link with -lR.

  16. Liang-Bo Wang
    $ R CMD config --ldflags
      -F/usr/local/Cellar/r/3.1.1/R.framework/.. -framework R -llzma -licucore -lm -liconv
    $ python setup.py install
    # argparse output
    Namespace(I=['/usr/local/Cellar/r/3.1.1/R.framework/Resources/include'], L=None, l=None) []
    Namespace(I=None, L=None, l=['lzma', 'icucore', 'm', 'iconv']) ['-F/usr/local/Cellar/r/3.1.1/R.framework/..', '-framework', 'R']
    

    Looks good

  17. Liang-Bo Wang

    Currently will fail on OSX because libraries is explicitly set as libraries = ['R',]. It should use the arguments given by argparse. After changing it, compilation looks good and passed tests on both Debian and OSX.

    EDIT: I have submitted a pull request. EDIT2: Passed tests (except for numpy ndarray no typeof error).

  18. john wright

    Found this thread searching for typeof errors. I get four errors when running from 2.5.0dev (hg parent: 2294:9e856a783d4d tip) with R 3.1.1 on OSX Lion. Three of the are 'typeof' errors and one is in test_rmagic.TestRmagic, key error: 'result'

    Hope that helps. I'm wanting to use this in an python notebook but the typeof errors limit the functionality quite a bit. thanks

  19. Laurent Gautier

    @jcwright . Thanks for the report. It is easier to have one issue per report and associated discussion. Please open a new issue, and include as many information's about the issue as possible.

  20. Nikhil S

    My apologies, the installation does work when I apply Liang Bo Wang's setup.py changes. I thought these changes had already been merged into the dev build.

    Original post (ignore):

    Hello. I still get ld: library not found for -lR when trying to do pip install rpy2, or pip install --allow-external=rpy2 https://bitbucket.org/lgautier/rpy2/downloads/rpy2-2.5.0dev.tar.gz.

    Environment:

    • OSX 10.9.2
    • Brewed Python version: 2.7.8
    • Brewed R version: 3.1.1
  21. randy3k

    My R was installed via homebrew, and I am on OS X 10.10. I have to do two things in order to install properly rpy2.

    1. brew edit r and add "--enable-R-shlib" in configure arguments.

    2. export LDFLAGS="-L/usr/local/opt/r/R.framework/Resources/lib/" before installing rpy2

  22. Julian de Ruiter

    Installing the 2.5.0 release works fine for me (default homebrew R install).

    pip install --allow-external=rpy2 https://bitbucket.org/lgautier/rpy2/get/RELEASE_2_5_0.tar.gz
    python -m 'rpy2.tests'
    
  23. Log in to comment