Cannot pip install rpy2 with latest R 3.4.0 (unsupported option '-fopenmp') on macOS

Issue #403 open
Clemens Brunner
created an issue

Installing rpy2 with the latest R 3.4.0 doesn't work on macOS. Apparently, R 3.4.0 was built using the '-fopenmp' flag with Clang 4.0.0, which is not supplied by Apple. Therefore, pip install rpy2 gives the following error:

    gcc -bundle -undefined dynamic_lookup -L/usr/local/anaconda/lib -L/usr/local/anaconda/lib -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/./rpy/rinterface/_rinterface.o -L/usr/local/lib -L/usr/local/anaconda/lib -Lbuild/temp.macosx-10.7-x86_64-3.6 -L/usr/local/lib -lpcre -llzma -lbz2 -lz -licucore -lm -liconv -lr_utils -o build/lib.macosx-10.7-x86_64-3.6/rpy2/rinterface/_rinterface.cpython-36m-darwin.so -fopenmp -F/Library/Frameworks/R.framework/.. -framework R
    clang: error: unsupported option '-fopenmp'
    error: command 'gcc' failed with exit status 1

Everything worked fine with R 3.3.3 (which was built with the system Clang).

Comments (62)

  1. Laurent Gautier

    Disclaimer: I don't have a dev OS X to test anything of this.

    Could this is be because R was compiled with gcc rather than clang ? Did you try with gcc (easiest might be to install it with homebrew)

  2. Laurent Gautier
    • changed status to open

    An alternative might be to work out precompiled binaries for OS X but since I currently do not have access to a dev environment with OS X for rpy2, and since this would likely open up additional complexity because of R versions, this would have to be contributed by someone.

  3. Clemens Brunner reporter

    @pkulkarni2110 as a workaround, you can continue to use R 3.3.3, which is the last version that works with rpy2. @Laurent Gautier I have a Mac, but setting up the tools that were used to build R 3.4.0 could be a PITA. Unless I could just use the versions in Homebrew - I saw that they have LLVM version 4.0.0, but gcc is at 7.1.0 (and gcc@6 is 6.3.0 and not 6.1.0), so not sure it will be that easy. IMO it was quite an unfortunate decision from the R devs not to use Apple provided tools.

  4. Clemens Brunner reporter

    @Laurent Gautier either way this situation is not ideal, because it might entail having two pip packages for macOS depending on which R version you want to connect to. I don't think that this is even possible. Unless of course rpy2 built with the newer compiler versions also works with older R versions built with Apple-provided stuff.

  5. pkulkarni2110

    I installed R ver 3.3. I installed XQuartz as it was mentioned in the R webpage to install in macOs. I cannot certainly say it was helped to resolve the issue in anyway. Yes it is not ideal situation but it did work for temporarily.

  6. Clemens Brunner reporter

    Installing XQuartz is not necessary. I agree that this is just a temporary workaround - all future versions of R will be built with the newer tools, so eventually rpy2 will need to support it. MacOS High Sierra will likely ship with these updated build tools, so when rpy2 is built for this platform everything should hopefully work with R 3.4.0 and higher. Still, the question remains how rpy2 will handle older macOS versions. I guess we'd have to try if building with the newer tools produces a package that works both with R 3.4.0 as well as older versions such as R 3.3.3. I can try to build it and see if it works when I have time.

  7. Laurent Gautier

    @Clemens Brunner : I might be missing something here. How is this a support issue with rpy2 rather than an inconvenience of the R binaries on OS X ? rpy2 is getting its compilation flags (-fopenmp, presumably among other flags, here) from the installed R binary.

  8. Clemens Brunner reporter

    Because rpy2 doesn't work with R 3.4.0 or later, since setup.py requires a clang version that supports the -fopenmp flag. Since rpy2 gets the compiler flags from the R binary, is it possible to disable the -fopenmp flag if it is found? This would probably solve the issue.

  9. Clemens Brunner reporter

    Besides that, it is quite a bit of work to install the latest clang. I just tried it using Homebrew - and rpy2 installs fine! Here's what I did:

    brew install --with-toolchain llvm
    export PATH="/usr/local/opt/llvm/bin:$PATH"
    export LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"; pip3 install rpy2
    

    I'm not sure if we should expect rpy2 users on macOS to run through these steps (especially since the first step takes a long time because it compiles llvm from source).

  10. Laurent Gautier

    It might install, and may be even run, but that not something I'd like to hardcode in setup.py (which would mean support it). We can leave your workaround, someone may want to contribute "wheel" binaries for OS X, but otherwise building from source will require the toolchain the R binary itself was built with.

  11. Clemens Brunner reporter

    You mean the workaround using R < 3.4.0? Since I have the toolchain already setup I could provide a binary wheel for macOS. Two issues I see here:

    1. The binary wheel works only with R >= 3.4.0 - can pip determine which R version the user has installed and decide which wheel to install?
    2. I don't know how to create a binary wheel, maybe you could give me some instructions.
  12. Clemens Brunner reporter

    OK, but so far the source wheels worked and you only need a binary wheel for macOS using R >= 3.4.0 and all sorts of Python versions. The minimum should be Python 3.6 here, so this would be only one wheel if patch version levels are not important (i.e. Python 3.6 works like 3.6.1, similarly for R 3.4.0 and 3.4.1 etc). Let me know if you need anything from me to create this binary wheel.

  13. David Grunderson

    To piggyback of this issue, I have also been experiencing this same bug, though I am using R version 3.4 and Python version 2.7. would there be any way to get rpy2 to function with Python 2.7? I have been experiencing a great number of difficulties in trying to install rpy2, despite having tried pip, easy_install, and downloading the source tar. Despite my R being up to date and located in the PATH environmental variable, I have consistently received the same error:

      clang: error: unsupported option '-fopenmp'
        error: command 'cc' failed with exit status 1
    

    I have tried numerous approaches to solving this, including downloading XCode for OSX, as well as importing architecture flags, such as:

    env ARCHFLAGS="-arch i386 -arch x86_64" pip install rpy2
    

    I have also tried to installing the latest GCC compilers, but that made no change on the error yielded during rp2's installation process. Then I tried implementing commands suggested by Clemens Brunner above, and found that they worked perfectly, but only for Python 3.4. Is there any other approach you would recommend in order for rpy2 to function with Python 2.7?

  14. Clemens Brunner reporter

    Another much simpler option is to remove the -fopenmp linker flag (at least on macOS, I have no idea if this flag works on Windows). I just added the following lines to setup.py after line 272:

    if "-fopenmp" in unknown:  # remove linker argument
        unknown.remove("-fopenmp")
    

    Lo and behold, everything works just fine with the latest R 3.4.1 (all tests pass). But of course I'm not using OpenMP.

  15. Dan Li

    I have followed @ Clemens Brunner and run the following three command. I installed llvm, set the environment in .bash_profile, and source it to activate the change. I installed rpy2 in python3. But I still got the same error.

    brew install --with-toolchain llvm
    export PATH="/usr/local/opt/llvm/bin:$PATH"
    export LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"; pip3 install rpy2
    

    I also tried to install OpenMPI in http://thecoatlessprofessor.com/programming/openmp-in-r-on-os-x/#after-3-4-0. But it didn't help.

    It is so annoying. Can anyone give me some help?

  16. Clemens Brunner reporter

    @Dan Li try uninstalling and then re-installing by ignoring the cache:

    pip3 uninstall rpy2
    export PATH="/usr/local/opt/llvm/bin:$PATH"
    export LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"
    pip3 install --no-cache-dir rpy2
    
  17. Dan Li

    Sorry, I still get the error.

        clang: error: unsupported option '-fopenmp'
        error: command 'gcc' failed with exit status 1
    
        ----------------------------------------
    Command "/Users/danli/anaconda/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/gn/y3k01wts6yvg9z_m0zlqnllm0000gn/T/pip-build-viF4iB/rpy2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/gn/y3k01wts6yvg9z_m0zlqnllm0000gn/T/pip-XQWVG4-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/gn/y3k01wts6yvg9z_m0zlqnllm0000gn/T/pip-build-viF4iB/rpy2/
    
  18. Dan Li

    Hi, Clemens Brunner. Here is the info.

    DanmatoMacBook-Pro:paper danli$ which clang
    /usr/local/opt/llvm/bin/clang
    DanmatoMacBook-Pro:paper danli$ clang --version
    clang version 4.0.1 (tags/RELEASE_401/final)
    Target: x86_64-apple-darwin16.7.0
    Thread model: posix
    InstalledDir: /usr/local/opt/llvm/bin
    
  19. Clemens Brunner reporter

    Somehow the build process still uses gcc instead of clang. I don't know why this worked for me out of the box, but you could either symlink gcc to clang (and g++ to clang++), or explicitly tell pip to use clang (this is actually where I'll stop, because I have no experience with distutils, but you could try to go from here or here). Alternatively, you could try the simpler option I mentioned above (add the two lines to setup.py and install via python3 setup.py install). Lastly, if you install the package directly without pip, you could also try python3 setup.py install --compiler=clang.

  20. Clemens Brunner reporter

    BTW, although Apple has updated clang in High Sierra, they have disabled the fopenmp option. So unfortunately this means that installing rpy2 on a Mac via pip still doesn't work out of the box (if the latest R version is installed).

  21. Sanjiv Das

    I have tried everything to install rpy2 on High Sierra running R3.4.1 and Python3.6 (anaconda). Btw, downgrading to R3.3.3 also does not work. I get the following error:

    unable to execute 'x86_64-apple-darwin13.4.0-ar': No such file or directory error: command 'x86_64-apple-darwin13.4.0-ar' failed with exit status 1

    I think its related to the same issue in this thread, but if not, and there is hope for a fix, please let me know. I ran into this problem only after upgrading OSX to High Sierra.

  22. zach
    xcode-select --install
    xcode-select: error: command line tools are already installed, use "Software Update" to install updates
    

    But brew install --with-toolchain llvm gives

    ==> cmake -G Unix Makefiles /private/tmp/llvm-20171023-25020-f0tmy8/llvm-5.0.0.s
    Last 15 lines from /Users/me/Library/Logs/Homebrew/llvm/01.cmake:
      command.
    
    
    CMake Warning at tools/xcode-toolchain/CMakeLists.txt:60 (message):
      Failed to detect the version of an installed copy of Xcode, falling back to
      highest supported version.  Set XCODE_VERSION to override.
    
    
    CMake Error at tools/xcode-toolchain/CMakeLists.txt:80 (message):
      Could not identify toolchain dir
    
    
    -- Configuring incomplete, errors occurred!
    See also "/tmp/llvm-20171023-25020-7zds3a/CMakeFiles/CMakeOutput.log".
    See also "/tmp/llvm-20171023-25020-7zds3a/CMakeFiles/CMakeError.log".
    
    READ THIS: https://docs.brew.sh/Troubleshooting.html
    
    These open issues may also help:
    faust. v2.1.0 (new formula) https://github.com/Homebrew/homebrew-core/pull/16724
    gdb: Automatic code signing https://github.com/Homebrew/homebrew-core/pull/19421
    cling 0.4 (new formula) https://github.com/Homebrew/homebrew-core/pull/18124
    infer 0.13.0 https://github.com/Homebrew/homebrew-core/pull/19646
    pypy crashes with dyld: lazy symbol binding failed: Symbol not found: _clock_gettime https://github.com/Homebrew/homebrew-core/issues/6949
    
  23. zach

    To clarify, this thread began with references to fopenmp but I'm not seeing this in any of my output. My current pip install rpy2 attempt ends with

        x86_64-apple-darwin13.4.0-ar rc build/temp.macosx-10.9-x86_64-3.6/libr_utils.a build/temp.macosx-10.9-x86_64-3.6/./rpy/rinterface/r_utils.o
        unable to execute 'x86_64-apple-darwin13.4.0-ar': No such file or directory
        error: command 'x86_64-apple-darwin13.4.0-ar' failed with exit status 1
    
  24. zach
    $ brew config
    HOMEBREW_VERSION: 1.3.6
    ORIGIN: https://github.com/Homebrew/brew
    HEAD: 270b752f5d9d218bfbed6fe85b6974fa653fb25f
    Last commit: 6 days ago
    Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
    Core tap HEAD: b7d1e926e9f2e3e950866b7e82ad1fbe34f1fb12
    Core tap last commit: 23 hours ago
    HOMEBREW_PREFIX: /usr/local
    HOMEBREW_REPOSITORY: /usr/local/Homebrew
    HOMEBREW_CELLAR: /usr/local/Cellar
    HOMEBREW_BOTTLE_DOMAIN: https://homebrew.bintray.com
    CPU: quad-core 64-bit broadwell
    Homebrew Ruby: 2.3.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.3/bin/ruby
    Clang: 9.0 build 900
    Git: 2.13.5 => /Library/Developer/CommandLineTools/usr/bin/git
    Perl: /usr/bin/perl
    Python: /Users/me/Applications/anaconda3/bin/python => /Users/me/Applications/anaconda3/bin/python3.6
    Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
    Java: N/A
    macOS: 10.12.6-x86_64
    Xcode: 9.0.1
    CLT: 9.0.1.0.1.1506734476
    X11: N/A
    
  25. Clemens Brunner reporter

    Looks good too. I have no idea why brew install --with-toolchain llvm fails on your machine. However, you could try to use the system clang and just remove the -fopenmp flag as I've mentioned here.

  26. zach

    OK. python setup.py install (after editing, which doesn't really seem to make a difference) gives

    $ python setup.py install
    R version 3.4.2 (2017-09-28) -- "Short Summer"
    /Library/Frameworks/R.framework/Resources/bin/R CMD config --ldflags
    /Library/Frameworks/R.framework/Resources/bin/R CMD config --cppflags
    Traceback (most recent call last):
      File "setup.py", line 357, in <module>
        ri_ext = getRinterface_ext()
      File "setup.py", line 272, in getRinterface_ext
        if "-fopenmp" in unknown:  # remove linker argument
    UnboundLocalError: local variable 'unknown' referenced before assignment
    Zachs-MBP:rpy2-2.9.0 me$ python setup.py install
    R version 3.4.2 (2017-09-28) -- "Short Summer"
    /Library/Frameworks/R.framework/Resources/bin/R CMD config --ldflags
    /Library/Frameworks/R.framework/Resources/bin/R CMD config --cppflags
    
        Compilation parameters for rpy2's C components:
            include_dirs    = ['/Library/Frameworks/R.framework/Resources/include']
            library_dirs    = ['/usr/local/lib']
            libraries       = ['pcre', 'lzma', 'bz2', 'z', 'icucore', 'm', 'iconv']
            extra_link_args = ['-fopenmp', '-F/Library/Frameworks/R.framework/..', '-framework', 'R']
    
    running install
    running bdist_egg
    running egg_info
    writing rpy2.egg-info/PKG-INFO
    writing dependency_links to rpy2.egg-info/dependency_links.txt
    writing requirements to rpy2.egg-info/requires.txt
    writing top-level names to rpy2.egg-info/top_level.txt
    R version 3.4.2 (2017-09-28) -- "Short Summer"
    reading manifest file 'rpy2.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*patch*' found anywhere in distribution
    warning: no previously-included files matching '*diff*' found anywhere in distribution
    warning: no previously-included files matching '.hg' found anywhere in distribution
    warning: no files found matching 'MANIFEST'
    warning: no files found matching 'README'
    no previously-included directories found matching 'dist'
    warning: no files found matching 'doc/source/rpy2_logo.png'
    writing manifest file 'rpy2.egg-info/SOURCES.txt'
    installing library code to build/bdist.macosx-10.9-x86_64/egg
    running build_clib
    building 'r_utils' library
    x86_64-apple-darwin13.4.0-ar rc build/temp.macosx-10.9-x86_64-3.6/libr_utils.a build/temp.macosx-10.9-x86_64-3.6/./rpy/rinterface/r_utils.o
    unable to execute 'x86_64-apple-darwin13.4.0-ar': No such file or directory
    error: command 'x86_64-apple-darwin13.4.0-ar' failed with exit status 1
    
  27. Clemens Brunner reporter

    Weird. I tried it on my Mac and the procedure works (with Homebrew llvm and with the system clang and modified setup.py). Let's see if the Homebrew devs can help.

  28. Sanjiv Das

    I was finally able to install as follows (Mac OS High Sierra + R 3.4.0).

    (a) Install the latest version of the Java SDK: jdk-8u151-nb-8_2-macosx-x64.dmg

    (b) Then use conda as follows: conda install rpy2

    Hope this helps everyone.

  29. Clemens Brunner reporter

    You don't need to install Java SDK, this has nothing to do with this issue. Also, conda install rpy2 has always worked, it's pip install rpy2 which causes trouble.

  30. Sanjiv Das

    Thanks Clemens, I was also not sure why I needed to upgrade JDK, but it came up when I used conda. There may be some interaction in my set up. Also conda install rpy2 does not always work on Mac environments (I have tried several), but seems to always work on all my Ubuntu setups. Not sure why, but I am relieved, and happy to have my Python and R systems talking happily to each other again on my main Macbook Pro. Appreciate all your help with this on this forum.

  31. Tomás Díaz

    I was having the same issue and followed these instructions but installation (pip install rpy2) failed with:

        ld: in '/usr/local/opt/llvm/lib/libunwind.dylib', file was built for x86_64 which is not the architecture being linked (i386): /usr/local/opt/llvm/lib/libunwind.dylib for architecture i386
        clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
        error: command '/usr/local/opt/llvm/bin/clang' failed with exit status 1
    

    I ran it again with : env ARCHFLAGS="-arch x86_64" pip install rpy2

    and then it worked.

  32. Yuriy Gorishniy

    Hi! I have faced the problem with openmp on Sierra and have solved it.
    1. Setting the environment variable CC in .bash_profile or aliasing gcc to gcc-7 (installed by brew) doesn't work
    2. The solution proposed by @jzweig works perfectly, namely:

    env CC=/usr/local/Cellar/gcc/7.2.0/bin/gcc-7 pip install rpy2
    
  33. Clemens Brunner reporter

    Interesting to see that almost everyone needs to do something different in order to install rpy2. Bottom line is that rpy2 currently cannot be installed on Macs out of the box via pip. A binary wheel would solve all these issues. I'm still willing to contribute a Mac binary, @Laurent Gautier let me know when you are ready.

  34. Log in to comment