GSL/batch compilation issue

Issue #30 resolved
Alastair Pharo
created an issue

Not sure if this is to do with the GSL or what (I have only tried with pure-gsl), but here is what I am seeing on Arch Linux with Pure 0.62 when attempting to batch-compile some code that uses pure-gsl. First, the example code, in a file called test.pure:

using system;
using gsl;

(puts.str) $ {1,2,3}*{1;2;3};

I can run this fine like so:

$ pure test.pure  
{14}

However, batch compilation gives a strange linker error:

$ pure -v64 -c test.pure -o test 
{14}
opt -f -std-compile-opts test.bc | llc -filetype=obj -o test.o
g++ -o test /usr/lib/pure/pure_main.o test.o -Wl,--no-as-needed /usr/lib/pure/gsl.so -L/usr/lib -lpure
/usr/bin/ld: test.o: undefined reference to symbol 'gsl_matrix_int_add_constant'
/usr/lib/libgsl.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

I can get it to work if I add -lgsl:1

$ pure -v64 -c test.pure -o test.o
{14}                                 
opt -f -std-compile-opts test.o.bc | llc -filetype=obj -o test.o
Link with: g++ test.o -Wl,--no-as-needed /usr/lib/pure/gsl.so -L/usr/lib -lpure
$ g++ -o test /usr/lib/pure/pure_main.o test.o -Wl,--no-as-needed /usr/lib/pure/gsl.so -L/usr/lib
 -lpure -lgsl
$ ./test
{14}

So I suppose somehow ld is not finding libgsl.so via gsl.so?


  1. As an aside, the -v64 advice about how to link seems incorrect, as it does not include pure_main.o

Comments (18)

  1. Alastair Pharo reporter

    Ah, actually this can also be fixed by adding --copy-dt-needed-entries to the linker flags. Now that I think of it, I believe this was an issue some time ago on the mailing list.

    $ g++ -o test /usr/lib/pure/pure_main.o test.o -Wl,--copy-dt-needed-entries /usr/lib/pure/gsl.so -L/usr/lib -lpure
    $ ./test
    {14}
    

    In that example I removed --no-as-needed, but it works if that flag is present as well.

  2. Alastair Pharo reporter

    Another idea I had was to explicitly add using "lib:libgsl" to gsl/common.pure, i.e. making it explicit that many of the symbols referenced in pure-gsl are not actually in gsl.so. However this doesn't seem to work in batch compilation either. First, the contents of my modified common.pure:

    /* Definitions and declarations common to all pure-gsl modules. This loads the
       C module and performs some necessary initializations. */
    
    // Force libgsl to be loaded on Windows.
    #! --if *-mingw32
    using "lib:libgsl-0";
    #! --endif
    
    using "lib:libgsl";
    using "lib:gsl";
    
    namespace __C;
    private extern void pure_gsl_init();
    
    pure_gsl_init();
    

    I can run pure libs like before no problem:

    $ pure test.pure 
    {14}
    

    But, batch compilation fails because it doesn't know where libgsl.so is:

    $ pure -v64 -c test.pure -o test   
    {14}
    opt -f -std-compile-opts test.bc | llc -filetype=obj -o test.o
    g++ -o test /usr/lib/pure/pure_main.o test.o -Wl,--no-as-needed libgsl.so /usr/lib/pure/gsl.so -L/usr/lib -lpure
    g++: error: libgsl.so: No such file or directory
    

    If I alter the libgsl.so part to include the path, then this will work again (I guess this is equvalent to using -lgsl).

  3. Albert Graef

    Hi Alastair,

    Thanks for investigating this. The problem with all these approaches is that they're all highly platform-dependent. Neither Pure not the C compiler can always know about all libraries a given module needs to be linked against, so you often have to go through the usual incantations that you'd need when compiling a C application using the same library.

    That said, anything that we can do to make this process simpler at least on some systems is A Good Thing IMHO, so I'll have a look at your proposals asap. Thanks for the report!

  4. Albert Graef

    Ah, actually this can also be fixed by adding --copy-dt-needed-entries to the linker flags.

    Unfortunately, this doesn't work on the Mac. It also won't work on Linux systems which have the gold linker as the default ld. So there's really no easy way to make that work, even across different Linux systems, much less on other Un*x systems.

    If I alter the libgsl.so part to include the path, then this will work again (I guess this is equvalent to using -lgsl).

    In general, the Pure batch compiler really has no way of knowing where the library can be found; that's up to the dynamic linker of the host system which may search any number of special locations unknown to Pure. So I'm afraid that this isn't a solution either. I also don't like the idea of having to find an ad-hoc solution like this for every Pure module out there.

    Ok, so I think this is largely just me not understanding that -lgsl should be part of my calls to pure -c. So this is user confusion rather than a bug!

    If anything it's a bug in the current generation of linkers, which don't support this kind of recursive linking for whatever reason. So Pure's batch compiler isn't any worse than the other compilers out there.

    I recall that this used to work on older Linux systems, but that kind of behavior never really worked across different platforms. So I guess that we'll just have to live with the inconvenience for now, until a new linker generation comes along which finally gets rid of this annoyance. But I wouldn't hold my breath for it. ;-)

  5. Alastair Pharo reporter

    Diverging from the above,but still somewhat related ... re the using "lib:libgsl"; issue -- doesn't this mean that any time a pure program directly links to a library outside of PURE_LIBRARY and friends that batch compilation won't work? For instance:

    using "system";
    using "lib:libz";
    extern char* zlibVersion();
    puts zlibVersion;
    

    (I love by the way that I could just pick any random library, and execute a function from it!)

    $ pure -w z.pure    
    1.2.8
    $ pure -c z.pure -o z -v64            
    1.2.8
    opt -f -std-compile-opts z.bc | llc -filetype=obj -o z.o
    g++ -o z /usr/lib/pure/pure_main.o z.o -Wl,--no-as-needed libz.so -L/usr/lib -lpure
    g++: error: libz.so: No such file or directory
    $ pure -c z.pure -o z -v64 -lz
    1.2.8
    opt -f -std-compile-opts z.bc | llc -filetype=obj -o z.o
    g++ -o z /usr/lib/pure/pure_main.o z.o -Wl,--no-as-needed libz.so -lz -L/usr/lib -lpure
    g++: error: libz.so: No such file or directory
    

    Perhaps where a library is in using "lib:lib<whatever>" format, and pure is not able to find the library in PURE_LIBRARY the batch compiler could write -l<whatever> instead of lib<whatever>.so? i.e. assume that if the library is not somewhere that pure knows about, it must be somewhere that ld knows.

    The other solution would be for the user to add their system library location to PURE_LIBRARY, so that the batch compiler knows where to look:

    $ PURE_LIBRARY=/usr/lib pure -c z.pure -o z -v64
    1.2.8
    opt -f -std-compile-opts z.bc | llc -filetype=obj -o z.o
    g++ -o z /usr/lib/pure/pure_main.o z.o -Wl,--no-as-needed /usr/lib/libz.so -L/usr/lib -lpure
    $ ./z
    1.2.8
    
  6. Alastair Pharo reporter

    Actually (and sorry if this is too much spammage), I'm not sure how universal this is, but gcc and clang allow one to write -l: to tell the linker to search for a file name, so the following works:

    $ pure -c z.pure -o z.o -v64             
    1.2.8
    opt -f -std-compile-opts z.o.bc | llc -filetype=obj -o z.o
    Link with: g++ z.o -Wl,--no-as-needed libz.so -L/usr/lib -lpure
    $ g++ -o z z.o /usr/lib/pure/pure_main.o -Wl,--no-as-needed -l:libz.so -L/usr/lib -lpure
    $ ./z
    1.2.8
    $ clang++ -o z z.o /usr/lib/pure/pure_main.o -Wl,--no-as-needed -l:libz.so -L/usr/lib -lpure
    $ ./z
    1.2.8
    
  7. Albert Graef

    Diverging from the above,but still somewhat related ... re the using "lib:libgsl"; issue -- doesn't this mean that any time a pure program directly links to a library outside of PURE_LIBRARY and friends that batch compilation won't work?

    I'm afraid that this is true. But, as you pointed out, you can set PURE_LIBRARY (it should also be possible to add -L options to the compile command instead) to make the batch compiler find things.

    (I love by the way that I could just pick any random library, and execute a function from it!)

    Yeah, isn't that nice. :)

    Perhaps where a library is in using "lib:lib<whatever>" format, and pure is not able to find the library in PURE_LIBRARY the batch compiler could write -l<whatever> instead of lib<whatever>.so? i.e. assume that if the library is not somewhere that pure knows about, it must be somewhere that ld knows.

    That's an awful lot of hidden magic going on behind the scenes. I'm not sure I like that.

    I like the -l: trick, though, I didn't know that this was possible. I'd have to check how portable that is, however.

  8. Albert Graef

    That's an awful lot of hidden magic going on behind the scenes. I'm not sure I like that. I like the -l: trick, though, I didn't know that this was possible. I'd have to check how portable that is, however.

    On second thought, both solutions aren't really that much different. They both involve the same kind of test to see whether some shared library file actually exists on Pure's library search path. Which at any rate is much less hidden magic than what the dynamic linker does. The second solution is just a little tidier and easier to implement.

    Thanks, that's really a clever solution and it might just work. It doesn't really solve the original issue, but mitigates it at least. I'll give it a go.

  9. Albert Graef

    Where did you find the -l:libz.so syntax? I don't see this documented anywhere. The only reference that I could find anywhere is in the HP-UX linker documentation, which doesn't speak for its portability...

  10. Albert Graef

    Ok, it seems that at least newer versions of GNU ld support this. I couldn't find out which versions exactly, so I added a configure check for the -l: option. If the linker doesn't support it, the batch compiler will still try to use a standard -l option, as you suggested earlier.

    Committed and pushed in rev. e56985ea3669. Please let me know how that works for you.

  11. Alastair Pharo reporter

    Yep, works for me with either bfd or gold. Thanks for you efforts on this.

    For pure-gsl, there is already some check under mingw32, so if you think the idea has merit, I could make a pull request adding another check that adds using "lib:libgsl" on linux systems. Otherwise I think this is already solved in that a user can just write -lgsl like we already discussed. In that case I could have a go at a pull request that updates the pure-gsl documentation to indicate that the extra option may be needed.

  12. Albert Graef

    Well, it doesn't work on the Mac yet. Apparently clang accepts the -l: option, so the configure check passes, but the linker then refuses the option anyway. So I first need to fix the configure check.

  13. Log in to comment