-ffunction-sections/-fdata-sections for phobos/druntime

dsimcha avatardsimcha created an issue

Shouldn't Phobos and druntime be built with -ffunction-sections and -fdata-sections so that --gc-sections can be used at link time for a smaller binary?

Comments (12)

  1. art_

    I was curious about the impact this would make, so i ran i few quick tests [1]; building a small and simple program, which uses only a few modules.

    The resulting executable sizes (identical builds, differing only in gc-sections options):

    stock libphobos, no --gc-sections:  755104 bytes
    stock libphobos plus gc-sections:  634688 bytes
    libphobos built with -ffunction-sections -fdata-sections plus gc-sections:  328128 bytes
    

    So it makes a huge difference, more than cutting the binary size in half.

    Hopefully it won't confuse the GC, and i will not run into yet another ICE issue forcing me to revert this. :)

    [1] basically:

    [...]
      make -j9 2>&1 | tee build.log
      
    +  (
    +  cd */libphobos/
    +   sed <Makefile >Makefile.new -e 's/^DFLAGS =.*/& -ffunction-sections -fdata-sections/'
    +   mv Makefile.new Makefile
    +   make clean
    +  )
    +  make -j9 2>&1 | tee build.log
      
      make -j2 install
    [...]
    

    and "-ffunction-sections -fdata-sections -Wl,--gc-sections" when compiling and linking the program.

  2. art_

    I'd expect no significant diff in compile times, and small increase in link times (when the section GC actually happens). but i have no large enough D project to time...

    For the simple program mentioned above, the final compile-main-unit+link invocation takes 1.97s with section-gc, and 1.96s w/o, so it's in the noise.

    FWIW, "make 2>&1 | grep -c 'Removing unused section'" tells me that the GC removed 3168 unneeded sections from the resulting executable...

  3. Iain Buclaw

    To me, this looks like more of a cludge or stop gap until libphobos is set to be built as a shared library by default.

    And having a look up on the behaviour of the flags, it seems that building with -ffunction-sections/-fdata-sections will generally create larger binaries unless one remembers to always compile with --gc-sections. Ugh.

  4. art_

    It will be many years until a std shared d runtime could even be considered a real option... Hence static linking will be the norm for at least the next five years, maybe much longer...

    Larger binaries? The linker will merge the sections even when -Wl,--gc-sections is omitted.

    BTW i also tried gc-sections together with LTO in the mean time, and that gets me an ICE again, but i guess that's expected (lto1: internal compiler error: in ipcp_iterate_stage, at ipa-cp.c:891)

  5. Iain Buclaw

    It will be many years until a std shared d runtime could even be considered a real option... Hence static linking will be the norm for at least the next five years, maybe much longer...

    I expect to roll out shared library support over the next two months...

  6. art_

    I certainly appreciate your (and everybody elses) work on this, and hope shared libs happen sooner rather than later.

    But it's not as simple as making the dlls build. Can you imagine somebody shipping a (binary) product linked to a shared system D runtime (phobos etc) in three months time? Half a year? Two years?

    Because if it can not use system provided libs, it might just as well be linked statically; in fact doing this avoids a lot of problems, for which the only solution would be "make sure the very library we shipped you is used" anyway.

    Is phobos stable enough that you can rely on the dll ABI not changing for the lifetime of the product? Being included in every relevant distribution/OS, with all of them being compatible?

    How many versions of the dlls will there be? IOW, will you have to pick one of llvm, gdc or dmd variants? Support all of them?

    The problem may be a little smaller for source (or pseudo-source, such as ubuntu etc) based distributions, but it is still there. I imagine the windows way would be a bundled dll with every single D app anyway, so that doesn't impact much.

    My point is just that statically linked D programs [1] will be relevant for quite some time yet, and the "stop gap" would be useful even long after shared libs became officially supported. Hence the multi-year estimate above.

    Is there any technical problem with the per-function|data sections approach? The few tests i was able to run here didn't turn up anything, other than the huge difference in executable sizes. (First thing i was worried about was the possibility of the D GC being confused by data/bss layout changes, but things seem ok so far. Normal (ie. w/o ld's --gc-sections) linking seems to work reasonably (same number of sections in the resulting executable, similar size)

    [1] well, "programs statically linked with phobos", as the issues with other (common) libs are not much different from Cs.

  7. Iain Buclaw

    1) Do some GC intensive tests:

    2) The phobos library shipped with GDC is to reside in the GCC library directories, so is not to be confused with system libraries. Generally, you release a binary, you say it requires GDC runtime library version 4.x.x to run. It is not assumed that mixing libraries between DMD, GDC and LDC will work, due to ABI differences, but with the issue you raised on library variants, this is only an issue if you ship libraries as a standalone package/install. Applications on the other hand will have no such problem as they tend to ship their own dependencies in their own library directories.

    Regards

  8. Anonymous

    1. resulting executable is already doubly bloated, so if one doesn't care about size, he may not care about forgetting --gc-sections. It should be also solvable with compiler config, right?

    2. shared and static linking serve different purposes and have different tradeoffs (like distribution and maintainability issues). Static linking will always have sense, so ideally both options should be available (just like for c++ libs). This proposal improves static linking and seemingly doesn't break shared one.

    3. shared linking has even worse size issues because nothing can be omitted from the library.

  9. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.