Build fails with LTO

[Eli Schwartz]

I tried to build with the following *FLAGS to optimize the build: -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing

Note the -Werror=* flags are used to help detect cases where the compiler tries to optimize by assuming UB cannot exist in the source code -- if it does exist, ordinarily the code would be miscompiled, and this says to make the miscompilation a fatal error.

I got this error:

FAILED: libx265.so.199 
: && /usr/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=native -fstack-protector-all -O2 -pipe -fdiagnostics-color=always -frecord-gcc-switches -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-clash-protection -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing  -Wformat -Werror=format-security  -Wl,-O1 -Wl,--as-needed -flto=4 -Werror=odr -Werror=lto-type-mismatch -Werror=strict-aliasing -Wl,--defsym=__gentoo_check_ldflags__=0  -L. -Wl,-Bsymbolic,-znoexecstack -shared -Wl,-soname,libx265.so.199 -o libx265.so.199 encoder/CMakeFiles/encoder.dir/analysis.cpp.o encoder/CMakeFiles/encoder.dir/search.cpp.o encoder/CMakeFiles/encoder.dir/bitcost.cpp.o encoder/CMakeFiles/encoder.dir/motion.cpp.o encoder/CMakeFiles/encoder.dir/slicetype.cpp.o encoder/CMakeFiles/encoder.dir/frameencoder.cpp.o encoder/CMakeFiles/encoder.dir/framefilter.cpp.o encoder/CMakeFiles/encoder.dir/level.cpp.o encoder/CMakeFiles/encoder.dir/nal.cpp.o encoder/CMakeFiles/encoder.dir/sei.cpp.o encoder/CMakeFiles/encoder.dir/sao.cpp.o encoder/CMakeFiles/encoder.dir/entropy.cpp.o encoder/CMakeFiles/encoder.dir/dpb.cpp.o encoder/CMakeFiles/encoder.dir/ratecontrol.cpp.o encoder/CMakeFiles/encoder.dir/reference.cpp.o encoder/CMakeFiles/encoder.dir/encoder.cpp.o encoder/CMakeFiles/encoder.dir/api.cpp.o encoder/CMakeFiles/encoder.dir/weightPrediction.cpp.o common/CMakeFiles/common.dir/x86/pixel-a.asm.o common/CMakeFiles/common.dir/x86/const-a.asm.o common/CMakeFiles/common.dir/x86/cpu-a.asm.o common/CMakeFiles/common.dir/x86/ssd-a.asm.o common/CMakeFiles/common.dir/x86/mc-a.asm.o common/CMakeFiles/common.dir/x86/mc-a2.asm.o common/CMakeFiles/common.dir/x86/pixel-util8.asm.o common/CMakeFiles/common.dir/x86/blockcopy8.asm.o common/CMakeFiles/common.dir/x86/pixeladd8.asm.o common/CMakeFiles/common.dir/x86/dct8.asm.o common/CMakeFiles/common.dir/x86/seaintegral.asm.o common/CMakeFiles/common.dir/x86/sad-a.asm.o common/CMakeFiles/common.dir/x86/intrapred8.asm.o common/CMakeFiles/common.dir/x86/intrapred8_allangs.asm.o common/CMakeFiles/common.dir/x86/v4-ipfilter8.asm.o common/CMakeFiles/common.dir/x86/h-ipfilter8.asm.o common/CMakeFiles/common.dir/x86/ipfilter8.asm.o common/CMakeFiles/common.dir/x86/loopfilter.asm.o common/CMakeFiles/common.dir/x86/asm-primitives.cpp.o common/CMakeFiles/common.dir/vec/vec-primitives.cpp.o common/CMakeFiles/common.dir/vec/dct-sse3.cpp.o common/CMakeFiles/common.dir/vec/dct-ssse3.cpp.o common/CMakeFiles/common.dir/vec/dct-sse41.cpp.o common/CMakeFiles/common.dir/primitives.cpp.o common/CMakeFiles/common.dir/pixel.cpp.o common/CMakeFiles/common.dir/dct.cpp.o common/CMakeFiles/common.dir/lowpassdct.cpp.o common/CMakeFiles/common.dir/ipfilter.cpp.o common/CMakeFiles/common.dir/intrapred.cpp.o common/CMakeFiles/common.dir/loopfilter.cpp.o common/CMakeFiles/common.dir/constants.cpp.o common/CMakeFiles/common.dir/cpu.cpp.o common/CMakeFiles/common.dir/version.cpp.o common/CMakeFiles/common.dir/threading.cpp.o common/CMakeFiles/common.dir/threadpool.cpp.o common/CMakeFiles/common.dir/wavefront.cpp.o common/CMakeFiles/common.dir/md5.cpp.o common/CMakeFiles/common.dir/bitstream.cpp.o common/CMakeFiles/common.dir/yuv.cpp.o common/CMakeFiles/common.dir/shortyuv.cpp.o common/CMakeFiles/common.dir/picyuv.cpp.o common/CMakeFiles/common.dir/common.cpp.o common/CMakeFiles/common.dir/param.cpp.o common/CMakeFiles/common.dir/frame.cpp.o common/CMakeFiles/common.dir/framedata.cpp.o common/CMakeFiles/common.dir/cudata.cpp.o common/CMakeFiles/common.dir/slice.cpp.o common/CMakeFiles/common.dir/lowres.cpp.o common/CMakeFiles/common.dir/piclist.cpp.o common/CMakeFiles/common.dir/predict.cpp.o common/CMakeFiles/common.dir/scalinglist.cpp.o common/CMakeFiles/common.dir/quant.cpp.o common/CMakeFiles/common.dir/deblock.cpp.o common/CMakeFiles/common.dir/scaler.cpp.o  -Wl,-Bstatic  -lx265_main12  -lx265_main10  -Wl,-Bdynamic  -lpthread  -lrt  -ldl && :
/var/tmp/portage/media-libs/x265-3.5-r3/work/x265_3.5/source/./x265.h:193:16: error: type ‘struct x265_analysis_distortion_data’ violates the C++ One Definition Rule [-Werror=odr]
  193 | typedef struct x265_analysis_distortion_data
      |                ^
/var/tmp/portage/media-libs/x265-3.5-r3/work/x265_3.5/source/./x265.h:193:16: note: a different type is defined in another translation unit
  193 | typedef struct x265_analysis_distortion_data
      |                ^
/var/tmp/portage/media-libs/x265-3.5-r3/work/x265_3.5/source/./x265.h:195:19: note: the first difference of corresponding definitions is field ‘ctuDistortion’
  195 |     sse_t*        ctuDistortion;
      |                   ^
/var/tmp/portage/media-libs/x265-3.5-r3/work/x265_3.5/source/./x265.h:195:19: note: a field of same name but different type is defined in another translation unit
  195 |     sse_t*        ctuDistortion;
      |                   ^
lto1: some warnings being treated as errors
lto-wrapper: fatal error: /usr/bin/x86_64-pc-linux-gnu-g++ returned 1 exit status
compilation terminated.
/usr/lib/gcc/x86_64-pc-linux-gnu/13/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Also reported downstream: https://bugs.gentoo.org/875854

Full build log is attached.

[Priyadarsini Shanmugam]

Thank you for your report regarding the ODR violation when LTO is enabled. But, we are unable to support LTO in our multilib configuration due to the complexities and potential conflicts that arise from having multiple library versions.

As a result, We recommend disabling LTO in your build configuration by removing -flto flag from your build configuration. This will help avoid the current warnings and ensure stability within the multilib environment.

[Eli Schwartz]

Link-Time Optimization is a massively global compiler optimization pass which is pretty handy for producing faster executables. It also has the interesting property that because the compiler does whole-program analysis using bytecode, it can save type information and perform error checks that it normally doesn’t have enough insight for. In particular, checking for ODR issues and checking function type signature mismatches.

This bug report is NOT about an ODR violation when LTO is enabled. At all!

This bug report is about the fact that you have an ODR violation no matter what! Having an ODR violation is bad – it means you are breaking the rules of the C++ language.

There is no complexity here – LTO is a silent, invisible optimization pass just like -O2 or -O3. Also, I have no idea what this has to do with multilib – we build x265 and MANY other software products with multiib, most of them with LTO.