build_ext fails when hardening options enabled

Issue #498 closed
Ben Finney created an issue

With various “hardening” options enabled in the C compiler, the build_ext command fails with many “undefined reference” errors.

Comments (7)

  1. Ned Batchelder repo owner

    @bignose all of the undefined references seem to be C API symbols. I don't understand what hardening is, why it might be enabled, or what I am supposed to do about it. Do you have any insight and/or guidance?

  2. Ben Finney reporter

    I don't understand what hardening is

    Ah, my apologies for assuming the context. This is referring to the “hardening” of an executable against common security vulnerabilities, by enabling specific improvements – the “hardening options” – when invoking the C-language compiler.

    A lot of detail can be had in this 2005 article introducing C compiler improvements. In brief, it encompasses features like stack-smashing protection, position-independent executables, buffer overflow detection, and other recent compiler features that protect against many vulnerabilities to which C code is especially prone.

    why it might be enabled, or what I am supposed to do about it.

    The options are enabled by the party compiling the C module, by setting environment variables that are used by the C compiler.

    The resulting C compiler command line options are displayed in the attachment; you can use those options yourself to reproduce the behaviour with that compiler.

  3. Ben Finney reporter

    I will attempt to reproduce this with fewer options, to narrow down the causes.

    Looking at many of the errors, though, I think some errors (e.g. referencing names not explicitly defined) might be from a more-strict set of checks that are active by default in recent GCC versions.

  4. Ben Finney reporter

    I will attempt to reproduce this with fewer options, to narrow down the causes.

    The behaviour is reproducible by requesting a position independent executable:

    $ gcc --version
    gcc (Debian 5.4.0-6) 5.4.0 20160609
    []
    
    $ export CFLAGS="-pie"
    $ python3 ./setup.py build_ext
    running build_ext
    building 'coverage.tracer' extension
    []
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -pie -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o build/temp.linux-x86_64-3.5/coverage/ctracer/filedisp.o build/temp.linux-x86_64-3.5/coverage/ctracer/module.o build/temp.linux-x86_64-3.5/coverage/ctracer/tracer.o -o build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
    (.text+0x20): undefined reference to `main'
    build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o: In function `DataStack_grow':
    /home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:31: undefined reference to `PyMem_Realloc'
    /home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:34: undefined reference to `PyErr_NoMemory'
    build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o: In function `DataStack_dealloc':
    /home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:21: undefined reference to `PyMem_Free'
    build/temp.linux-x86_64-3.5/coverage/ctracer/module.o: In function `PyInit_tracer':
    /home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/module.c:31: undefined reference to `PyModule_Create2'
    /home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/module.c:41: undefined reference to `PyType_GenericNew'
    []
    collect2: error: ld returned 1 exit status
    
  5. Loic Dachary

    I only now learn about hardening, from wikipedia Position independent code and the Debian page. Although adding -lpython3.5m to the compile line at

    $ x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -pie -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o build/temp.linux-x86_64-3.5/coverage/ctracer/filedisp.o build/temp.linux-x86_64-3.5/coverage/ctracer/module.o build/temp.linux-x86_64-3.5/coverage/ctracer/tracer.o  -lpython3.5m -o build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
    /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
    (.text+0x20): undefined reference to `main'
    collect2: error: ld returned 1 exit status
    

    resolves most linkage issues, it fails because main is missing, which makes sense because this is a library and not a standalone executable. I'm not sure I understand why a shared library would need pie in addition to PIC ?

    Unless I missed something the shared library seems to be properly hardened by default:

    $ hardening-check build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
    build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so:
     Position Independent Executable: no, regular shared library (ignored)
     Stack protected: yes
     Fortify Source functions: unknown, no protectable libc functions used
     Read-only relocations: yes
     Immediate binding: no, not found!
    
  6. Thomas Wouters

    This problem doesn't appear to have anything to do with coverage, as such, as any extension module would have the same problem: you're (indirectly) passing -pie to the link of a shared library. -pie only makes sense on the final link of an executable, which is why it insists there is a 'main' to reference, as well as that all symbols are resolved.

    Extension modules are shared libraries. They intentionally leave a lot of symbols unresolved -- all symbols that would be satisfied by the python executable they are loaded into. It's not a good idea to dynamically link extension modules to (e.g.) libpython3.5 to resolve those symbols, as that library only exists when Python is built with --enable-shared (and it might not be). It's also useless: either the libpythonX.Y shared library will already have been loaded by the python process, or it shouldn't be loaded.

    The equivalent of -pie for shared libraries is -fPIC, which is already the default (because of how they work, shared libraries have to be linked position-independently, which means all names go through a runtime lookup table, just like PIE). There's also -fPIE, but I believe code generated that way can't be linked into a shared library.

  7. Ned Batchelder repo owner
    • changed status to closed
    • edited description

    Seems like the library is already hardened? If you can provide more information about steps I can take in building coverage.py, please feel free to re-open.

  8. Log in to comment