1.12 SEGVs when OpenMP is enabled

Issue #227 resolved

yurivict created an issue 2022-01-08

$ ART

(ART:37787): GLib-GObject-CRITICAL **: 13:45:43.319: g_object_unref: assertion 'old_ref > 0' failed

(ART:37787): GLib-GObject-CRITICAL **: 13:45:43.319: g_object_unref: assertion 'old_ref > 0' failed

(ART:37787): GLib-GObject-CRITICAL **: 13:45:43.326: g_object_unref: assertion 'old_ref > 0' failed
OMP: Error #13: Assertion failure at kmp_runtime.cpp(3689).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
Abort trap
[yuri@yv /disk-samsung/freebsd-ports/graphics/art]$ ART

(ART:37789): GLib-GObject-CRITICAL **: 13:46:02.394: g_object_unref: assertion 'old_ref > 0' failed
Segmentation fault

‌

Comments (25)

yurivict reporter

(gdb) bt
[?2004l#0  thr_kill () at thr_kill.S:4
#1  0x0000000803bb13d4 in __raise (s=s@entry=6) at /disk-samsung/freebsd-src/lib/libc/gen/raise.c:52
#2  0x0000000803c65f39 in abort () at /disk-samsung/freebsd-src/lib/libc/stdlib/abort.c:67
#3  0x0000000802ae78c0 in __kmp_abort_process() () at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.cpp:449
#4  0x0000000802ae0014 in __kmp_fatal(kmp_msg_t, ...) (message=...) at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_i18n.cpp:868
#5  0x0000000802ac73a5 in __kmp_debug_assert(char const*, char const*, int) (msg=<optimized out>, file=<optimized out>, line=3689) at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/r
untime/src/kmp_debug.cpp:74
#6  0x0000000802ae6cba in __kmp_register_root(int) (initial_thread=<optimized out>, initial_thread@entry=0) at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.
cpp:3689
#7  0x0000000802ae6791 in __kmp_get_global_thread_id_reg() () at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.cpp:253
#8  0x00000000008140e7 in  ()
#9  0x00000000009d43c9 in  ()
#10 0x00000000009c5bda in  ()
#11 0x0000000000702b78 in  ()
#12 0x00000000006fc7e2 in  ()
#13 0x00000000004bc3c0 in  ()
#14 0x000000000083f40c in  ()
#15 0x000000000083f23c in  ()
#16 0x00000008018777ea in thread_start (curthread=0x807214a00) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_create.c:292
#17 0x0000000000000000 in  ()

‌

2022-01-08T21:55:27+00:00

yurivict reporter
clang-12

‌

OS: FreeBSD 13

‌
- 2022-01-08T21:56:00+00:00
agriggio repo owner
Hi,

can you provide a GDB backtrace of a debug build? Thanks!
- 2022-01-13T07:51:15+00:00

yurivict reporter

(gdb) bt
#0  thr_kill () at thr_kill.S:4
#1  0x00000008041b13d4 in __raise (s=s@entry=6) at /disk-samsung/freebsd-src/lib/libc/gen/raise.c:52
#2  0x0000000804265f39 in abort () at /disk-samsung/freebsd-src/lib/libc/stdlib/abort.c:67
#3  0x00000008031428c0 in __kmp_abort_process() () at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.cpp:449
#4  0x000000080313b014 in __kmp_fatal(kmp_msg_t, ...) (message=...) at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_i18n.cpp:868
#5  0x00000008031223a5 in __kmp_debug_assert(char const*, char const*, int) (msg=<optimized out>, file=<optimized out>, line=3689)
    at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_debug.cpp:74
#6  0x0000000803141cba in __kmp_register_root(int) (initial_thread=<optimized out>, initial_thread@entry=0)
    at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.cpp:3689
#7  0x0000000803141791 in __kmp_get_global_thread_id_reg() () at /disk-samsung/freebsd-src/contrib/llvm-project/openmp/runtime/src/kmp_runtime.cpp:253
#8  0x0000000000ce636f in rtengine::Imagefloat::multiply(float, bool) (this=0x110f310 <rtengine::ICCStore::getInstance()::instance>, factor=0, multithread=false)
    at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtengine/imagefloat.cc:395
#9  0x0000000000cee24d in rtengine::Imagefloat::normalizeFloatTo1(bool) (this=0x814e09800, multithread=true)
    at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtengine/imagefloat.cc:429
#10 0x0000000000f8c94a in rtengine::StdImageSource::colorSpaceConversion(rtengine::Imagefloat*, rtengine::procparams::ColorManagementParams const&, void*, rtengine::IIO_Sample_Format, rtengine::ProgressListener*) (im=0x814e09800, cmp=..., embedded=0x0, sampleFormat=rtengine::IIOSF_UNKNOWN, plistener=0x0)
    at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtengine/stdimagesource.cc:301
#11 0x0000000000f6e1ff in rtengine::Thumbnail::processImage(rtengine::procparams::ProcParams const&, rtengine::SensorType, int, rtengine::TypeInterpolation, rtengine::FramesMetaData const*, double&, bool, bool)
    (this=0x814e6db80, params=..., sensorType=rtengine::ST_NONE, rheight=126, interp=rtengine::TI_Bilinear, metadata=0x808420028, myscale=@0x7fffdf5f8af0: 1, forMonitor=true, forHistogramMatching=false) at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtengine/rtthumbnail.cc:1041
#12 0x0000000000b0d149 in Thumbnail::processThumbImage(rtengine::procparams::ProcParams const&, int, double&) (this=0x808420000, pparams=..., h=126, scale=@0x7fffdf5f8af0: 1)
    at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtgui/thumbnail.cc:687
#13 0x0000000000b00b92 in ThumbImageUpdater::Impl::processNextJob() (this=0x81006ac80) at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtgui/thumbimageupdater.cc:152
#14 0x0000000000b02442 in sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>::operator()() const (this=0x8122a1ab8) at /usr/local/include/sigc++-2.0/sigc++/functors/mem_fun.h:1991
#15 0x0000000000b023bd in std::__1::__invoke<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&) (__f=...)
    at /usr/include/c++/v1/type_traits:3694
#16 0x0000000000b0238d in std::__1::__apply_functor<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>, std::__1::tuple<>, , std::__1::tuple<> >(sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&, std::__1::tuple<>&, std::__1::__tuple_indices<>, std::__1::tuple<>&&) (__f=..., __bound_args=..., __args=...) at /usr/include/c++/v1/functional:2857
#17 0x0000000000b0234b in std::__1::__bind<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>::operator()<>() (this=0x8122a1ab8) at /usr/include/c++/v1/functional:2890
#18 0x0000000000b022fd in std::__1::__invoke<std::__1::__bind<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>&>(std::__1::__bind<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>&) (__f=...) at /usr/include/c++/v1/type_traits:3694
#19 0x0000000000b02141 in std::__1::__packaged_task_func<std::__1::__bind<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>, std::__1::allocator<std::__1::__bind<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&> >, void ()>::operator()() (this=0x8122a1ab0) at /usr/include/c++/v1/future:1684
#20 0x00000000006e87fa in std::__1::__packaged_task_function<void ()>::operator()() const (this=0x812293e40) at /usr/include/c++/v1/future:1866
#21 0x00000000006e8721 in std::__1::packaged_task<void ()>::operator()() (this=0x812293e40) at /usr/include/c++/v1/future:2086
#22 0x0000000000b0430d in rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}::operator()() const (this=0x814e00018) at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtgui/../rtengine/threadpool.h:159
#23 0x0000000000b042cd in std::__1::__invoke<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}&>(sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&) (__f=...) at /usr/include/c++/v1/type_traits:3694
#24 0x0000000000b0427d in std::__1::__invoke_void_return_wrapper<void, true>::__call<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}&>(rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}&) (__args=...) at /usr/include/c++/v1/__functional_base:348
#25 0x0000000000b0424d in std::__1::__function::__alloc_func<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}, std::__1::allocator<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}>, void ()>::operator()() (this=0x814e00018) at /usr/include/c++/v1/functional:1558
#26 0x0000000000b03129 in std::__1::__function::__func<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_me
m_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}, std::__1::allocator<rtengine::ThreadPool::enqueue<sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&>(rtengine::ThreadPool::Priority, sigc::bound_mem_functor0<void, ThumbImageUpdater::Impl>&)::{lambda()#1}>, void ()>::operator()() (this=0x814e00010) at /usr/include/c++/v1/functional:1732
#27 0x0000000000d34522 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x814e00010) at /usr/include/c++/v1/functional:1885
#28 0x0000000000d31f95 in std::__1::function<void ()>::operator()() const (this=0x814e00010) at /usr/include/c++/v1/functional:2560
#29 0x0000000000d33a6d in std::__1::__invoke<rtengine::ThreadPool::Task&>(rtengine::ThreadPool::Task&) (__f=...) at /usr/include/c++/v1/type_traits:3694
#30 0x0000000000d33a1d in std::__1::__invoke_void_return_wrapper<void, true>::__call<rtengine::ThreadPool::Task&>(rtengine::ThreadPool::Task&) (__args=...)
    at /usr/include/c++/v1/__functional_base:348
#31 0x0000000000d339ed in std::__1::__function::__alloc_func<rtengine::ThreadPool::Task, std::__1::allocator<rtengine::ThreadPool::Task>, void ()>::operator()() (this=0x814e00010)
    at /usr/include/c++/v1/functional:1558
#32 0x0000000000d32ac9 in std::__1::__function::__func<rtengine::ThreadPool::Task, std::__1::allocator<rtengine::ThreadPool::Task>, void ()>::operator()() (this=0x814e00000)
    at /usr/include/c++/v1/functional:1732
#33 0x0000000000d34522 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x7fffdf5f8f00) at /usr/include/c++/v1/functional:1885
#34 0x0000000000d31f95 in std::__1::function<void ()>::operator()() const (this=0x7fffdf5f8f00) at /usr/include/c++/v1/functional:2560
#35 0x0000000000d31cfb in rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const (this=0x80790c378)
    at /disk-samsung/freebsd-ports/graphics/art/work/ART-1.12/rtengine/threadpool.h:131
#36 0x0000000000d31b6d in std::__1::__invoke<rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}>(rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}&&) (__f=...)
    at /usr/include/c++/v1/type_traits:3694
#37 0x0000000000d31b45 in std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}>&, std::__1::__tuple_indices<>) (__t=...) at /usr/include/c++/v1/thread:280
#38 0x0000000000d318d0 in std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, rtengine::ThreadPool::ThreadPool(unsigned long)::{lambda()#1}> >(void*) (__vp=0x80790c370) at /usr/include/c++/v1/thread:291
#39 0x0000000801ed07ea in thread_start (curthread=0x807814a00) at /disk-samsung/freebsd-src/lib/libthr/thread/thr_create.c:292
#40 0x0000000000000000 in  ()

‌

2022-01-13T08:16:37+00:00

yurivict reporter
It appears to crash while processing this image.
- 2022-01-13T08:29:38+00:00
yurivict reporter
This assert causes failure:

KMP_ASSERT(gtid < __kmp_threads_capacity);

‌
- 2022-01-15T19:48:50+00:00
yurivict reporter
Building with OPTION_OMP=OFF helps.

‌
- 2022-01-15T19:53:40+00:00
yurivict reporter
- changed title to 1.12 SEGVs when OpenMP is enabled
- 2022-01-15T19:55:41+00:00
yurivict reporter
I added the OPENMP option to the FreeBSD port, making it OFF by default. This way there’s no SEGV.

‌

ART FreeBSD port
- 2022-01-15T20:14:18+00:00
agriggio repo owner
thanks. Seems like a very costly workaround though

I have no experience with freebsd but I'll try getting a VM and see if I can find the culprit
- 2022-01-15T20:16:18+00:00
agriggio repo owner
did you try compiling with clang btw?
- 2022-01-15T20:17:55+00:00
yurivict reporter
Thanks!

‌

I have instructions on how to install FreeBSD into a VM:

‌

Is there an easy way to test under FreeBSD?
1. Install the FreeBSD VM image into VirtualBox.
2. Boot FreeBSD
3. Install git: pkg install git
4. Check out the ports tree: git clone https://git.FreeBSD.org/ports.git /usr/ports
5. cd /usr/ports/graphics/art
6. Install dependencies: pkg install -A make missing
7. Build: make
8. Install: make install
‌
- 2022-01-15T20:18:48+00:00
yurivict reporter
did you try compiling with clang btw?

Port normally builds with clang-12.

‌

‌
- 2022-01-15T20:20:21+00:00
agriggio repo owner
thanks for the instructions! Is there a way to compile with GCC as well?
- 2022-01-15T20:43:46+00:00
yurivict reporter
Is there a way to compile with GCC as well?

Add
```
USE_GCC=        yes
```
in the middle of Makefile first.

‌
- 2022-01-15T20:49:42+00:00
agriggio repo owner
Reading through https://reviews.llvm.org/D77609, it seems ART is not the only app that hits the assertion. Can you try setting LIBOMP_NUM_HIDDEN_HELPER_THREADS=0 (as suggested in one of the comments there) and see if that helps?
- 2022-01-16T08:29:53+00:00
agriggio repo owner
Hi,

did you have the chance to try and see if LIBOMP_NUM_HIDDEN_HELPER_THREADS=0

helps? I’d like to release a version 1.12.1 to fix a few (minor) glitches, but I wanted to sort this out first…

Thanks!
- 2022-01-19T10:35:02+00:00
yurivict reporter
The problem is gone with
```
 LIBOMP_NUM_HIDDEN_HELPER_THREADS=0
```
‌
- 2022-01-20T08:25:07+00:00
agriggio repo owner
Thanks for checking. Would it be possible to tweak your port to use a wrapper script to run ART with the env var set?

Alternatively, it might also be that this is an LLVM-specific issue, and compiling with gcc would also solve it.

(FWIW I tried with clang-12 on Linux but I couldn’t reproduce)
- 2022-01-20T12:26:27+00:00
yurivict reporter
Would it be possible to tweak your port to use a wrapper script to run ART with the env var set?

‌

I think it’s better to add this environment variable in the beginning on the main function. It can be if-defed only for clang if this is clang-specific.

‌

I just verified - this works.
- 2022-01-20T17:36:50+00:00
agriggio repo owner
If this is an issue with LLVM/clang, it might be version and platform dependent. So, I would prefer this to be addressed at the distro level. For example, can you apply the patch when building the port? I haven’t been able to reproduce the bug on Linux, even when using clang
- 2022-01-21T14:08:42+00:00
yurivict reporter
Ok, I will add such patch.
- 2022-01-21T16:35:53+00:00
yurivict reporter
The patch was added to the port.
- 2022-01-21T17:46:47+00:00
agriggio repo owner
great, thanks! I'll close this for now then
- 2022-01-21T17:56:37+00:00
agriggio repo owner
- changed status to resolved
- 2022-01-21T17:57:24+00:00
Log in to comment

Assignee: –

Type: bug

Priority: major

Status: resolved

Votes: 0

Watchers: 1