Intermittent segfault when encoding y4m source on Linux using 1.9

In our regression suite, we've noticed an occasional segfault across multiple test cases. It appears to be reproducible without any options other than the following:

x265 --input /tmp/short.y4m --y4m --output /tmp/encoded.hevc

Setting "--frame-threads 1" seems to prevent the segfault. On the other hand, setting "--preset fast" increases the frequency of the segfault. Running the above command in a loop, we're seeing an average of 1 in 20 executions segfaulting. Adding --preset fast increases this significantly. It seems likely that it's a threading issue.

Doing a git bisect points to this commit as the cause:

https://bitbucket.org/multicoreware/x265/commits/d8500d0a3664e74b39ac3d8df3dd4ca6dc9699ef

Built by going into build/linux and running :

./make-Makefiles.bash

Changed CMAKE_BUILD_TYPE to Debug (happens with Release too)

Then "make"

From commit 5b01678f6fb4e89e23cd41295592a9aa5d51d4ba (latest)

(gdb) r
Starting program: x265 --input /tmp/short.y4m --y4m --output /tmp/encoded.hevc

[Thread debugging using libthread_db enabled]
y4m  [info]: 200x112 fps 30000/1001 i420p8 sar 1:1 frames 0 - 14 of 15
[New Thread 0x7ffff6622700 (LWP 40634)]
raw  [info]: output file: /tmp/encoded.hevc
x265 [info]: HEVC encoder version +-
x265 [info]: build info [Linux][GCC 4.4.7][64 bit] 8bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-2 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: frame threads / pool features       : 3 / wpp(2 rows)
[New Thread 0x7ffff5c21700 (LWP 40635)]
[New Thread 0x7ffff5220700 (LWP 40636)]
[New Thread 0x7ffff481f700 (LWP 40637)]
[New Thread 0x7ffff3e1e700 (LWP 40638)]
[New Thread 0x7ffff341d700 (LWP 40639)]
[New Thread 0x7ffff2a1c700 (LWP 40640)]
[New Thread 0x7ffff201b700 (LWP 40641)]
[New Thread 0x7ffff161a700 (LWP 40642)]
x265 [warning]: Source height < 720p; disabling lookahead-slices
[New Thread 0x7ffff0c19700 (LWP 40643)]
[New Thread 0x7fffebfff700 (LWP 40644)]
[New Thread 0x7fffeb5fe700 (LWP 40645)]
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 25 / 250 / 40
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / 1 / 1
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 signhide tmvp strong-intra-smoothing
x265 [info]: tools: deblock sao
[Thread 0x7ffff6622700 (LWP 40634) exited]
[6.7%] 1/15 frames, 11.93 fps, 522.70 kb/s, eta 0:00:01  
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff481f700 (LWP 40637)]
0x00007ffff7848b31 in x265::SAO::rdoSaoUnitCu (this=0x6de480, saoParam=0x7fffac0008c0, rowBaseAddr=4, idxX=0, addr=4) at /mnt/hgfs/x265-hg/source/encoder/sao.cpp:1291
1291                            estDist += estSaoDist(m_count[plane][typeIdx][classIdx + bandPos], mergeOffset, m_offsetOrg[plane][typeIdx][classIdx + bandPos]);
(gdb) bt
#0  0x00007ffff7848b31 in x265::SAO::rdoSaoUnitCu (this=0x6de480, saoParam=0x7fffac0008c0, rowBaseAddr=4, idxX=0, addr=4) at /mnt/hgfs/x265-hg/source/encoder/sao.cpp:1291
#1  0x00007ffff783f48d in x265::FrameFilter::ParallelFilter::processTasks (this=0x6de3c0) at /mnt/hgfs/x265-hg/source/encoder/framefilter.cpp:364
#2  0x00007ffff7b580b8 in x265::WorkerThread::threadMain (this=0x6616a0) at /mnt/hgfs/x265-hg/source/common/threadpool.cpp:116
#3  0x00007ffff7b57e9f in x265::stackAlignMain (instance=0x6616a0) at /mnt/hgfs/x265-hg/source/common/threading.cpp:91
#4  0x00007ffff7b57ec1 in x265::ThreadShim (opaque=0x6616a0) at /mnt/hgfs/x265-hg/source/common/threading.cpp:131
#5  0x00007ffff7569851 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ffff670b94d in clone () from /lib64/libc.so.6
(gdb) print m_count
$1 = {{{1640, 318, 128, 142, 324, 0 <repeats 27 times>}, {1909, 182, 244, 229, 209, 0 <repeats 27 times>}, {1742, 362, 123, 102, 397, 0 <repeats 27 times>}, {1792, 328, 124, 144, 338, 0 <repeats 27 times>}, {0, 0, 0, 0, 0, 0, 5, 45, 118, 220, 345, 208, 144, 134, 106, 114, 130, 134, 146, 
      169, 152, 199, 289, 144, 28, 2, 0, 0, 0, 0, 0, 0}}, {{352, 25, 101, 116, 22, 0 <repeats 27 times>}, {423, 17, 107, 107, 13, 0 <repeats 27 times>}, {361, 31, 114, 107, 31, 0 <repeats 27 times>}, {346, 26, 115, 131, 26, 0 <repeats 27 times>}, {0 <repeats 12 times>, 44, 298, 344, 10, 
      0 <repeats 16 times>}}, {{363, 67, 49, 101, 36, 0 <repeats 27 times>}, {463, 16, 83, 80, 25, 0 <repeats 27 times>}, {376, 68, 51, 93, 56, 0 <repeats 27 times>}, {375, 66, 70, 85, 48, 0 <repeats 27 times>}, {0 <repeats 17 times>, 49, 120, 180, 173, 103, 71, 0, 0, 0, 0, 0, 0, 0, 0, 
      0}}}
(gdb) print plane
$2 = 0
(gdb) print m_count[plane]
$3 = {{1640, 318, 128, 142, 324, 0 <repeats 27 times>}, {1909, 182, 244, 229, 209, 0 <repeats 27 times>}, {1742, 362, 123, 102, 397, 0 <repeats 27 times>}, {1792, 328, 124, 144, 338, 0 <repeats 27 times>}, {0, 0, 0, 0, 0, 0, 5, 45, 118, 220, 345, 208, 144, 134, 106, 114, 130, 134, 146, 
    169, 152, 199, 289, 144, 28, 2, 0, 0, 0, 0, 0, 0}}
(gdb) print typeIdx
$4 = 2038135676
(gdb) print m_count[plane][typeIdx]
Cannot access memory at address 0x3cbe2ba280
================================================================================================
From commit d8500d0a3664e74b39ac3d8df3dd4ca6dc9699ef (where segfault was introduced, per bisect)
================================================================================================
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff2023700 (LWP 32607)]
0x00007ffff784fc8d in x265::SAO::sao2ChromaParamDist (this=0x6d6a58, saoParam=0x7fffec100450, addr=4, addrUp=0, addrLeft=-1, 
    mergeSaoParam=0x7ffff2022c50, mergeDist=0x7ffff2022d10) at /mnt/hgfs/x265-hg/source/encoder/sao.cpp:1732
1732                            estDist += estSaoDist(m_count[plane][typeIdx][classIdx + bandPos + 1], mergeOffset, m_offsetOrg[plane][typeIdx][classIdx + bandPos + 1]);
(gdb) bt
#0  0x00007ffff784fc8d in x265::SAO::sao2ChromaParamDist (this=0x6d6a58, saoParam=0x7fffec100450, addr=4, addrUp=0, addrLeft=-1, 
    mergeSaoParam=0x7ffff2022c50, mergeDist=0x7ffff2022d10) at /mnt/hgfs/x265-hg/source/encoder/sao.cpp:1732
#1  0x00007ffff784e882 in x265::SAO::rdoSaoUnitCu (this=0x6d6a58, saoParam=0x7fffec100450, rowBaseAddr=4, idxX=0, addr=4)
    at /mnt/hgfs/x265-hg/source/encoder/sao.cpp:1393
#2  0x00007ffff784487c in x265::FrameFilter::ParallelFilter::processTasks (this=0x6d69a0)
    at /mnt/hgfs/x265-hg/source/encoder/framefilter.cpp:193
#3  0x00007ffff7b5aa28 in x265::WorkerThread::threadMain (this=0x661880) at /mnt/hgfs/x265-hg/source/common/threadpool.cpp:113
#4  0x00007ffff7b5a80f in x265::stackAlignMain (instance=0x661880) at /mnt/hgfs/x265-hg/source/common/threading.cpp:91
#5  0x00007ffff7b5a831 in x265::ThreadShim (opaque=0x661880) at /mnt/hgfs/x265-hg/source/common/threading.cpp:131
#6  0x00007ffff7571851 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff671394d in clone () from /lib64/libc.so.6

Comments (7)