x265_denoise_dct_sse4 segmentation fault

x265 hg version 10979:44cc2ce22c43, CPU has sse4_1 sse4_2 (Core i5 2500K), x86_64 Fedora 22, gcc 5.2.1-2.

Running ffmpeg with -x265-params ssim=0:crf=16:nr-inter=1000:nr-intra=1000 -vcodec libx265 -preset medium

No segfault if nr-inter & nr-intra are 0.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe514a700 (LWP 10969)]
0x00007ffff58281e0 in x265_denoise_dct_sse4 () from /lib64/libx265.so.74
(gdb) bt
#0  0x00007ffff58281e0 in x265_denoise_dct_sse4 () from /lib64/libx265.so.74
#1  0x00007ffff5b18078 in x265::Quant::transformNxN(x265::CUData const&, unsigned char const*, unsigned int, short const*, unsigned int, short*, unsigned int, x265::TextType, unsigned int, bool) () from /lib64/libx265.so.74
#2  0x00007ffff57976ec in x265::Search::estimateResidualQT(x265::Mode&, x265::CUGeom const&, unsigned int, unsigned int, x265::ShortYuv&, x265::Search::Cost&, unsigned int const*) () from /lib64/libx265.so.74
#3  0x00007ffff5797ee2 in x265::Search::estimateResidualQT(x265::Mode&, x265::CUGeom const&, unsigned int, unsigned int, x265::ShortYuv&, x265::Search::Cost&, unsigned int const*) () from /lib64/libx265.so.74
#4  0x00007ffff579c379 in x265::Search::encodeResAndCalcRdInterCU(x265::Mode&, x265::CUGeom const&) () from /lib64/libx265.so.74
#5  0x00007ffff5787b00 in x265::Analysis::checkMerge2Nx2N_rd0_4(x265::Mode&, x265::Mode&, x265::CUGeom const&) () from /lib64/libx265.so.74
#6  0x00007ffff578cc58 in x265::Analysis::compressInterCU_rd0_4(x265::CUData const&, x265::CUGeom const&, int) () from /lib64/libx265.so.74
#7  0x00007ffff578ed3a in x265::Analysis::compressCTU(x265::CUData&, x265::Frame&, x265::CUGeom const&, x265::Entropy const&) () from /lib64/libx265.so.74
#8  0x00007ffff57ae097 in x265::FrameEncoder::processRowEncoder(int, x265::ThreadLocalData&) () from /lib64/libx265.so.74
#9  0x00007ffff57acc71 in x265::FrameEncoder::processRow(int, int) () from /lib64/libx265.so.74
#10 0x00007ffff5afad48 in x265::WaveFront::findJob(int) () from /lib64/libx265.so.74
#11 0x00007ffff5afa5da in x265::WorkerThread::threadMain() () from /lib64/libx265.so.74
#12 0x00007ffff5af9e8a in x265::ThreadShim(void*) () from /lib64/libx265.so.74
#13 0x00007ffff603f555 in start_thread (arg=0x7fffe514a700) at pthread_create.c:333
#14 0x00007ffff17e6b9d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) d32
Dump of assembler code from 0x7ffff58281c0 to 0x7ffff5828200:
   0x00007ffff58281c0 <x265_idct8_ssse3+112>:   add    %al,(%rcx)
   0x00007ffff58281c2 <x265_idct8_ssse3+114>:   add    %al,(%rax)
   0x00007ffff58281c4 <x265_idct8_ssse3+116>:   retq   
   0x00007ffff58281c5 <x265_idct8_ssse3+117>:   nopl   0x0(%rax,%rax,1)
   0x00007ffff58281ca <x265_idct8_ssse3+122>:   nopw   0x0(%rax,%rax,1)
   0x00007ffff58281d0 <x265_denoise_dct_sse4+0>:    pxor   %xmm5,%xmm5
   0x00007ffff58281d4 <x265_denoise_dct_sse4+4>:    shr    $0x3,%ecx
   0x00007ffff58281d7 <x265_denoise_dct_sse4+7>:    movdqa (%rdi),%xmm0
   0x00007ffff58281db <x265_denoise_dct_sse4+11>:   pabsw  %xmm0,%xmm1
=> 0x00007ffff58281e0 <x265_denoise_dct_sse4+16>:   movdqa (%rsi),%xmm2
   0x00007ffff58281e4 <x265_denoise_dct_sse4+20>:   pmovsxwd %xmm1,%xmm3
   0x00007ffff58281e9 <x265_denoise_dct_sse4+25>:   paddd  %xmm3,%xmm2
   0x00007ffff58281ed <x265_denoise_dct_sse4+29>:   movdqa %xmm2,(%rsi)
   0x00007ffff58281f1 <x265_denoise_dct_sse4+33>:   movdqa 0x10(%rsi),%xmm2
   0x00007ffff58281f6 <x265_denoise_dct_sse4+38>:   movdqa %xmm1,%xmm3
   0x00007ffff58281fa <x265_denoise_dct_sse4+42>:   psrldq $0x8,%xmm3
   0x00007ffff58281ff <x265_denoise_dct_sse4+47>:   pmovsxwd %xmm3,%xmm4
End of assembler dump.
(gdb) x/4x $rsi
0x7fffdc2d6eb8: 0x00000000  0x00000000  0x00000000  0x00000000

$ grep ^7fffdc /proc/10953/maps
7fffdc000000-7fffdc4e2000 rw-p 00000000 00:00 0 
7fffdc4e2000-7fffe0000000 ---p 00000000 00:00 0

Comments (4)