Cannot encode a raw YUV video

[Former user]

Hello,

I tried to encode a raw YUV file into a HEVC raw file but it crashed:

yuv  [info]: 1920x1080 fps 23980/1000 i420 frames 0 - 130425 of 130426
[New Thread 0x7ffff5f1a700 (LWP 31281)]
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: HEVC encoder version 0.7+375-6662df480e39
x265 [info]: build info [Linux][GCC 4.8.2][64 bit] 8bpp
x265 [info]: Main profile, Level-4 (Main tier)
[New Thread 0x7ffff5719700 (LWP 31282)]
[New Thread 0x7ffff4f18700 (LWP 31283)]
[New Thread 0x7ffff4717700 (LWP 31284)]
[New Thread 0x7ffff3f16700 (LWP 31285)]
x265 [info]: WPP streams / pool / frames         : 17 / 4 / 2
x265 [info]: CU size                             : 64
x265 [info]: Max RQT depth inter / intra         : 1 / 1
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 23 / 250 / 40
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / refs          : 1 / 1 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : CRF-28.0 / 1.0 / 1
x265 [info]: tools: rect amp rd=3 lft sao-lcu sign-hide 
[New Thread 0x7ffff2099700 (LWP 31286)]
[New Thread 0x7ffff00f2700 (LWP 31287)]
[4.5%] 5912/130426 frames, 2.14 fps, 849.07 kb/s, eta 16:09:46   
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4717700 (LWP 31284)]
0x0000000000622e3e in x265::BitCost::bitcost (this=0x264f7e8, mv=...)
    at /home/da/x265/source/encoder/bitcost.h:52
52                            s_bitsizes[(abs(mv.y - m_mvp.y) << 1) + !!(mv.y < m_mvp.y)] + 0.5f);
(gdb) bt
#0  0x0000000000622e3e in x265::BitCost::bitcost (this=0x264f7e8, mv=...)
    at /home/da/x265/source/encoder/bitcost.h:52
#1  0x00000000006196b7 in x265::TEncSearch::predInterSearch (this=0x264f658, 
    cu=0x3fe9eb0, predYuv=0x3970d10, bUseMRG=false, bLuma=true, bChroma=false)
    at /home/da/x265/source/Lib/TLibEncoder/TEncSearch.cpp:2707
#2  0x000000000062f6f2 in x265::TEncCu::xComputeCostInter (this=0x264fb58, 
    outTempCU=0x3fe9eb0, outPredYuv=0x3970d10, partSize=x265::SIZE_Nx2N, 
    bUseMRG=false) at /home/da/x265/source/encoder/compress.cpp:205
#3  0x0000000000630ff8 in x265::TEncCu::xCompressInterCU (this=0x264fb58, 
    outBestCU=@0x7ffff4716cb8: 0x3fd41f0, outTempCU=@0x1dbb190: 0x3fbe530, 
    cu=@0x7ffff4716ca0: 0x2924aba0, depth=0, PartitionIndex=0, 
    minDepth=0 '\000') at /home/da/x265/source/encoder/compress.cpp:447
#4  0x00000000005f29a1 in x265::TEncCu::compressCU (this=0x264fb58, 
    cu=0x2924aba0) at /home/da/x265/source/Lib/TLibEncoder/TEncCu.cpp:392
#5  0x000000000051e0e2 in x265::CTURow::processCU (this=0x264f178, 
    cu=0x2924aba0, slice=0x27f94c90, bufferSbac=0x0, bSaveSBac=false)
    at /home/da/x265/source/encoder/cturow.cpp:82
#6  0x0000000000519121 in x265::FrameEncoder::processRowEncoder (
    this=0x8cc788, row=14)
    at /home/da/x265/source/encoder/frameencoder.cpp:1086
#7  0x000000000051b46b in x265::FrameEncoder::processRow (this=0x8cc788, 
    row=28) at /home/da/x265/source/encoder/frameencoder.h:101
#8  0x00000000005edad0 in x265::WaveFront::findJob (this=0x8cc788)
    at /home/da/x265/source/common/wavefront.cpp:126
#9  0x000000000050ac2c in x265::PoolThread::threadMain (this=0x8c8600)
    at /home/da/x265/source/common/threadpool.cpp:159
#10 0x0000000000496003 in x265::ThreadShim (opaque=0x8c8600)
    at /home/da/x265/source/common/threading.cpp:73
#11 0x00007ffff7bc6e0e in start_thread (arg=0x7ffff4717700)
    at pthread_create.c:311
#12 0x00007ffff6edc0fd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb) p (abs(mv.y - m_mvp.y) << 1) + !!(mv.y < m_mvp.y)
$1 = 41217
(gdb) p s_bitsizes[41217]
Cannot access memory at address 0x7ffff7fff414
(gdb) p s_bitsizes[30000]
$2 = 31.4634457
(gdb) p s_bitsizes[40000]
$3 = -1.02578287e+34
(gdb) p (abs(mv.x - m_mvp.x) << 1)
$4 = 5376

Regards

[Steve Borho]

Thanks for the detailed stack dump. So evidently mv.y or mvp.y are out of range. Likely mvp from the looks of the code. Is the source video public? I'll analyze the logic to see if it can get into this place with bad data, but it would be nice to be able to verify locally if it is fixed.

[Former user]

Unfortunately the video I am trying to convert is really huge, no way you could get it but I will give it a try whenever a patch is available.

[Steve Borho]

Please try a recent tip; predInterSearch() has been rewritten

[Former user]

Still the same bug :

$ gdb -q ./x265
Reading symbols from /home/da/x265/build/linux/x265...done.
(gdb) r --input-res 1920x1080 --fps 23.976 -o out.hevc --input /home/films/Anime/Aladdin.1992.COMPLETE.BLURAY-PCH/Aladdin_1992.1080p.FR.EN.yuv
Starting program: /home/da/x265/build/linux/./x265 --input-res 1920x1080 --fps 23.976 -o out.hevc --input raw.yuv
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
yuv  [info]: 1920x1080 fps 23976/1000 i420 frames 0 - 130425 of 130426
[New Thread 0x7ffff5f19700 (LWP 26680)]
x265 [info]: HEVC encoder version 0.8+162-dc700298419d
x265 [info]: build info [Linux][GCC 4.8.2][64 bit] 8bpp
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
x265 [info]: Main profile, Level-4 (Main tier)
[New Thread 0x7ffff5718700 (LWP 26681)]
[New Thread 0x7ffff4f17700 (LWP 26682)]
[New Thread 0x7ffff4716700 (LWP 26683)]
[New Thread 0x7ffff3f15700 (LWP 26684)]
x265 [info]: WPP streams / pool / frames         : 17 / 4 / 2
x265 [info]: CU size                             : 64
x265 [info]: Max RQT depth inter / intra         : 1 / 1
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 2
x265 [info]: Keyframe min / max / scenecut       : 23 / 250 / 40
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / refs          : 1 / 1 / 3
x265 [info]: Rate Control / AQ-Strength / CUTree : CRF-28.0 / 1.0 / 1
x265 [info]: tools: rect amp rd=3 lft sao-lcu sign-hide 
[New Thread 0x7ffff2098700 (LWP 26685)]
[New Thread 0x7ffff00f1700 (LWP 26686)]
[4.5%] 5912/130426 frames, 2.39 fps, 839.86 kb/s, eta 14:28:39  
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5718700 (LWP 26681)]
0x000000000061e04a in x265::BitCost::bitcost (this=0x26bc770, mv=...)
    at /home/da/x265/source/encoder/bitcost.h:51
51                            s_bitsizes[(abs(mv.y - m_mvp.y) << 1) + !!(mv.y < m_mvp.y)] + 0.5f);
(gdb) p (abs(mv.y - m_mvp.y) << 1) + !!(mv.y < m_mvp.y)
$1 = 41217
(gdb) p s_bitsizes[41217]
Cannot access memory at address 0x7ffff7fff414
(gdb) p mv.y
$2 = 0
(gdb) p m_mvp.y
$3 = 20608
(gdb) quit
[...]
$ hg branches 
default                     6539:dc700298419d
stable                      6513:eba8844609f2 (inactive)

[Steve Borho]

Thanks for re-testing. Would it be possible to see the contents of list[0] and list[1] at the point it crashes? This is a most interesting bug.

[Former user]

Well, here is what you want :

[...]
list[0] @ 0x7ffff3f144c0, list[1] @ 0x7fffc40308d0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff4716700 (LWP 19340)]
0x000000000061e0bc in x265::BitCost::bitcost (this=0x26bc770, mv=...) at /home/da/x265/source/encoder/bitcost.h:51
51                            s_bitsizes[(abs(mv.y - m_mvp.y) << 1) + !!(mv.y < m_mvp.y)] + 0.5f);
(gdb) p (MotionData)*0x7ffff3f144c0
$1 = {mv = {{{x = 0, y = 0}, word = 0}}, mvp = {{{x = 32767, y = 0}, word = 32767}}, mvpIdx = 6169680, ref = 0, cost = 0, bits = 0}
(gdb) p (MotionData)*0x7fffc40308d0
$2 = {mv = {{{x = 10, y = 54}, word = 3538954}}, mvp = {{{x = 70, y = 82}, word = 5374022}}, mvpIdx = 6488156, ref = 7274602, cost = 7864436, bits = 8388732}

Hope it is helpful.

[Steve Borho]

TEncSearch: validate unidirectional MVPs prior to trying MV0 bidir (closes #34)

This should fix some rare non-determinism as well as prevent the crash seen in issue #34. [CHANGES OUTPUTS]

→ <<cset e06f2a068622>>