Crash in frameencoder when wavefront processing is disabled

[Former user]

It is found that when wavefront processing is disabled, frameencoder will crash due to out-of-boundary access to the m_outStreams array,

The crash flow is described below:

In frameencoder.cpp

in the FrameEncoder::compressFrame()

   uint32_t numSubstreams = m_param->bEnableWavefront ? slice->m_sps->numCuInHeight : 1;
    if (!m_outStreams)
    {
        m_outStreams = new Bitstream[numSubstreams];
        m_substreamSizes = X265_MALLOC(uint32_t, numSubstreams);

here the m_outStreams array is allocated with "numSubstreams" Bitstreams, according to the condition, when Wavefront is disabled, numSubstreams is 1, so accordingly the array length is 1.

In FrameEncoder::processRowEncoder() function

   if (bIsVbv)
        {  ........

               if (row == col && row)
              {
               ............
                        if (reEncode < 0)
                       { 
                            ..................

                              for (uint32_t r = m_numRows - 1; r >= row; r--)
                              {
                                   ..................

                                    m_outStreams[r].resetBits();
                                   stopRow.completed = 0;

as it can be seen from the code above, m_outStreams array is accessed without checking how many bitstreams are created. When wavefront is disabled, the array of length 1, and an access out of boundary occurred and then it crashed.

Of course as described above, it only happens when some conditions are met, so not every video or every frame will hit it.

(btw, a workaround would be using the parameters combination that always turn on wavefront processing, the encoding could then finish without problem)

Thanks!

[Deepthi Nandakumar]

Thanks. We're looking into this, but essentially it seems we cannot allow VBV row encode restarts (for stronger VBV control) when wpp is disabled.

Can you share your full commandline and source?

[Deepthi Nandakumar]

It is curious how you encountered such a situation. The encoder will fail to open successfully if VBV is enabled with no-wpp.

You are either a) Encoding with 1 CTU row or less than 3 CTU columns (wpp is turned off here, since it becomes unstable) b) Thread pool creation failed

[Former user]

After some checking, I found it is b) Thread pool creation failed, since my test specified "--pool 0" in parameter , and the number of numa nodes is 1 on my machine, so in the allocThreadPool() function eventually the thread pool creation failed (no thread pool created), and then wavefront processing is disabled because of this. This should be a way to trigger the problem.

Hope it helps, sorry having difficulty to share the source, but do let me know if you need any further information on this, thanks!

[Deepthi Nandakumar]

The only solution to this is to disable VBV if wpp gets disabled (with explicit warnings, of course). Pushing in a patch for this and closing the issue.