Crash in TComPicSym class when out-of-memory happens during encoding.

[Former user]

There're cases when memory is not enough, a crash will happen in TComPicSym::destroy(), since some entries in m_cuData are uninitialized garbage value, the suggested fix is to add memset, and when destroy, check for each entry for validity before deleting it. ( concern is that may need to see how much it will affect the performance though)

the code is at source/Lib/TLibCommon/TComPicSym.cpp

Tested fix ( in diff ) is as follows:

@@ -86,6 +86,8 @@
     if (!m_slice || !m_cuData)
         return false;

+   memset(m_cuData,0, m_numCUsInFrame*sizeof(TComDataCU*));
+
     for (i = 0; i < m_numCUsInFrame; i++)
     {
         m_cuData[i] = new TComDataCU;
@@ -105,8 +107,11 @@

     for (int i = 0; i < m_numCUsInFrame; i++)
     {
-        m_cuData[i]->destroy();
-        delete m_cuData[i];
+       if(m_cuData[i])
+       {
+           m_cuData[i]->destroy();
+           delete m_cuData[i];
+       }
     }

     delete [] m_cuData;

[Steve Borho]

TComDataCU has a virtual destructor (it probably doesn't need one), but the presence of a vtable means we can't just memset the whole allocated array of them to zero. You'll need to fix the TComDataCU constructor to initialize all its pointers correctly.

Anyway, it appears the refactor of TComDataCU that Ashok performed has fixed these problems, for the most part. Adding one last check in <<cset 539e94e7aa34>>