Corrupted stat file on cpu with many threads

[Former user]

x265 sometimes creates corrupted stat file Two examples of corrupted stat files http://pastebin.com/KBeCMgyi http://pastebin.com/8UKx43Lc

Screenshot http://i.cubeupload.com/Dp2PuQ.png

[Aarthi Thirumalai]

suspect the rate control lock not being enforced while flushing the last few frames in RateControlEnd() is causing this issue. In multipass encodes, the RC lock must ensure ordering in RCEnd for all frames.

[Ma0]

It looks like this bug is only with MinGW (I'm able to reproduce it with -F 5 option), so we should make changes only for GCC && Windows, for example:

*** ratecontrol-o.cpp   Sat Jun 27 07:48:04 2015
--- ratecontrol.cpp Sat Jun 27 08:00:51 2015
***************
*** 2213,2219 ****
          char cType = rce->sliceType == I_SLICE ? (rce->poc > 0 && m_param->bOpenGOP ? 'i' : 'I')
              : rce->sliceType == P_SLICE ? 'P'
              : IS_REFERENCED(curFrame) ? 'B' : 'b';
!         if (fprintf(m_statFileOut,
                      "in:%d out:%d type:%c q:%.2f q-aq:%.2f tex:%d mv:%d misc:%d icu:%.2f pcu:%.2f scu:%.2f ;\n",
                      rce->poc, rce->encodeOrder,
                      cType, curEncData.m_avgQpRc, curEncData.m_avgQpAq,
--- 2213,2223 ----
          char cType = rce->sliceType == I_SLICE ? (rce->poc > 0 && m_param->bOpenGOP ? 'i' : 'I')
              : rce->sliceType == P_SLICE ? 'P'
              : IS_REFERENCED(curFrame) ? 'B' : 'b';
!     #if _WIN32 && __GNUC__
!         static Lock lockFile;
!         lockFile.acquire();
!     #endif
!         int fprintfRet = fprintf(m_statFileOut,
                      "in:%d out:%d type:%c q:%.2f q-aq:%.2f tex:%d mv:%d misc:%d icu:%.2f pcu:%.2f scu:%.2f ;\n",
                      rce->poc, rce->encodeOrder,
                      cType, curEncData.m_avgQpRc, curEncData.m_avgQpAq,
***************
*** 2222,2228 ****
                      curFrame->m_encData->m_frameStats.miscBits,
                      curFrame->m_encData->m_frameStats.percent8x8Intra * m_ncu,
                      curFrame->m_encData->m_frameStats.percent8x8Inter * m_ncu,
!                     curFrame->m_encData->m_frameStats.percent8x8Skip  * m_ncu) < 0)
              goto writeFailure;
          /* Don't re-write the data in multi-pass mode. */
          if (m_param->rc.cuTree && IS_REFERENCED(curFrame) && !m_param->rc.bStatRead)
--- 2226,2236 ----
                      curFrame->m_encData->m_frameStats.miscBits,
                      curFrame->m_encData->m_frameStats.percent8x8Intra * m_ncu,
                      curFrame->m_encData->m_frameStats.percent8x8Inter * m_ncu,
!                     curFrame->m_encData->m_frameStats.percent8x8Skip  * m_ncu);
!     #if _WIN32 && __GNUC__
!         lockFile.release();
!     #endif
!         if (fprintfRet < 0)
              goto writeFailure;
          /* Don't re-write the data in multi-pass mode. */
          if (m_param->rc.cuTree && IS_REFERENCED(curFrame) && !m_param->rc.bStatRead) 

[Ma0]

From MinGW-w64 sources:

 * Compiling this module with "-D INSTALL_AS_DEFAULT" will change this
 * recommended convention, such that references to "fprintf()" in user
 * code will ALWAYS be redirected to "__mingw_fprintf()"; if this option
 * is adopted, then users wishing to use the MSVCRT implementation of
 * "fprintf()" will be forced to use a "back-door" mechanism to do so.
 * Such a "back-door" mechanism is provided with MinGW, allowing the
 * MSVCRT implementation to be called as "__msvcrt_fprintf()"; however,
 * since users may not expect this behaviour, a standard libmingwex.a
 * installation does not employ this option. 

This mean that always fprintf = __mingw_fprintf and it is not possible to use fprintf from msvcrt.dll. __mingw_fprintf is not atomic, so we can restrict threads or use atomic function, for example fputs. We can divide fprintf to sprintf(buf, ...) && fputs(buf, m_statFileOut).

[Deepthi Nandakumar]

Thanks for your inputs on this. A fix should be in soon.

[Selur]

any news on this? Got a two users which end up with: x265 [error]: statistics are damaged at line 11465, parser out=0 x265 [error]: failed to open encoder messages

[Ma0]

This bug is in Mingw-w64 code, not in x265 code. Patch proposal to mingw_pformat.c source file is waiting for review. If you can't wait, you can use MSVC builds or patched mingw-w64 builds, for example my builds www.msystem.waw.pl/x265/ Or you can recompile mingw-w64 with this patch (and then build x265):

*** mingw_pformat.c.org Thu Jul  9 23:51:32 2015
--- mingw_pformat.c Tue Jul 14 14:43:33 2015
***************
*** 64,69 ****
--- 64,71 ----
  #include <limits.h>
  #include <locale.h>
  #include <wchar.h>
+ #include <intrin.h>
+ #include <synchapi.h>

  #ifdef __ENABLE_DFP
  #ifndef __STDC_WANT_DEC_FP__
***************
*** 2284,2289 ****
--- 2286,2297 ----
                                                     -1 means to be determined. */
    };

+   if( flags & PFORMAT_TO_FILE )
+     while( InterlockedBitTestAndSet( (volatile __LONG32*)&((FILE*)dest)->_flag, 27 ) )
+       do
+         Sleep( 0 );
+       while( (volatile __LONG32)(((FILE*)dest)->_flag) & (1 << 27) );
+ 
    format_scan: while( (c = *fmt++) != 0 )
    {
      /* Format string parsing loop...
***************
*** 3105,3110 ****
--- 3113,3121 ----
    /* When we have fully dispatched the format string, the return value is the
     * total number of bytes we transferred to the output destination.
     */
+   if( flags & PFORMAT_TO_FILE )
+     ((FILE*)dest)->_flag &= ~(1 << 27);
+ 
    return stream.count;
  }

[Selur]

Thanks for the clarification and the update. :)

[Steve Borho]

Issue #159 was marked as a duplicate of this issue.

[Steve Borho]

We believe this is a race-hazard in the ordering of the final file-writes. We have a fix planned for it that should land in the stable branch soon.

[Deepthi Nandakumar]

rc: fix corrupted stat file issue in multipass encodes (ref #147)

→ <<cset 7ef88e31062e>>

[Steve Borho]

we believe this is resolved. please re-open if you find it reproducible with current tip