Issue #2119 resolved

Crash on large pull of hgsubversion repo

FlorianGeorge
created an issue

When pulling a large amount of commits (over 1000), tortoiseHg crashed for me twice so far. One time around 300/1000, the second time at 440/821.

The Output Log Window (whole window is greyed out but the visible part is still readable) doesn't say something special: {{{ [r14762] user: M trunk/.../.../abc.cpp M trunk/.../.../abc.h committed to "default" as 2b0f4752e886 [r14762] user: commit comment }}}

I have looked around for a log file, but I can't find one.

When thgw.exe crashed, I could select "Debug", so I took a look with Visual Studio 2008.

Call Stack is attached as "crash call stack 120830-1.txt".

The line where it crashed is calloc_impl.c, line 91 {{{ if (pvReturn == NULL) --> pvReturn = HeapAlloc(_crtheap, HEAP_ZERO_MEMORY, size); }}}

Local Variables: {{{ num 18415616 unsigned int size 0 unsigned int + errno_tmp 0x00001e60 int size_orig 18415616 unsigned int pvReturn 0x00000000 void _Expr_val 18415616 int }}}

I'm running Windows 7 SP1 x86, tortoiseHg 2.4.3, hgSubversion newest repo version (Changeset: 924 (e252f9355933) test_fetch_command: fix if local encoding is not UTF-8).

Any other information that could help?

Comments (13)

  1. FlorianGeorge reporter

    I ran thg in a command prompt with admin rights, THGDEBUG=1 and [ui]traceback=1

    The Output Log ended in

    ...
    _Unsorted/Release/lib/external/xerces-c_static_3D.lib
    transaction abort!
    rollback completed
    
    [command returned code 255 Thu Aug 30 16:22:48 2012]
    DevCrashTest%
    

    The console ended with this output (above that only GarbageCollector stuff):

    Traceback (most recent call last):
      File "tortoisehg\hgqt\thread.pyo", line 274, in run
      File "tortoisehg\util\hglib.pyo", line 724, in dispatch
      File "mercurial\dispatch.pyo", line 740, in _dispatch
      File "mercurial\dispatch.pyo", line 514, in runcommand
      File "mercurial\dispatch.pyo", line 830, in _runcommand
      File "mercurial\dispatch.pyo", line 801, in checkargs
      File "mercurial\dispatch.pyo", line 737, in <lambda>
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 567, in generic
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "hgext\rebase.pyo", line 700, in pullrebase
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\commands.pyo", line 4607, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 81, in wrapper
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 104, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 386, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\replay.py", line 189, in convert_rev
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 91, in svn_commitctx
      File "mercurial\localrepo.pyo", line 1419, in commitctx
      File "mercurial\localrepo.pyo", line 1214, in _filecommit
      File "mercurial\filelog.pyo", line 46, in add
    MemoryError
    

    Tried again, without restarting, just hit the Pull button again, got this traceback:

    GarbageCollector.check: 4780 0 11
    collected gen 0, found 299 unreachable
    GarbageCollector.check: 11436 1 11
    collected gen 0, found 0 unreachable
    GarbageCollector.check: 64960 2 11
    collected gen 0, found 0 unreachable
    GarbageCollector.check: 49493 3 11
    collected gen 0, found 0 unreachable
    Traceback (most recent call last):
      File "tortoisehg\hgqt\thread.pyo", line 274, in run
      File "tortoisehg\util\hglib.pyo", line 724, in dispatch
      File "mercurial\dispatch.pyo", line 740, in _dispatch
      File "mercurial\dispatch.pyo", line 514, in runcommand
      File "mercurial\dispatch.pyo", line 830, in _runcommand
      File "mercurial\dispatch.pyo", line 801, in checkargs
      File "mercurial\dispatch.pyo", line 737, in <lambda>
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 567, in generic
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "hgext\rebase.pyo", line 700, in pullrebase
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\commands.pyo", line 4607, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 81, in wrapper
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 104, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 386, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\replay.py", line 78, in convert_rev
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnwrap\svn_swig_wrapper.py", line 414, in get_replay
      File "libsvn\ra.pyo", line 860, in svn_ra_replay
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnwrap\svn_swig_wrapper.py", line 84, in fun
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\editor.py", line 272, in add_directory
      File "mercurial\context.pyo", line 498, in data
      File "mercurial\filelog.pyo", line 38, in read
      File "mercurial\revlog.pyo", line 901, in revision
      File "mercurial\revlog.pyo", line 824, in _chunkbase
      File "mercurial\revlog.pyo", line 821, in _chunk
      File "mercurial\revlog.pyo", line 87, in decompress
    MemoryError
    

    Tried a third time without restarting, just hit the Pull button again, then it went down in flames:

    GarbageCollector.check: 6590 4 11
    collected gen 0, found 236 unreachable
    GarbageCollector.check: 45499 5 11
    collected gen 0, found 0 unreachable
    GarbageCollector.check: 34001 6 11
    collected gen 0, found 0 unreachable
    Traceback (most recent call last):
      File "tortoisehg\hgqt\thread.pyo", line 274, in run
      File "tortoisehg\util\hglib.pyo", line 724, in dispatch
      File "mercurial\dispatch.pyo", line 740, in _dispatch
      File "mercurial\dispatch.pyo", line 514, in runcommand
      File "mercurial\dispatch.pyo", line 830, in _runcommand
      File "mercurial\dispatch.pyo", line 801, in checkargs
      File "mercurial\dispatch.pyo", line 737, in <lambda>
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 567, in generic
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "hgext\rebase.pyo", line 700, in pullrebase
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\commands.pyo", line 4607, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 81, in wrapper
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 104, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 386, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\replay.py", line 78, in convert_rev
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnwrap\svn_swig_wrapper.py", line 414, in get_replay
      File "libsvn\ra.pyo", line 860, in svn_ra_replay
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnwrap\svn_swig_wrapper.py", line 84, in fun
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\editor.py", line 272, in add_directory
      File "mercurial\context.pyo", line 498, in data
      File "mercurial\filelog.pyo", line 38, in read
      File "mercurial\revlog.pyo", line 899, in revision
      File "mercurial\revlog.pyo", line 818, in _chunkraw
      File "mercurial\revlog.pyo", line 811, in _getchunk
      File "mercurial\revlog.pyo", line 792, in _loadchunk
    MemoryError
    Qt has caught an exception thrown from an event handler. Throwing
    exceptions from an event handler is not supported in Qt. You must
    reimplement QApplication::notify() and catch all exceptions there.
    
    
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    
  2. FlorianGeorge reporter

    Tried again, this time I noticed in the taskmanager that memory consumption started to go through the roof.

    Here's the footprint towards the end:

    Pull 1180/9591 Thg.exe 1.468.484 K Memory Allocated

    Pull 1276/9591 Thg.exe 1.620.324 K Memory Allocated

    Crash at

    Pull 1375/9591 Thg.exe 1.766.264 K Memory Allocated

    At the point of the crash, my free memory was at 0 (zero). I guess this issue would take a lot more time for the app to crash in a 64bit environment.

    A Bug reporter opened, but greyed out as the whole application had crashed, so I took a screenshot and attached it as Bug Reporter 120830- 1.png.

    The complete traceback was mirrored in the console:

    Traceback (most recent call last):
      File "tortoisehg\hgqt\repowidget.pyo", line 839, in onRevisionSelected
      File "tortoisehg\hgqt\revdetails.pyo", line 276, in onRevisionSelected
      File "tortoisehg\hgqt\csinfo.pyo", line 426, in update
      File "tortoisehg\hgqt\csinfo.pyo", line 339, in get_markup
      File "tortoisehg\hgqt\csinfo.pyo", line 285, in get_markup
      File "tortoisehg\hgqt\csinfo.pyo", line 225, in get_data
      File "tortoisehg\hgqt\revpanel.pyo", line 67, in data_func
      File "mercurial\context.pyo", line 190, in branch
      File "mercurial\util.pyo", line 246, in __get__
      File "mercurial\context.pyo", line 137, in _changeset
      File "mercurial\changelog.pyo", line 193, in read
      File "mercurial\revlog.pyo", line 903, in revision
      File "mercurial\revlog.pyo", line 821, in _chunk
      File "mercurial\revlog.pyo", line 87, in decompress
    MemoryError
    Traceback (most recent call last):
      File "tortoisehg\hgqt\thread.pyo", line 274, in run
      File "tortoisehg\util\hglib.pyo", line 724, in dispatch
      File "mercurial\dispatch.pyo", line 740, in _dispatch
      File "mercurial\dispatch.pyo", line 514, in runcommand
      File "mercurial\dispatch.pyo", line 830, in _runcommand
      File "mercurial\dispatch.pyo", line 801, in checkargs
      File "mercurial\dispatch.pyo", line 737, in <lambda>
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 567, in generic
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\extensions.pyo", line 144, in wrap
      File "mercurial\util.pyo", line 472, in check
      File "hgext\rebase.pyo", line 700, in pullrebase
      File "mercurial\util.pyo", line 472, in check
      File "mercurial\commands.pyo", line 4607, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 81, in wrapper
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 104, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\wrappers.py", line 386, in pull
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\replay.py", line 189, in convert_rev
      File "V:\Tools, Libraries, Ressources\SVN & CVS Tools\hgsubversion\hgsubversion\svnrepo.py", line 91, in svn_commitctx
      File "mercurial\localrepo.pyo", line 1407, in commitctx
      File "mercurial\context.pyo", line 171, in manifest
      File "mercurial\util.pyo", line 246, in __get__
      File "mercurial\context.pyo", line 141, in _manifest
      File "mercurial\manifest.pyo", line 57, in read
    MemoryError
    Qt has caught an exception thrown from an event handler. Throwing
    exceptions from an event handler is not supported in Qt. You must
    reimplement QApplication::notify() and catch all exceptions there.
    
    Qt has caught an exception thrown from an event handler. Throwing
    exceptions from an event handler is not supported in Qt. You must
    reimplement QApplication::notify() and catch all exceptions there.
    
    
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    
  3. Yuya Nishihara

    Is the remote repository SVN? If so, can you test it with command-line hg client for comparison?

    I heard before that hgsubversion tends to consume plenty of memory for large pull.

  4. FlorianGeorge reporter

    Yes, the repository is SVN.

    I ran a

    hg pull --debug --trace
    

    while parallel opening thg from a command prompt with admin rights, THGDEBUG=1 and [ui]traceback=1. This caused thg continuously updating the graphical view of the repository, so I could see the pull and the UI update now in different processes in the task manager.

    At pull 872/8229, the memory consumption was

    hg.exe 415.188 K

    thb.exe 839.020 K

    This was during a pull stall, when the pull continued, the memory consumption of hg.exe dropped down to around 100.000 K again, while thg.exe continued to increase its footprint.

    So the memory consumption of hg.exe/hgSubversion seems to increase indeed during the pull of a single commit, but is freed again when the next commit is pulled.

    At pull 1055/8229, the memory consumption was

    hg.exe 99.128 K

    thg.exe 1.153.108 K

    Somewhere around pull 1970/8229, thg.exe finally crashed, while hg.exe continued to pull from the repository.

    The memory consumption of thg.exe had reached 1.932.156 K at that point.

    The command prompt running thg.exe showed the output

    ...
    locked, aborting
    locked, aborting
    locked, aborting
    locked, aborting
    detected repository change
    Traceback (most recent call last):
      File "tortoisehg\hgqt\repowidget.pyo", line 967, in repositoryChanged
      File "tortoisehg\hgqt\repowidget.pyo", line 899, in rebuildGraph
      File "mercurial\localrepo.pyo", line 361, in __len__
      File "mercurial\scmutil.pyo", line 898, in __get__
      File "mercurial\localrepo.pyo", line 319, in changelog
      File "mercurial\changelog.pyo", line 115, in __init__
      File "mercurial\revlog.pyo", line 215, in __init__
    MemoryError
    add file to watcher: D:\User\DevCrashTest\.hg/phaseroots
    locked, aborting
    locked, aborting
    locked, aborting
    ...
    

    After that, in the command line, the "locked, aborting" message, as well as a single "add file to watcher: D:\User\DevCrashTest\.hg/phaseroots" message were going on for about 70 more lines, then it stopped too.

    So it appears to me that the tortoiseHg GUI thg.exe/thgw.exe is the cause of the steadily increasing memory consumption, while hg.exe/hgSubversion are not to blame.

  5. Yuya Nishihara

    Thanks, your investigation is very solid.

    The behavior of hgsubversion, which commits each pulled changeset?, kicks our bug. Probably "refresh repository" of TortoiseHg keeps reference to garbage.

  6. FlorianGeorge reporter

    I tested the new tortoiseHg 2.5 version.

    Memory consumption seemed to increase now at a low linear rate:

    Pull --- Memory consumption thg.exe

    0 --- 50.000 K

    400 --- 60.000 K

    800 --- 70.000 K

    1200 --- 80.000 K

    1600 --- 90.000 K

    At this rate, it appears that I would need a repository with 64000 revisions to reach 1650.000 K memory consumption, so I guess we are safe.....for now ;)

    Marked as resolved.

  7. Steve Borho

    The subversion Python bindings have notorious memory leaks. You should pull in a fixed number of revisions at a time, if possible. This is especially important on new clones from a subversion repo.

  8. Log in to comment