durin42 / hgsubversion (http://groups.google.com/group/hgsubversion/)

hgsubversion is an extension for Mercurial that allows using Mercurial as a Subversion client.

Clone this repository (size: 951.1 KB): HTTPS / SSH
$ hg clone http://bitbucket.org/durin42/hgsubversion/
follow

#110 Memory leak

Reported by VZ, created 7 months ago, last edited unknown.

I'm trying out hgsubversion under Debian, here is the version information:

% uname -msr
Linux 2.6.29-acct x86_64
% hg svn version
hg: 1.3
svn bindings: 1.5.1
hgsubversion: ff69f1855086

I want to import a large (60000 revisions) remote svn repository and the import seems to run out of memory, at least this is what I see

% time hg clone https://svn.wxwidgets.org/svn/wx/wxWidgets/ hg
...
[1]   30423 killed     hg clone https://svn.wxwidgets.org/svn/wx/wxWidgets/ hg
hg clone https://svn.wxwidgets.org/svn/wx/wxWidgets/ hg
[~12 hours total running time, ~30000 revisions pulled]
% time hg pull -u https://svn.wxwidgets.org/svn/wx/wxWidgets/
...
[r47525] FM: fix an issue with GTK filedialog (already fixed in the trunk)
[2]    31608 killed     hg pull -u https://svn.wxwidgets.org/svn/wx/wxWidgets/
hg pull -u https://svn.wxwidgets.org/svn/wx/wxWidgets/
13459.60s user 424.80s system 41% cpu 9:19:03.82 total

It seems that this happens because it runs out of memory, at least looking at munin statistics for the machine it's running on I see that its RAM consumption topped at 8GB shortly before it crashed.

Looking at %MEM column in top I see that it starts at ~1% and seems to grow quite quickly, at least it's already 6% after importing just a couple of dozen of revisions.

This might be not a huge problem if there are not many revisions to clone/pull but it literally brings my (4 CPU with 4GB of physical RAM) machine to its knees :-(

Followers: 2
Status: new Responsible: nobody Type: bug
Milestone: none Component: none Version: none

Attachments

No attachments added for this issue yet.

Comments and changes

#1 Anonymous

written 6 months ago, last edited unknown.

This is a known problem. You could just clone and kill it everytime it grows to large, then pull from inside the clone to resume the cloning. This is a proven workaround.


#2

Dirkjan Ochtman / djc
written 6 months ago, last edited unknown.

(That was me, BTW.)


#3

VZ
written 6 months ago, last edited unknown.

Thanks for confirming it, I should have probably updated the report myself to indicate that I was indeed able to finish cloning by redoing pull a few times (4 IIRC). I guess this mustn't be trivial to fix but maybe it's worth mentioning this issue in the README and, ideally, maybe there should be an option telling it to stop after cloning/pulling a certain number of revisions (10000 seems to be about the maximum, realistically).

Also, just as a comparison point, I used git-svn to check if this was an issue with repository or not and it finished in 10 hours IIRC (compared to 24 for hgsubversion) and never consumed any noticeable amount of RAM.


#4

rockhymas
written 5 months ago, last edited unknown.

Thanks for the workaround. Since this is a known problem, has anyone investigated? I can start looking into it, if I have some idea what to start looking at.


#5

Augie Fackler / durin42
written 4 months ago, last edited unknown.

I'm not currently investigating this. Please go ahead and take a look.


#6

Augie Fackler / durin42
written 2 months ago, last edited unknown.

The new workaround is to do something like this:

hg init foo
cd foo
hg pull <svn url>
# wait for OOM
# (repeat hg pull as necessary)

Add comment / attachment

Show/hide preview

Verification: Please write the text from the image in the box (letters only)

captcha

Is that you, Humanoid? Is this me?