Stop spawning python for better performance with mercurial

Issue #308 new
Sylvain Avril created an issue

If I have clearly understood the code, mercurial web access to the repositories is done by spawning a python process and using the mercurial hgweb CGI interface. Spawning a process at each user request is not very performant.

I have suggestions: 1. A possible enhancement would be to use jython and execute hgweb/mercurial directly in the JVM. 2. Another possibility would be to use a FastCGI interface to hgweb (with ?). 3. And the third option would be to setup an apache server with mod_wsgi outside SCM Manager and pass all request for mercurial to this server.

Option 1 would probably be better but would mean more work. Option 2 seems simplier but performant also.

Comments (5)

  1. Sebastian Sdorra repo owner


    You are understand the code correctly, for each user request a new mercurial process is created. I've done a lot of test to improve this behaviour, but this way seems to be the only way which works for every system. To your suggestions:

    1. I would love to use jython, but it does not work with mercurial. A lot of people have tried to run mercurial on jython (including me), but as far as i know all failed. I think the main problem is that mercurial uses c code for critical and special os operations. This c operations must be implemented in java and this seems to be a lot of work.

    2. For the current structure of mercurial and scm-manager, we have to start a fast-cgi process for each mercurial repository and this is not possible if you have a lot of mercurial repositories.

    3. This option would break the easy and all-in-one feeling of scm-manager.

    Another possible option is to keep the spawned mercurial process for more than one request and stop the process after a short time of inactivity.

  2. DRayX

    What about running ajp-wsgi and having scm-manager talk to the ajp-wsgi instance and then have it run mercurial through wsgi? Also, I don't see why Mercurial wouldn't work with Jython as it doesn't have any native code (as far as I can tell anyway, there are no C/C++ files in the source). I have run into python code that doesn't run under jython before, but it usually isn't too much work to add jython specific cases to handle the incompatibilities (might even be easier to get working against Jython 2.7). It seems like running mercurial with modjy would be the best option.

  3. Sebastian Sdorra repo owner

    Mercurial uses native code, have a look at base85.c or osutil.c for example. Mercurial uses native code for performance critical operations. It is possible to compile mercurial without the native extensions, but is a lot slower. I've done some tests with jython and mercurial in the past and i came to the same result as Martin Geisler from Aragost.

    I will have a look at ajp-wsgi.

  4. DRayX

    Ya, I noticed the c files not long after my post, and while a lot of them are just there for performance enhancementes, it would suck to remove them (plus the os specific stuff like the hardlink detection would be impossible without using JNI or JNA). Also, my other idea looks to be a bust as well since there is no Java AJP client as far as I can tell (which I find really surprising). There are other lightweight wsgi servers (including some pure python ones that could be run with no need to build native code), but a lot of them haven't seen much active development in the last couple years (to be fair, ajp-wsgi hasn't either).

  5. John Peacock

    Really SCMManager should be using the Mercurial CommandServer to talk over a pipe to Mercurial. That would allow you to ignore the fact that Mercurial is written in Python apart from actually starting the CommandServer process (one per repo?).

  6. Log in to comment