Issue #135 resolved

Mercurial push error with very large changesets

Anonymous avatarAnonymous created an issue

I'm novice at programming.

I wanna make remote repository by scm-manager and tortoiseHg.

I tried to push large file(about 450 MB).

After uploading progress bar in status bar is finished, error message appear in output log.

---------------------------------------------------------------

clone% % hg --repository C:\Users\xxx\Desktop\clone push http://localhost:8080/scm/hg/Repo pushing to http://localhost:8080/scm/hg/Repo http authorization required realm: SONIA :: SCM Manager searching for changes 'http://localhost:8080/scm/hg/Repo' does not appear to be an hg repository:

---%<--- (no content-type)

---%<---

[command returned code 255 Thu Apr 26 18:46:45 2012]

----------------------------------------------------------------

I wrote in clone's hgrc. hgrc code is...

----------------------------------------------------------------

[paths]

default = http://localhost:8080/scm/hg/Repo

[extensions]

largefiles =

-----------------------------------------------------------------

Why appear this error message??

Comments (19)

  1. Sebastian Sdorra

    Are you able to push a small changeset? Could you post the scm-manager log for such a push?

    It is possible that you create an bitbucket account? because the notification system of bitbucket does not notify on anonymous comments.

  2. DRayX

    I have seen this error in TortoiseHG when a hook or extension throws an exception. Is there any chance you could try doing a local push?

  3. Anonymous

    I'm sorry because I reply so late..

    Thanks Sdorra & DRayX.

    I solved by writing in .hgrc file of each repositories.

    [extensions]

    largefiles =

    .

    But I saw again other case.

    I want to push project that has many files(about 40000 files).

    Size of this project is about 11GB.

    I push this project, and I saw same message after sending all files.

    After about 25 minutes when I try to push, message is appear in cmd window.

    Message is "파이프가 끝났습니다." in Korean. I think "The pipe has been ended" in English.

    I also saw "no repository request listener defined"

    But this project has being sent to repository.

    After finish sending, I can see this message.

    I think connection time limit is over.

    Is there any way to push at once?

  4. Anonymous

    I saw Issue #14. There was a comment.

    Could you please edit the scm-server/bin/scm-server.bat. Search the the following line:
    
    set EXTRA_JVM_ARGUMENTS=
    
    and change this line to:
    
    set EXTRA_JVM_ARGUMENTS= "-Dsun.net.client.defaultConnectTimeout=300000 -Dsun.net.client.defaultReadTimeout=300000"
    
    Restart the scm-server and test the plugin page again.
    

    I think this is a way that extend connection timeout. So I modified that:

    set EXTRA_JVM_ARGUMENTS=-Dlogback.configurationFile=logging.xml "-Dsun.net.client.defaultConnectTimeout=7200000 -Dsun.net.client.defaultReadTimeout=7200000 -Dsun.net.client.defaultWriteTimeout=7200000"
    

    I hope to connect during push time about 2 hours.

    I try to push by --debug.

    This is output.

    clone% % hg --repository C:\Users\Test\Desktop\clone push --debug http://localhost:8080/scm/hg/LocalTest
    pushing to http://localhost:8080/scm/hg/LocalTest
    using http://localhost:8080/scm/hg/LocalTest
    sending capabilities command
    http authorization required
    realm: SONIA :: SCM Manager
    http auth: user scmadmin, password ********
    query 1; heads
    sending batch command
    http auth: user scmadmin, password ********
    searching for changes
    1 total queries
    sending statlfile command
    http auth: user scmadmin, password ********
    sending statlfile command
    http auth: user scmadmin, password ********
    sending statlfile command
    http auth: user scmadmin, password ********
    sending statlfile command
    http auth: user scmadmin, password ********
    sending statlfile command
    http auth: user scmadmin, password ********
    query 1; heads
    sending batch command
    http auth: user scmadmin, password ********
    searching for changes
    1 total queries
    1 changesets found
    list of changesets:
    ae95f5360032170b0fb154cea919889e9e3eb79d
    sending unbundle command
    sending 12144923510 bytes
    http auth: user scmadmin, password ********
    requested URL: 'http://localhost:8080/scm/hg/LocalTest?cmd=unbundle'
    'http://localhost:8080/scm/hg/LocalTest' does not appear to be an hg repository:
    ---%<--- (no content-type)
    
    ---%<---
    
    [command returned code 255 Wed May 02 17:41:55 2012]
    clone% 
    
  5. Anonymous

    Hi, Some problem. Tried to push a ZIP (205MB). python.exe went nuts. Now I get the Error:

    %repositoryPath% does not appear to be an hg repository:
    ---%<--- (no content-type)
    
    ---%<---
    
  6. Novice

    OS == Windows7 Ultimate K 64bit Service Pack 1

    RAM == 8GB

    SCM-Manager == 1.14

    TortoiseHg == 2.3.1

    Mercurial == 2.1.1

    Python == 2.6.6 64bit

    I installed Mercurial 2.1.2 Python 2.6 package - x64 Windows in http://mercurial.selenic.com/downloads/.

    I tried version check by --version.

    I read HandlingLargeFiles page. But I didn't be understand very well.

    I was read these word first time. : diff/delta algorithm, revlogs, dirstate, mmap, magic string, etc.

    So this problem won't be solved yet.

  7. Sebastian Sdorra

    Is scm-manager running on the same machine as your mercurial client? Could you please monitor your memory during a push? Because mercurial keeps most of the files in memory and convert binaries to text.

    What are you trying to push? Keep in mind that mercurial (and scm-manager) are primary made for handling text files. Perhaps you are using the wrong tools for your need.

  8. Novice

    I tried to push binary file and source code.

    I thought extensions 'largefiles' is able to push not only source code but also binary files.

    Isn't it? Do I understand wrong?

    My large project is 3D project.

    I saw resource monitor, pushing this project.

    Most memory space was changed to waiting mode.

    Remain memory space is above 20MB.

    Sometimes it is 0MB.

    Do I succeed if I push this project partially? And do I succeed if I try to pull whole project?

    I want to use scm-manager and TortoiseHg.

  9. Sebastian Sdorra

    Yes, you are right the largefiles extension is for large binary files. Yesterday if tried to reproduce this error, but i was able to push a 6GB big changeset without a problem. It takes a long time but it works. Could please monitor the memory of the java process on the server, the python process on the server and the python process on the client.

    You could try to push it partially, i don't know if it will work.

  10. DRayX

    Realted issue, it would be nice if errors were propagated instead of just getting

    ${REPOSITORY} does not appear to be an hg repository:
    ---%<--- (no content-type)
    
    ---%<---
    
  11. Novice

    I tried to push 11GB project partially. I divided it into 5 parts.

    Each parts is 1.2GB ~ 1.8GB. I repeated commit and push 5 times.

    So I succeeded!!! I also succeeded pulling once!!!

    I think memory of server and client is not enough.

    I hope that anyone who try to push large changesets can do that once.

    Please improve this problem if you can do it.

  12. DRayX

    To someone who is seeing this issue, try commenting out the scm manager hooks from the repo's hgrc, run hg serve (not as a daemon) in the root of the repository, and try breaking push to port 8000 on the machine. I'm guessing (not sure) that the push will fail, but you should get a python stack trace from the running hg serve that could illuminate the cause of this issue.

  13. DRayX

    Ok, I had a very similar issue with large pushes when using the Tomcat servlet container. My problem was that one of the mercurial GET requests encodes 512 changesets being pushed into an http header by simply concatenating the full hashes together. Since each hash takes 40 bytes, this header is going to be at least 20K, significantly larger than the 4K default maxHttpHeaderSize set by Tomcat. By simply setting my maxHttpHeaderSize to 32K (32768) in the Connector section of the server.xml file, I was able to fix my issue. I'm not sure if others are having the same issue or not as I can't find a similar limit in Jetty (however it does have a headerBufferSize that defaults to 4K).

  14. Log in to comment
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.