Forking of git repositories does not work on windows

Issue #135 closed
Former user created an issue

Error description

Forking a git repository on a windows system hangs in an infinite loop while displaying

Repository "test-fork" is being created, you will be redirected when this process is finished. 

The log reveals that the reason for this error is git failing to clone the repository due to incorrectly escaped repository paths.

2015-05-22 17:54:25.391 ERROR [kallithea.lib.vcs.backends.git.repository] Couldn't run git command (git -c core.quotepath=false clone -q --bare -- 'file:///c:\Kallithea2\Repo\test' 'c:\Kallithea2\Repo\test-fork').
Original error was:Subprocess exited due to an error:
fatal: could not create leading directories of ''c:\Kallithea2\Repo\test-fork''

On windows, git does not accept single quotation marks.

Cause of error

The cause of the error seems to be in lib/vcs/backends/git/repository.py. In this module pipes.quote is used in multilple places to escape repository paths. However the output of pipes.quote is only suitable for unix shells but nor for the windows shell.

Possible solution

A possible solution would be to replace lines 20 - 25 in the file repository.py, which are

try:
    # Python <=2.7
    from pipes import quote
except ImportError:
    # Python 3.3+
    from shlex import quote

by a platform independent version of quote. My suggestion would be:

import sys
mswindows = (sys.platform == "win32")

if mswindows:
    from subprocess import list2cmdline
    def quote(s):
        return list2cmdline([s])
else:
    try:
        # Python <=2.7
        from pipes import quote
    except ImportError:
        # Python 3.3+
        from shlex import quote

Kind regards, Matthias

System info:

Kallithea version:Z
    0.2.1 
Python version:
    2.7.9
Platform:
    Windows-8-6.2.9200
Git version:
    1.9.5
Git path:
    git
Upgrade info endpoint:
    Note: please make sure this server can access this URL.

Python Packages
amqplib         1.0.2
anyjson         0.3.3
Babel           1.3
Beaker          1.6.4
celery          2.2.10
decorator       3.4.2
docutils        0.11
dulwich         0.9.9
FormEncode      1.2.6
Kallithea       0.2.1
kombu           1.5.1
Mako            1.0.0
Markdown        2.2.1
MarkupSafe      0.23
mercurial       3.3.3
mock            1.0.1
nose            1.3.6
Paste           2.0.1
PasteDeploy     1.5.2
PasteScript     1.7.5
pip             6.1.1
pycrypto        2.6.1
Pygments        2.0.2
Pylons          1.0
pyparsing       1.5.7
python-dateutil 1.5
pytz            2015.2
repoze.lru      0.6
Routes          1.13
setuptools      15.0
simplejson      2.5.2
six             1.9.0
SQLAlchemy      0.7.10
Tempita         0.5.2
URLObject       2.3.4
waitress        0.8.8
WebError        0.10.3
WebHelpers      1.3
WebOb           1.0.8
WebTest         1.4.3
Whoosh          2.5.7

Comments (14)

  1. Mads Kiilerich

    The proposed patch has been applied to the stable branch.

    As discussed on https://bitbucket.org/conservancy/kallithea/pull-request/141/fix-for-issue-135-added-platform/diff:

    The need for this patch do however point at a root cause that requires a different and real solution - please see https://bitbucket.org/Unity-Technologies/kallithea/commits/38d5d9686ddf4101b63bc9aa3a77bbc77a665576 . I don't use git and can't give it much testing - please review and test it carefully.

    I thus don't consider this issue fully solved.

  2. Former user Account Deleted

    I reviewed the changes. Most changes are fine. Only the changes in kallithea.lib.middleware.pygrack are not OK. They break the ability to pull from the repository. This is not detected by the unit tests as pygrack.py is not covered. I had to revert the changes to get git remote operations working again (https://bitbucket.org/mkroll/kallithea/commits/1780906d945270101a5a15701dcb9e40a05b13d3) . I haven't found out yet what exactly was broken. I also found that the test kallithea.tests.vcs.test_git. GitRepositoryTest.test_git_cmd_injection is broken on Windows due to invalid characters in the tested path names (" is not allowed on Windows, fixed in https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7).

    I did some manual testing and ran the test suite under Ubuntu 14.04 and WIndows 8.1. The results of the manual tests are:

    repository creation ... OK
    adding file on frontend ... OK
    cloning ... OK
    pushing ... OK
    pushing to different branch ... OK
    forking ... OK
    coparing revisions ... OK
    comparing forks ... OK
    creating pull request ... OK
    fetching from fork ... OK
    viewing changesets ... OK
    viewing history ... OK
    downloading ZIP file ... Failure (created zip file is locked by other process -> download works on second try because zip file is cached)
    

    The unit tests under Ubuntu give 9 errors and 8 failure which are all unrelated to git. There is no regression compared to https://bitbucket.org/conservancy/kallithea/commits/c44885d0e546303772771141418c5c7d91e0180c.

    On Windows there are quite a lot of failling tests:

    test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error
    test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerGIT) ... Error
    test_create_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error
    test_delete_non_ascii (kallithea.tests.functional.test_admin_repos.TestAdminReposControllerHG) ... Error
    test_compare_forks_on_branch_extra_commits_origin_has_incomming_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
    test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
    test_org_repo_new_commits_after_forking_simple_diff_git (kallithea.tests.functional.test_compare.TestCompareController) ... Error
    test_org_repo_new_commits_after_forking_simple_diff_hg (kallithea.tests.functional.test_compare.TestCompareController) ... Error
    test_add_file_into_git_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure
    test_add_file_into_hg_2 (kallithea.tests.functional.test_files.TestFilesController) ... Failure
    test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_delete_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_delete_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_delete_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_delete_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_delete_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_edit_file_view_commit_changes_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_edit_file_view_commit_changes_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_edit_file_view_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_edit_file_view_not_on_branch_git (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_edit_file_view_not_on_branch_hg (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_raw_file_ok (kallithea.tests.functional.test_files.TestFilesController) ... Failure
    test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure
    test_clone_after_repo_was_locked_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_after_repo_was_locked_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_and_create_lock_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_and_create_lock_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_git_dir_as_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_hg_repo_as_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_hg_repo_by_admin (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_non_existing_path_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_non_existing_path_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_clone_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_ip_restriction_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_ip_restriction_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_back_to_wrong_url_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
    test_push_back_to_wrong_url_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_invalidates_cache_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
    test_push_invalidates_cache_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_new_file_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
    test_push_new_file_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_on_locked_repo_by_other_user_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
    test_push_on_locked_repo_by_other_user_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_unlocks_repository_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_unlocks_repository_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_push_wrong_credentials_git (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Error
    test_push_wrong_credentials_hg (kallithea.tests.other.manual_test_vcs_operations.TestVCSOperations) ... Failure
    test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure
    test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.GitFileNodeUnicodePathTest) ... Error
    test_filenode_path (kallithea.tests.vcs.test_filenodes_unicode_path.HgFileNodeUnicodePathTest) ... Error
    test_mimetype (kallithea.tests.vcs.test_nodes.NodeBasicTest) ... Failure
    test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error
    

    Compared to https://bitbucket.org/conservancy/kallithea/commits/c44885d0e546303772771141418c5c7d91e0180c there are the following regressions:

    test_archival (kallithea.tests.functional.test_files.TestFilesController) ... Error
    test_diff_markup (kallithea.tests.models.test_diff_parsers.DiffLibTest) ... Failure
    test_archive_default_stream (kallithea.tests.vcs.test_archives.HgArchiveTest) ... Failure
    test_get_repo_multialias (kallithea.tests.vcs.test_vcs.VCSTest) ... Error
    

    It seems to me that none of those is related to to the changes in the git code.

    I would say that https://bitbucket.org/mkroll/kallithea/commits/4558ca5e6a275b2784054061389c1ea66701a0a7 shoulfd be fine. However, the reason why the changes to pygrack.py are not working needs to be investigated. Also the large number of failing tets on windows needs further attention.

  3. Mads Kiilerich

    Thanks for the thorough testing!

    Did the manual download of zip fail both on windows and ubuntu?

    It is weird you see failures on ubuntu. I don't see any failures. That should be investigated separately.

    The failures on Windows are more likely. We try to be cross platform but most of the contributors are on some kind of unix.

  4. Former user Account Deleted

    Yes, works both on Windows and Ubuntu.

    Btw: Zip downloading works on Ubuntu. Apparently python does not lock the file for other processes o Linux but only on windows.

  5. Mads Kiilerich

    Just to clarify: This is an unrelated problem, right? zip downloading was broken on windows before too?

    Will this fix it:

    --- a/kallithea/controllers/files.py
    +++ b/kallithea/controllers/files.py
    @@ -566,2 +566,3 @@ class FilesController(BaseRepoController
                 fd, archive = tempfile.mkstemp()
    +            os.close(fd)
                 temp_stream = open(archive, 'wb')
    @@ -582,4 +583,2 @@ class FilesController(BaseRepoController
                         stream.close()
    -                    if fd:  # fd means we used temporary file
    -                        os.close(fd)
                         if not archive_cache_enabled:
    
  6. Former user Account Deleted

    Yes, it is unrelated and was broken before. It just turned up while testing. The patch is working.

  7. Mads Kiilerich

    I think this has been solved now and can be closed. Please follow up on other issues separately.

  8. Log in to comment