Issue #210 invalid

empty changesets cloning ImapPusherService repository

Vsevolod Parfenov
created an issue

Link ImapPusherService to repo is https://ImapPusherService.svn.codeplex.com/svn Web page is http://imappusherservice.codeplex.com/ Commands to clone: {{{ hg clone svn+https://ImapPusherService.svn.codeplex.com/svn ImapPusherService }}} and {{{ hg clone --layout single svn+https://ImapPusherService.svn.codeplex.com/svn ImapPusherService }}}

After cloning only Initial commit has changes (rev 0). All other is empty except the last one with comment "Checked in by server upgrade"

Browsing repository in web shows that changesets are not empty.

Comparision of downloaded sources and workdir of cloned repo shows the difference.

Comments (23)

  1. Anton Afanasyev

    You may or may not need this anymore, but others undoubtedly will. I ran into the same problem cloning Terminals (http://terminals.codeplex.com/) and found a way to solve it! Set the hgsubversion.stupid to true in your hg config file, and cloning works just fine. I guess SvnBridge isn't fully SVN compatible, but the 'stupid' way of using svn works well enough, although absurdly slow, so I would still leave this bug open, and add my solution as a workaround for the time being.

  2. Yonggang Luo
    Windows PowerShell
    Copyright (C) 2009 Microsoft Corporation. All rights reserved.
    
    PS D:\CI\bld> h
    PS D:\CI\bld>
    PS D:\CI\bld> hg clone svn+https://ImapPusherService.svn.codeplex.com/svn
    destination directory: svn
    abort: destination 'svn' is not empty
    PS D:\CI\bld> hg clone svn+https://ImapPusherService.svn.codeplex.com/svn b
    abort: destination 'b' is not empty
    PS D:\CI\bld> hg clone svn+https://ImapPusherService.svn.codeplex.com/svn bc
    using meta-single layout
    Pull from r1 to r80588!
    [r12542] RNO\_MCLWEB: Created team project folder $/ImapPusherService via the Team Project Creation Wizard
    [r12542] !Added paths:set(['/'])
    [r12542] !Changed paths:set(['/'])
    [r12940] SND\gmurray_cp: Initial Code upload
    [r12940] !Changed paths:set(['/'])
    [r13122] SND\gmurray_cp: Release build was missing linking config
    [r13122] !Changed paths:set(['/'])
    [r13275] SND\gmurray_cp: Tracing instrumentation and retry limit
    [r13275] !Changed paths:set(['/'])
    [r13278] SND\gmurray_cp: Tracing
    [r13278] !Changed paths:set(['/'])
    [r13310] SND\gmurray_cp: Fix for client not being able to restart itself on fatal errors
    [r13310] !Changed paths:set(['/'])
    [r13322] SND\gmurray_cp: Fixes for IDLE refresh
    [r13322] !Changed paths:set(['/'])
    [r13389] SND\gmurray_cp: fix if server decides it has an error
    [r13389] !Changed paths:set(['/'])
    [r14271] SND\gmurray_cp: Fixes for automatically connecting, resource leaks.
    [r14271] !Changed paths:set(['/'])
    [r17762] SND\gmurray_cp: Added way to save settings
    [r17762] !Changed paths:set(['/'])
    [r17838] SND\gmurray_cp:
    [r17838] !Changed paths:set(['/'])
    [r17840] SND\gmurray_cp:
    [r17840] !Changed paths:set(['/'])
    [r17841] SND\gmurray_cp:
    [r17841] !Changed paths:set(['/'])
    [r17842] SND\gmurray_cp:
    [r17842] !Changed paths:set(['/'])
    [r17908] SND\gmurray_cp:
    [r17908] !Changed paths:set(['/'])
    [r17932] SND\gmurray_cp:
    [r17932] !Changed paths:set(['/'])
    [r17935] SND\gmurray_cp:
    [r17935] !Changed paths:set(['/'])
    [r18357] SND\gmurray_cp:
    [r18357] !Changed paths:set(['/'])
    [r57274] RNO\_TFSSERVICE: Checked in by server upgrade
    [r57274] !Changed paths:set(['/'])
    updating to branch default
    25 files updated, 0 files merged, 0 files removed, 0 files unresolved
    PS D:\CI\bld>
    

    It's works by using the newmeta. Is there any revision lost?

  3. Vsevolod Parfenov reporter

    I confirm that when I add

    [hgsubversion]
    stupid=true
    

    to mercurial.ini file then cloning done right.

    And it doesn't work without this option yet.

    Anton, thanks for sharing your workaround.

  4. Anton Afanasyev

    @dreamkxd Would you mind sharing your steps for getting subvertpy to install correctly? After building and installing it I first get "No module named hgsubversion" when subvertpy_wrapper tries to load subvertpy. Commented out references to hgsubversion (just for testing), and now I get an error that the subvertpy module cannot be found. I'm fairly sure I'm just doing it all wrong, so I'd rather have a nice and concise how-to for getting this to work.

  5. Anton Afanasyev

    @dreamkxd Sorry, should've mentioned. I'm using Windows as well. As I mentioned, I did built and installed subvertpy by doing "setup.py build" and "setup.py install". Did the same with your "hgsubversion.svnmeta". The results are as I mention in my previous message. Could you please describe how you got this to work? Thanks!

  6. Yonggang Luo

    Anton Afanasyev, That's werid.

    The line for p in sorted(paths.keys()) is the place I getting the things to be works. Maybe you just didn't configure the HG properly to using hgsubversion.svnmeta

    import errno
    import traceback
    
    from mercurial import node
    from mercurial import context
    
    import svnexternals
    import util
    
    
    class MissingPlainTextError(Exception):
        """Exception raised when the repo lacks a source file required for replaying
        a txdelta.
        """
    
    class ReplayException(Exception):
        """Exception raised when you try and commit but the replay encountered an
        exception.
        """
    
    def updateexternals(ui, meta, current):
        if not current.externals:
            return
    
        # accumulate externals records for all branches
        revnum = current.rev.revnum
        branches = {}
        for path, entry in current.externals.iteritems():
            subpath, branch_info = meta.split_path(path, revnum)
            if not meta.is_active_branch(branch_info):
                ui.warn('WARNING: Invalid path "%s" in externals\n' % path)
                continue
            bp = branch_info[0]
            if bp not in branches:
                parent = meta.get_parents(branch_info)[2]
                #assert parent != node.nullid, (branch_info, path, entry)
                pctx = meta.repo[parent]
                branches[bp] = (svnexternals.parse(ui, pctx), pctx)
            branches[bp][0][subpath] = entry
    
        # register externals file changes
        for bp, (external, pctx) in branches.iteritems():
            if bp and bp[-1] != '/':
                bp += '/'
            updates = svnexternals.getchanges(ui, meta.repo, pctx, external)
            for fn, data in updates.iteritems():
                path = (bp and bp + fn) or fn
                if data is not None:
                    current.set(path, data, False, False)
                else:
                    current.delete(path)
    
    def commit_branch(meta, branch_path, in_files, is_tag):
        ui = meta.ui
        current = meta.editor.current
        r = current.rev
        date = meta.fixdate(r.date)
        files = dict(in_files)
    
        branch_info = meta.exist_branch(branch_path, r.revnum, 2)
        #Whenever what happened, we must modify on an active path.
        assert meta.is_active_branch(branch_info), branch_info
        copy_from_parent, direct_parent, real_parent = meta.get_parents(branch_info)
        parentctx = meta.repo.changectx(real_parent)
        def filectxfn(repo, memctx, path):
            if path == '.hgtags':
                return context.memfilectx(path=path, data=files[path],
                          islink=False, isexec=False,
                          copied=None)
    
            current_file = files[path]
            if current_file in current.deleted:
                raise IOError(errno.ENOENT, '%s is deleted' % path)
            copied = current.copies.get(current_file)
            flags = parentctx.flags(path)
            is_exec = current.execfiles.get(current_file, 'x' in flags)
            is_link = current.symlinks.get(current_file, 'l' in flags)
            if current_file in current.files:
                data = current.files[current_file]
                if is_link and data.startswith('link '):
                    data = data[len('link '):]
                elif is_link:
                    ui.debug('file marked as link, but may contain data: '
                             '%s (%r)\n' % (current_file, flags))
            else:
                data = parentctx.filectx(path).data()
            return context.memfilectx(path=path,
                                      data=data,
                                      islink=is_link, isexec=is_exec,
                                      copied=copied)
    
        if branch_path in meta.branch_tags:
            tagdata = ''
            if '.hgtags' in parentctx:
                tagdata = parentctx.filectx('.hgtags').data()
            files['.hgtags'] = tagdata + meta.branch_tags[branch_path]
    
        is_closed = not meta.inside_branch_interval(branch_info, r.revnum)
        extra = meta.genextra(r.revnum, branch_path, is_tag or is_closed)
        parents = (copy_from_parent, direct_parent)
        if copy_from_parent == node.nullid:
            parents = (direct_parent, node.nullid)
        current_ctx = context.memctx(meta.repo,
                                     parents,
                                     r.message or util.default_commit_msg(ui),
                                     files.keys(),
                                     filectxfn,
                                     meta.authors[r.author],
                                     date,
                                     extra)
    
        new_hash = meta.repo.commitctx(current_ctx)
        util.describe_commit(ui, new_hash, branch_path)
        assert (r.revnum, branch_path) not in meta.revmap, \
            "Now for each branch_path, revision pair, there is exactly one corresponding revmap record!"
        meta.revmap[r.revnum, branch_path] = new_hash
        _ = meta.repo.changectx(new_hash)
    
    def convert_rev(ui, meta, svn, r, firstrun):
        editor = meta.editor
        editor.current.clear()
        editor.current.rev = r
    
        if firstrun and meta.revmap.oldest <= 0:
            # We know nothing about this project, so fetch everything before
            # trying to apply deltas.
            ui.debug('replay: fetching full revision\n')
            svn.get_revision(r.revnum, editor)
        else:
            svn.get_replay(r.revnum, editor, meta.revmap.oldest)
    
        revnum = r.revnum
        paths = r.paths
        for p in sorted(paths.keys()):
            ed_p = p[1:]
            if ed_p in editor.current.touched:
                continue
            subpath, branch_info = meta.split_path(p, revnum)
            if not meta.is_active_branch(branch_info):
                continue
            action = paths[p].action
            kind = meta.checkpath(p, revnum)
            msg = "fetching %s:%s:%s:%s from %s:%s" % (p, revnum, kind, action, paths[p].copyfrom_path, paths[p].copyfrom_rev)
            ui.status('Start %s\n' % msg)
            if action == 'D' or action == 'R': #First deal with DELETE
                editor.delete_entry(ed_p, revnum - 1, None)
            if (kind == 'd'):
                #Because it's lost, so the property should be fetched separately.
                editor.current.missing_props.add(ed_p + '/')
                editor.current.findprops(svn, ed_p, ed_p + '/')
                if action == 'A' or action == 'R':
                    editor.add_directory(ed_p, None, paths[p].copyfrom_path, paths[p].copyfrom_rev)
                elif action == 'M':#Only can modify props
                    pass
            elif (kind == 'f'): #It's a file, directly missing it
                if action == 'A' or action == 'M' or action == 'R':
                    #TODO: Using unified diff to reduce bandwidth, ask dreamkxd@163.com
                    editor.current.missing.add(ed_p)
            elif action != 'D':
                raise RuntimeError("The kind must be an clear value when %s" % msg)
    
        current = editor.current
        current.findmissing(svn)
    
        updateexternals(ui, meta, current)
    
        if current.exception is not None:  #pragma: no cover
            traceback.print_exception(*current.exception)
            raise ReplayException()
        if current.missing:
            raise MissingPlainTextError()
    
        # Periodically generate the list of files to commit
        files_to_commit = set(current.files.keys())
        files_to_commit.update(current.symlinks.keys())
        files_to_commit.update(current.execfiles.keys())
        files_to_commit.update(current.deleted.keys())
        # back to a list and sort so we get sane behavior
        files_to_commit = list(files_to_commit)
        files_to_commit.sort()
    
        all_changed = set.union(editor.current.changed, meta.changed)
        tag_added = [t for t in meta.added if meta.is_tag(t, r.revnum)]
        tag_changed = [t for t in all_changed if meta.is_tag(t, r.revnum)]
        tag_deleted = [t for t in meta.deleted if meta.is_tag(t, r.revnum - 1)]
        tag_changes = set(tag_added + tag_changed + tag_deleted)
        if tag_changes:
            ui.status("[r%s] !Changed tags:%s\n" % (r.revnum, tag_changes))
        if meta.added:
            ui.status("[r%s] !Added paths:%s\n" % (r.revnum, meta.added))
        if all_changed:
            ui.status("[r%s] !Changed paths:%s\n" % (r.revnum, all_changed))
        if meta.deleted:
            ui.status("[r%s] !Deleted paths:%s\n" % (r.revnum, meta.deleted))
    
        all_modifyed = set.union(meta.added, all_changed, meta.deleted)
        meta.branch_tags = {}
        branch_batches = {}
        for branch_path in all_modifyed:
            branch_batches[branch_path] = []
    
        #Building up the branches that have files on them
        for f in files_to_commit:
            subpath, branch_info = meta.split_path(f, r.revnum)
            if not branch_info:
                continue
            branch_path = branch_info[0]
            if branch_path in all_modifyed:
                branch_batches[branch_path].append((subpath, f))
    
        # 1. handle tags commit first, because branches need tags information
        for branch_path in sorted(branch_batches.keys()):
            '''
            TODO:add test case!
            Only when an tag is changed to commit.
            We don't commit either when it's added or deleted.
            NOTICE:When an tag is added with modification, we will commit it.
            '''
            if branch_path in tag_changed:
                files = branch_batches[branch_path]
                commit_branch(meta, branch_path, files, True)
    
        # 2. handle tags ancestors, getting those ancestors to updating .hgtags files.
        if tag_changes:
            branch_tags = meta.get_branch_tags(r.revnum, tag_deleted, set(tag_added + tag_changed))
            for bp in branch_tags:
                tags_file = ''.join(branch_tags[bp])
                meta.branch_tags[bp] = tags_file
                if bp not in branch_batches:
                    branch_batches[bp] = []
    
        # 3. handle non-tags branch commit. they may be closed branch
        for branch_path in sorted(branch_batches.keys()):
            if branch_path not in tag_changes:
                files = branch_batches[branch_path]
                commit_branch(meta, branch_path, files, False)
    
  7. Anton Afanasyev

    @dreamkxd

    installing both hgsubversion AND hgsubversion.svnmeta helped, thanks. I'm running into more prolems now, however, although not directly with svnmeta (I think).

    To reproduce, use Subvertpy bindings (I tried with v 0.7.4. and v 0.8.1) and try cloning the Terminals repository I linked to in my first post here. Here's the SVN link: https://Terminals.svn.codeplex.com/svn

    Things get cloned fine until revision 20721, which has a "%20" in its name. The way this is handled is certainly a bug in SvnBridge that CodePlex uses. Specifically, this file: https://terminals.svn.codeplex.com/svn/!svn/bc/20721/Resources/Transport%20Examples.zip does not exist. Uri-encoding the '%' symbol doesnt help, as the "https://terminals.svn.codeplex.com/svn/!svn/bc/20721/Resources/Transport%2520Examples.zip" file does not exist either. However, this one does: "https://terminals.svn.codeplex.com/svn/!svn/bc/20721/Resources/Transport%252520Examples.zip". I've worked around this problem by checking for files with a '%' symbol in the name and downloading them via python code instead of using svn libs. A very rough hack, I know, but it got me through this file. Next, I encountered a problem at revisions 45315 and 45316. You see, there is a file named HidePanel.PNG that gets deleted in revision 45315, and added with a lower case extension (i.e. HidePanel.png) in revision 45316. SVN code in the extension *hgsubversion, subvertpy, OR svnmeta - I dont know* somehow bundles these two revisions into ONE, with the second change being listed without a file type, so the code breaks. My explanation sucks, so please just try cloning this repository to see what I mean. Problem happens with both Subvertpy and SWIG bindings.

    THANKS for your help so far!

  8. Anton Afanasyev

    I've tried the latest version, and it certainly seems better. I've moved away from Subvertpy to SWIG, however, since I couldn't get an x64 build of Subvertpy to compile, for whatever reason. I may try at a later point, but SWIG it is for me, for now at least. With that said, I've noticed that you've only included codeplex changes to the subvertpy wrapper, so I did the same to the swig one. Things seem to be working fine now.

    Edit: I've not updated to the very latest of svnmeta, and pulled in a few changes from the Terminals CodePlex repository. The first changest looked like it was pulled from an unrelated repo (see: http://www.twitpic.com/6zhfvm). I'm assuming this has something to do with the latest hash change detection, but not sure. I'll try a new clone of the repo now.

  9. Anton Afanasyev

    @dreamkxd

    I've run into a problem with unicode filenames, and fixed it as so (this is for revision tagged lyg_beta2, but I think it will patch fine for your later changes as well):

    diff -r 5d4528b7b3a9 hgsubversion/svnwrap/common.py
    --- a/hgsubversion/svnwrap/common.py	Thu Oct 13 23:54:22 2011 +0800
    +++ b/hgsubversion/svnwrap/common.py	Fri Oct 14 10:23:08 2011 -0700
    @@ -93,12 +93,13 @@
             return self.HEAD
     
         def get_file(self, path, revision):
    +        utfPath = urllib.quote(path.decode('utf8').encode('utf8'))
             try:
    -            return self.get_file_impl(path[1:], revision)
    +            return self.get_file_impl(utfPath[1:], revision)
             except IOError as e:
                 if self.root.find('.svn.codeplex.com/svn') != -1:
                     try:
    -                    p = urllib.quote(urllib.quote(path))
    +                    p = utfPath.replace('%', '%25')
                         f_url = '%s/!svn/bc/%s/%s' % (self.root, revision, p)
                         f = urllib2.urlopen(f_url)
                         data = f.read()
    
    

    Notice that I also, instead of urllib.quote'ing the path for CodePlex, just replace the '%' with '%25' I think this should be enough for it to work correctly (and it does work correctly for me like this), but I may be wrong.

  10. Yonggang Luo

    I will apply it like this because get_file_impl will quote it automatically.

    @@ -126,13 +126,15 @@
             return None
     
         def get_file(self, path, revision):
    +        utfPath = path.decode('utf8').encode('utf8')
             try:
    -            return self.get_file_impl(path[1:], revision)
    +            return self.get_file_impl(utfPath[1:], revision)
             except IOError as e:
                 if self.root.find('.svn.codeplex.com/svn') != -1:
                     try:
    -                    p = urllib.quote(urllib.quote(path))
    -                    f_url = '%s/!svn/bc/%s/%s' % (self.root, revision, p)
    +                    p = urllib.quote(utfPath)
    +                    p = p.replace('%', '%25')
    +                    f_url = '%s/!svn/bc/%s%s' % (self.root, revision, p)
                         f = urllib2.urlopen(f_url)
                         data = f.read()
                         # FIXME:Fetch codeplex urls, there is no way to get the mode.
    
  11. Log in to comment