Issue #171 open

hg svn verify fails: hgsubversion losing executable bit

Jonathan Noack
created an issue

It appears that hgsubversion is not properly tracking the executable bit for svn merges. I cloned the stable/8 branch of the FreeBSD SVN repo and then ran hg svn verify. The verify failed with this output: {{{ $ hg svn verify verifying c56ce81a78fc against r207386 difference in file mode release/scripts/make-memstick.sh difference in file mode release/scripts/mm-mtree.sh difference in file mode etc/rc.d/rtsold difference in file mode tools/regression/acltools/aclfuzzer.sh difference in file mode release/picobsd/floppy.tree/sbin/dhclient-script difference in file mode etc/rc.d/ubthidhci }}}

Here's a ViewVC link to the first showing it was merged: http://svn.freebsd.org/viewvc/base/stable/8/release/scripts/make-memstick.sh

Note that I modified the verify output to add newlines and separate data vs. mode differences: {{{ diff -r b72850177e5c hgsubversion/svncommands.py --- a/hgsubversion/svncommands.py Thu Apr 22 22:37:03 2010 -0500 +++ b/hgsubversion/svncommands.py Thu Apr 29 16:19:36 2010 -0400 @@ -60,8 +60,11 @@ fctx = ctx[fn] dmatch = fctx.data() == data mmatch = fctx.flags() == mode - if not (dmatch and mmatch): - ui.write('difference in file %s' % fn) + if not dmatch: + ui.write('difference in file data %s\n' % fn) + result = 1 + elif not mmatch: + ui.write('difference in file mode %s\n' % fn) result = 1

 hgfiles = set(ctx)

}}}

Here are my svn, hg, and hgsubversion version numbers: {{{ $ svn --version | head -n 1 svn, version 1.6.9 (r901367) $ hg version | head -n 1 Mercurial Distributed SCM (version 1.5.1) $ hg head | head -n 1 changeset: 605:b72850177e5c }}}

Comments (11)

  1. Augie Fackler repo owner

    (Reply via dur...@gmail.com):

    On Thu, Apr 29, 2010 at 3:34 PM, <issues-noreply@bitbucket.org> wrote:

    Can you send me an 'hg export'ed version of that patch? It seems eminently reasonable, and I'd like to give you credit.

    As for the corruption, I'll take a look when I get a chance, thanks for the report.

  2. Jonathan Noack reporter

    I dug into this a little tonight. I cloned using --stupid and it passed a verify! So the problem appears to be in the replay code.

    To investigate I added an excerpt from the "stupid" code into the replay code:

    diff -r 532c545d162c hgsubversion/replay.py
    --- a/hgsubversion/replay.py    Sat May 15 20:33:05 2010 +0200
    +++ b/hgsubversion/replay.py    Sun May 23 00:32:44 2010 -0400
    @@ -149,6 +149,12 @@
                 flags = parentctx.flags(path)
                 is_exec = current.execfiles.get(current_file, 'x' in flags)
                 is_link = current.symlinks.get(current_file, 'l' in flags)
    +            if r.revnum == 206278:
    +                svnpath = path
    +                if branch and branch is not None:
    +                    svnpath = branch + '/' + path
    +                data, mode = svn.get_file(svnpath, r.revnum)
    +                ui.warn('%s (orig = %r,%r,%r) (test = %r)\n' % (svnpath, is_exec, is_link, flags, mode))
                 if current_file in current.files:
                     data = current.files[current_file]
                     if is_link and data.startswith('link '):
    

    The clone output confirms that flags are not being tracked properly:

    [r206278] kensmith: Merge r204044:
    release/scripts/make-memstick.sh (orig = False,False,'') (test = 'x')
    

    I got lost in SWIG after that...

  3. Jonathan Noack reporter

    To speed up testing I switched to revision 196434 (from 206278). I correspondingly updated the replay.py patch listed in comment 3.

    I think I understand the SWIG callback semantics a bit. I added to following to editor.py to show the callbacks:

    diff -r 532c545d162c hgsubversion/editor.py
    --- a/hgsubversion/editor.py    Sat May 15 20:33:05 2010 +0200
    +++ b/hgsubversion/editor.py    Sun May 23 13:53:07 2010 -0400
    @@ -57,6 +57,8 @@
             self.exception = None
     
         def set(self, path, data, isexec=False, islink=False):
    +        if 'mm-mtree' in path:
    +            self.ui.warn('set: %s (%r,%r)\n' % (path, isexec, islink))
             if islink:
                 data = 'link ' + data
             self.files[path] = data
    @@ -200,6 +202,8 @@
                 # we know this branch will exist now, because it has at least one file. Rock.
                 self.meta.branches[branch] = None, 0, self.current.rev.revnum
             self.current.file = path
    +        if 'mm-mtree' in self.current.file:
    +            self.ui.warn('add_file: %s (%r,%r)\n' % (self.current.file, copyfrom_path, copyfrom_revision))
             if not copyfrom_path:
                 self.ui.note('A %s\n' % path)
                 self.current.set(path, '', False, False)
    @@ -294,6 +298,8 @@
     
         @ieditor
         def change_file_prop(self, file_baton, name, value, pool=None):
    +        if 'mm-mtree' in self.current.file:
    +            self.ui.warn('change_file_prop: %s (%r=%r)\n' % (self.current.file, name, value))
             if name == 'svn:executable':
                 self.current.execfiles[self.current.file] = bool(value is not None)
             elif name == 'svn:special':
    

    This shows that only add_file is called and no attempt is made to call change_file_prop:

    [r196564] dougb: MFC 196434:
    add_file: stable/8/release/scripts/mm-mtree.sh (None,-1)
    set: stable/8/release/scripts/mm-mtree.sh (False,False)
    release/scripts/mm-mtree.sh (orig = False,False,'') (test = 'x')
    
  4. Jonathan Noack reporter

    A quick status update:

    I installed Subvertpy and updated to the latest version (barely post-1.3) but the behavior remains the same; namely, hgsubversion still loses the exec bit for svn merges but works fine with --stupid. Versions listed below. Note that I ran the test suite and get 5 failures out of the 493 tests (see attached hgsubversion_tests.txt); should I open a separate issue for those?

    $ uname -a
    FreeBSD hyperion.noacks.org 8.2-RELEASE-p4 FreeBSD 8.2-RELEASE-p4 #8 r226043: Wed Oct  5 13:30:22 CDT 2011     root@hyperion.noacks.org:/usr/obj/usr/src/sys/GENERIC  amd64
    
    $ python -V
    Python 2.7.2
    
    $ hg version --svn
    Mercurial Distributed SCM (version 1.9.3)
    (see http://mercurial.selenic.com for more information)
    
    Copyright (C) 2005-2011 Matt Mackall and others
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    hgsubversion: 3292ff0380f3
    Subversion: 1.7.0
    bindings: Subvertpy 0.8.8
    
  5. Jonathan Noack reporter

    Here's a quick way to reproduce (using the svncommands.py verify patch attached):

    $ hg clone -r 196564 http://svn.freebsd.org/base/stable/8/release/scripts scripts
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    $ cd scripts
    $ hg log --template '{rev}:{node|short} [r{svnrev}]\n'
    1:e4106cda65d8 [r196564]
    0:6e43def5af9c [r196045]
    $ hg svn verify
    verifying e4106cda65d8 against r196564
    difference in file mode mm-mtree.sh
    $
    

    With --stupid:

    $ hg clone -r 196564 --stupid http://svn.freebsd.org/base/stable/8/release/scripts scripts-stupid
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    $ cd scripts-stupid 
    $ hg log --template '{rev}:{node|short} [r{svnrev}]\n'
    1:9fe19d094496 [r196564]
    0:6e43def5af9c [r196045]
    $ hg svn verify
    verifying 9fe19d094496 against r196564
    $
    
  6. Jonathan Noack reporter

    Just tested with ea8ef4dd967e and this appears to be working now.

    $ hg clone -r 196564 file:///home/svnmirror/base/stable/8/release/scripts scripts
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    $ cd scripts
    $ hg log --template '{rev}:{node|short} [r{svnrev}]\n'
    1:9fe19d094496 [r196564]
    0:6e43def5af9c [r196045]
    $ hg svn verify
    verifying 9fe19d094496 against file:///home/svnmirror/base/stable/8/release/scripts@196564
    $
    
  7. Jonathan Noack reporter
    • changed status to open

    Doh! It's still broken when cloning from a remote location.

    $ hg clone -r 196564 http://svn.freebsd.org/base/stable/8/release/scripts scripts
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    $ cd scripts 
    $ hg log --template '{rev}:{node|short} [r{svnrev}]\n'
    1:e4106cda65d8 [r196564]
    0:6e43def5af9c [r196045]
    $ hg svn verify
    verifying e4106cda65d8 against http://svn.freebsd.org/base/stable/8/release/scripts@196564
    wrong flags for: mm-mtree.sh
    $
    

    Here's my setup:

    $ uname -a
    FreeBSD hyperion.noacks.org 9.1-RC3 FreeBSD 9.1-RC3 #4 r242462: Thu Nov  1 19:18:42 MDT 2012     root@hyperion.noacks.org:/usr/obj/usr/src/sys/GENERIC  amd64
    
    $ python -V
    Python 2.7.3
    
    $ hg version --svn
    Mercurial Distributed SCM (version 2.4)
    (see http://mercurial.selenic.com for more information)
    
    Copyright (C) 2005-2012 Matt Mackall and others
    This is free software; see the source for copying conditions. There is NO
    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    
    hgsubversion: ea8ef4dd967e
    Subversion: 1.7.2
    bindings: Subvertpy 0.8.10
    
  8. Jonathan Noack reporter

    Looks like it's the conversion:

    $ hg clone -r 196564 file:///home/svnmirror/base/stable/8/release/scripts scripts-local
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    
    $ hg clone --stupid -r 196564 http://svn.freebsd.org/base/stable/8/release/scripts scripts-remote-stupid
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions                                                                    
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    
    $ hg clone -r 196564 http://svn.freebsd.org/base/stable/8/release/scripts scripts-remote
    [r196045] kensmith: Copy head to stable/8 as part of 8.0 Release cycle.
    [r196564] dougb: MFC 196434:
    pulled 2 revisions
    updating to branch default
    33 files updated, 0 files merged, 0 files removed, 0 files unresolved
    
    $ ls -l scripts-*/mm-mtree.sh
    -rwxr-xr-x  1 noackjr  noackjr  4254 Nov 16 15:44 scripts-local/mm-mtree.sh
    -rwxr-xr-x  1 noackjr  noackjr  4254 Nov 16 15:49 scripts-remote-stupid/mm-mtree.sh
    -rw-r--r--  1 noackjr  noackjr  4254 Nov 16 15:44 scripts-remote/mm-mtree.sh
    
  9. Log in to comment