1. TortoiseHg
  2. TortoiseHg
  3. thg
  4. Issues
Issue #2333 invalid

decode is not called when showing diff in Workbench windows

created an issue


I use customized Bigfiles extension and I found that decode is not called for showing diff in Workbench windows (when committing, for example).

Is it possible to fix this behaviour?

Many thanks in advance, Dmitriy

Comments (17)

  1. iassenev reporter

    encode/decode is set in the c:\users\<user_name>\mercurial.ini :

    external = c:\users\<user_name>\external.py
    **.* = upload: //
    **.* = download: //
  2. iassenev reporter

    yes, upload and download are defined in external.py, I may send it to you

    these procedures work well for us, apart from the described situation, when I suspect they are just not called

  3. Yuya Nishihara

    Without the extension, I got the following outputs. So it should at least try executing encode/decode filters.

    Z:\work\hghacks\testrepos\hello>hg stat                         
    M hello.c
    Z:\work\hghacks\testrepos\hello>"c:\Program Files\TortoiseHg"\thg.exe --nofork --debug
    filtering hello.c through upload: //
    'upload:' は、内部コマンドまたは外部コマンド、
    操作可能なプログラムまたはバッチ ファイルとして認識されていません。

    (Sorry for non-English messages which came from Windows. It says 'upload:' is not found or not a valid executable.)

    TortoiseHg 3.2.3, Windows XP 32bit

  4. iassenev reporter

    how can I attach a file? (didn't find any corresponding button)

    just pasting the whole file here:

    force_external = '.a.animation.aps.avi.bmp.chm.clip.clip_check_data.controller.cur.db.dds.dll.doc.elf.exe.exp.gif.hpi.hqx.htm.html.i.ico.idb.iff.im.jar.jpeg.jpg.lib.lnk.localization.log.mcp.mo.msi.ncb.nrproj.o.obj.ogf.opt.pbxproj.pch.pdb.pdf.png.ppm.rar.raw.res.resx.resources.rgb.rla.rtf.self.snk.strings.suo.sys.terr.tga.tif.user.vp3.vsd.wmhf.wmif.wmof.xls.xlsx.xml.xsl.xslt.zip.map.'
    ignored_by_programmers = '.psd.jpg.jpeg.mb.max.'
    import os, re, hashlib, inspect, sys, zlib, math, tempfile, time
    from mercurial import util, httprepo, ui
    if sys.version_info < (2, 4, 4):
        ui.ui().warn('Warning: external hook requires Python 2.4.4+ (2.5.1 preferred)\n')
    # V1.6
    # Workaround for a Python bug (in linecache.py?):
    # http://bugs.python.org/issue1728
    # http://bugs.python.org/issue1665
    # https://bugs.launchpad.net/ubuntu/+source/python-defaults/+bug/70902
    # http://mail.python.org/pipermail/python-bugs-list/2007-March/037472.html
    _inspect_findsource_orig = inspect.findsource
    def _inspect_findsource_robust(object):
            return _inspect_findsource_orig(object)
        except IndexError:
            raise IOError('workaround for linecache bug')
    # Compatibility for Hg not including f8ad3b76e923:
    def _findparamvalue(function, param):
        for framerec in inspect.getouterframes(inspect.currentframe()):
            if framerec[3] == function:
                funcvars = inspect.getargvalues(framerec[0])[3]
                if not param in funcvars:
                    raise util.Abort('No parameter named %s in function %s' % (param, function))
                return funcvars[param]
        raise util.Abort('No function named %s in stack' % function)
    def _filename(f):
        if f:
            return f
        return _findparamvalue('_filter', 'filename')
    def _ui(u):
        if u:
            return u
        return _findparamvalue('_filter', 'self').ui
    def _repo(r):
        if r:
            return r
        return _findparamvalue('_filter', 'self')
    def _trim(filename):
        return os.path.basename(filename)
    def _is_binary(data):
        if '\0' in data:
            return True
            return False
    def _is_force_external(ext):
        pattern = ext + '.'
        if pattern in force_external:
            return True
            return False
    def _is_ignored_by_programmers(ext):
        if ext == '':
            return False
        pattern = ext + '.'
        if pattern in ignored_by_programmers:
            return True
            return False
    def _sha1hash(data):            
        return hashlib.sha1(data).hexdigest().upper()
    def _external_download(url, ui, sha1, filename):
        url = url.lower()
    #    ui.status('Decompressing %s from %s ...' % (filename, url))
        st_time = time.time()
        hf = open(url, 'rb')
        hd = zlib.decompressobj()
        tfn = tempfile.mktemp('mcr_')
        htf = open(tfn, 'wb')
        while True:
            compr_part = hf.read(524288)
            if compr_part == '':
        htf.write(hd.decompress( compr_part ))
        portion = hd.flush()
        if portion != '':
        htf.write( portion )
        fn_time = time.time()
    #    ui.status(' [%u sec]' % int(fn_time-st_time) )
        st_time = time.time()
        htf = open(tfn, 'rb')
        data = htf.read()
        fn_time = time.time()
    #    ui.status(' [%u sec]' % int(fn_time-st_time) )
        st_time = time.time()
        sha1hsh = _sha1hash(data)
        fn_time = time.time()
    #    ui.status(' [%u sec]\n' % int(fn_time-st_time) )
        if sha1 != sha1hsh:
            raise util.Abort('hash mismatch in %s' % filename)
        return data
    def download(s, cmd, ui=None, filename=None, **kwargs):
        # To support dev versions of Hg with f8ad3b76e923 but not f3a8b5360100:
        cmd = re.sub(r'^download: *', '', cmd)
        filename = _filename(filename)
        ui = _ui(ui)
        n = _trim(filename)
        m = re.match(r'^<<<EXTERNAL (/([^/]+)/([0-9A-F]{40}))>>>\n$', s)
        (dummy, ext) = os.path.splitext(n.lower())  
        if not m or _is_ignored_by_programmers(ext):
            return s
        data = _external_download(url = cmd + m.group(1), ui = ui, sha1 = m.group(3), filename = filename)
        return data
    def upload(s, cmd, ui=None, repo=None, filename=None, **kwargs):
        (dummy, ext) = os.path.splitext(filename.lower())
        be_external = _is_force_external(ext) or (len(s) > 262144) or _is_binary(s)
        if not be_external or _is_ignored_by_programmers(ext):
        return s
        cmd = re.sub(r'^upload: *', '', cmd)
        filename = _filename(filename)
        ui = _ui(ui)
        repo = _repo(repo)
        n = _trim(filename)
        sha1 = _sha1hash(s)
        full = '/%s/%s' % (n , sha1)
        downloadurl = cmd + full
        downloadurl = downloadurl.lower()
        if os.path.isfile(downloadurl):
        return '<<<EXTERNAL %s>>>\n' % full
        dirname = os.path.dirname(downloadurl)
        if not os.path.isdir(dirname):
    #    ui.status('Uploading %s to %s (%s Kb)\n' % (filename, downloadurl, len(s) / 1024))
        fh = open(downloadurl, 'wb')
        co = zlib.compressobj( 1 )
        slice_size = 524288
        n_slices = int( math.ceil( float( len( s ) ) / float( slice_size ) ) )
        for n in range( n_slices ):
            fh.write( co.compress( s[(n*slice_size):(n*slice_size+slice_size)] ) )
        fh.write( co.flush() )
        except IOError, err:
        raise util.Abort('Problem uploading %s to %s (try it manually): %s\n' % (filename, downloadurl, err))
        # Now ensure that the upload actually worked, by downloading:
        _external_download(url = downloadurl, ui = ui, sha1 = sha1, filename = filename)
        return '<<<EXTERNAL %s>>>\n' % full
    def reposetup(ui, repo):
        if not repo.local():
        for name, fn in {'download:': download, 'upload:': upload}.iteritems():
                # Hg 0.9.6+ (with f8ad3b76e923):
                repo.adddatafilter(name, fn)
            except AttributeError:
                # Hg 0.9.5:
                util.filtertable[name] = fn
  5. Yuya Nishihara

    It seems I misunderstood the point.

    You want to call the decode filter (instead of encode) by diff, right?

    decode is the filter called when writing files to the working dir, so diff never executes it. encode filter should be called in this case to translate working-dir file to in-repository form.

  6. iassenev reporter

    You want to call the decode filter (instead of encode) by diff, right?


    since we store only links to files in the repository, I'd like to view difference of the decoded files, not the encoded ones, to view difference between files' contents, not between links to files

  7. Yuya Nishihara

    So encode/decode filters are not what you want. You'll need a different approach, e.g. monkey-patching fctx.data() to carefully translate the URL to content, etc. largefiles or facebook's remotefilelog extensions might have some hints.

    Closed because this is the design of Mercurial.

  8. Log in to comment