Jonathan Nieder  committed 68f532f

git-svn: use YAML format for mergeinfo cache when possible

Since v1.7.0-rc2~11 (git-svn: persistent memoization, 2010-01-30),
git-svn has maintained some private per-repository caches in
.git/svn/.caches to avoid refetching and recalculating some
mergeinfo-related information with every "git svn fetch".

These caches use the 'nstore' format from the perl core module
Storable, which can be read and written quickly and was designed for
transfer over the wire (the 'n' stands for 'network'). This format is
endianness-independent and independent of floating-point

Unfortunately the format is *not* independent of the perl version ---
new perl versions will write files that very old perl cannot read.
Worse, the format is not independent of the size of a perl integer.
So if you toggle perl's use64bitint compile-time option, then using
'git svn fetch' on your old repositories produces errors like this:

Byte order is not compatible at ../../lib/ (autosplit
into ../../lib/auto/Storable/ line 380, at
/usr/share/perl/5.12/Memoize/ line 21

That is, upgrading perl to a version that uses use64bitint for the
first time makes git-svn suddenly refuse to fetch in existing
repositories. Removing .git/svn/.caches lets git-svn recover.

It's time to switch to a platform independent serializer backend with
better compatibility guarantees. This patch uses YAML::Any.

Other choices were considered:

- thawing data from Data::Dumper involves "eval". Doing that without
creating a security risk is fussy.

- the JSON API works on scalars in memory and doesn't provide a
standard way to serialize straight to disk.

YAML::Any is reasonably fast and has a pleasant API. In most
backends, LoadFile() reads the entire file into a scalar anyway and
converts it as a second step, but having an interface that allows the
deserialization to happen on the fly without a temporary is still a

YAML::Any is not a core perl module, so we take care to use it when
and only when it is available. Installations without that module
should fall back to using Storable with all its quirks, keeping their
cache files in


Installations with YAML peacefully coexist by keeping a separate set
of cache files in


In most cases, switching between is a one-time thing, so it doesn't
seem worth the complication to migrate existing caches.

The upshot: after this patch, as long as YAML::Any is installed you
can move your git repository between machines with different perl
installations and "git svn fetch" will work fine. If you do not have
YAML::Any, the behavior is unchanged (and in particular does not get
any worse).

Reported-by: Sandro Weiser <>
Reported-by: Bdale Garbee <>
Signed-off-by: Jonathan Nieder <>
Signed-off-by: Eric Wong <>

  • Participants
  • Parent commits 9f7ad14

Comments (0)

Files changed (3)

File git-svn.perl

 use Memoize;  # core since 5.8.0, Jul 2002
 use Memoize::Storable;
 use POSIX qw(:signal_h);
+my $can_use_yaml;
+	$can_use_yaml = eval { require Git::SVN::Memoize::YAML; 1};
 my ($_gc_nr, $_gc_period);
 		command_oneline("rev-parse", "$commit~1^{tree}"));
+sub tie_for_persistent_memoization {
+	my $hash = shift;
+	my $path = shift;
+	if ($can_use_yaml) {
+		tie %$hash => 'Git::SVN::Memoize::YAML', "$path.yaml";
+	} else {
+		tie %$hash => 'Memoize::Storable', "$path.db", 'nstore';
+	}
 # The GIT_DIR environment variable is not always set until after the command
 # line arguments are processed, so we can't memoize in a BEGIN block.
 		my $cache_path = "$ENV{GIT_DIR}/svn/.caches/";
 		mkpath([$cache_path]) unless -d $cache_path;
-		tie my %lookup_svn_merge_cache => 'Memoize::Storable',
-		    "$cache_path/lookup_svn_merge.db", 'nstore';
+		my %lookup_svn_merge_cache;
+		my %check_cherry_pick_cache;
+		my %has_no_changes_cache;
+		tie_for_persistent_memoization(\%lookup_svn_merge_cache,
+		    "$cache_path/lookup_svn_merge");
 		memoize 'lookup_svn_merge',
 			LIST_CACHE => ['HASH' => \%lookup_svn_merge_cache],
-		tie my %check_cherry_pick_cache => 'Memoize::Storable',
-		    "$cache_path/check_cherry_pick.db", 'nstore';
+		tie_for_persistent_memoization(\%check_cherry_pick_cache,
+		    "$cache_path/check_cherry_pick");
 		memoize 'check_cherry_pick',
 			LIST_CACHE => ['HASH' => \%check_cherry_pick_cache],
-		tie my %has_no_changes_cache => 'Memoize::Storable',
-		    "$cache_path/has_no_changes.db", 'nstore';
+		tie_for_persistent_memoization(\%has_no_changes_cache,
+		    "$cache_path/has_no_changes");
 		memoize 'has_no_changes',
 			SCALAR_CACHE => ['HASH' => \%has_no_changes_cache],

File perl/Git/SVN/Memoize/

+package Git::SVN::Memoize::YAML;
+use warnings;
+use strict;
+use YAML::Any ();
+# based on Memoize::Storable.
+sub TIEHASH {
+	my $package = shift;
+	my $filename = shift;
+	my $truehash = (-e $filename) ? YAML::Any::LoadFile($filename) : {};
+	my $self = {FILENAME => $filename, H => $truehash};
+	bless $self => $package;
+sub STORE {
+	my $self = shift;
+	$self->{H}{$_[0]} = $_[1];
+sub FETCH {
+	my $self = shift;
+	$self->{H}{$_[0]};
+sub EXISTS {
+	my $self = shift;
+	exists $self->{H}{$_[0]};
+sub DESTROY {
+	my $self = shift;
+	YAML::Any::DumpFile($self->{FILENAME}, $self->{H});
+sub SCALAR {
+	my $self = shift;
+	scalar(%{$self->{H}});
+	'Fake hash from Git::SVN::Memoize::YAML';
+sub NEXTKEY {
+	undef;
+=head1 NAME
+Git::SVN::Memoize::YAML - store Memoized data in YAML format
+=head1 SYNOPSIS
+    use Memoize;
+    use Git::SVN::Memoize::YAML;
+    tie my %cache => 'Git::SVN::Memoize::YAML', $filename;
+    memoize('slow_function', SCALAR_CACHE => [HASH => \%cache]);
+    slow_function(arguments);
+This module provides a class that can be used to tie a hash to a
+YAML file.  The file is read when the hash is initialized and
+rewritten when the hash is destroyed.
+The intent is to allow L<Memoize> to back its cache with a file in
+YAML format, just like L<Memoize::Storable> allows L<Memoize> to
+back its cache with a file in Storable format.  Unlike the Storable
+format, the YAML format is platform-independent and fairly stable.
+Carps on error.
+See L<YAML::Any>.
+L<YAML::Any> from CPAN.
+None reported.
+=head1 BUGS
+The entire cache is read into a Perl hash when loading the file,
+so this is not very scalable.

File perl/Makefile.PL

 my %pm = (
 	'' => '$(INST_LIBDIR)/',
 	'Git/' => '$(INST_LIBDIR)/Git/',
+	'Git/SVN/Memoize/' => '$(INST_LIBDIR)/Git/SVN/Memoize/',
 	'Git/SVN/' => '$(INST_LIBDIR)/Git/SVN/',
 	'Git/SVN/' => '$(INST_LIBDIR)/Git/SVN/',
 	'Git/SVN/' => '$(INST_LIBDIR)/Git/SVN/',