Overview

tb-rsync-vss

A Microsoft Windows executable that uses the Windows Volume Snapshot Service (VSS) to create a snapshot of a Windows drive that can then be accessed via rsync. Because a snapshot is taken, you can rsync a drive that's in use: you will be able to read all files (including ones that are open on the source drive). In addition the snapshot will show all of the files at a consistent point in time.

I primarily use this program as a wrapper around rsync, so that rsync can be called by dirvish. I routinely back up Windows boxes to Linux servers using tb-rsync-vss with cygwin rsync and sshd.

tb-rsync-vss was written by Eric V. Smith of True Blade Systems, Inc., and is released under the Apache License, Version 2.0.

tb-rsync-vss has been installed and tested on Windows 7 (64-bit) and Windows Server 2008 R2 (64-bit).

Overview

tb-rsync-vss works in the following way:

  • It is called with the identical parameters as rsync would be. This is a requirement in order for it to be a drop-in replacement for rsync. The parameters to tb-rsync-vss are "hidden" in the last rsync parameter, which is normally the path to back up. This "hiding" is described later.
  • It creates a VSS snapshot of a local path, such as "c:\Users".
  • It maps this snapshot to another Windows drive letter, such as "z:". The snapshot will exist for as long as tb-rsync-vss and whatever program it executes are running.
  • It modifies its parameters to point to this mapped snapshot.
  • It calls another executable, typically rsync.

Note that tb-rsync-vss is a normal Windows excutable. In particular, it does not use cygwin.dll.

Compiling tb-rsync-vss

I've compiled this with Visual Studio 2010 using the included tb-rsync-vss.sln. This solution file allows you to compile either 32- or 64-bit versions of tb-rsync-vss. For 64-bit Windows, you must use the 64-bit version of tb-rsync-vss.

tb-rsync-vss requires the boost C++ libraries to compile, but not to run.

Pre-compiled .msi files

Pre-compiled .msi files, both 32- and 64-bit, are available on the project's bitbucket.org home page. The default installation location for these installers is "<program files>\True Blade Systems". That location will be used in all examples in this document.

The Visual Studio solution file includes projects to build these .msi files.

Using cygwin sshd

There was a long-standing problem using cygwin rsync over cygwin sshd. The problem would result in rsync hanging when backing up large numbers of files. Although I could not find any specific reference to this bug being fixed, I have been able to back up 75 GB in 55,000 files with no problems, using a recent cygwin (1.7.11-1). In the past, this has always caused rsync to hang. I consider this problem to be fixed.

In addition, there are issues with using public key authentication and Windows. The root of the problem is that you're asking Windows for certain permissions without ever having supplied a password. For more information on this issue, see this excellent page. I use the LSA solution (method 2). I have also used the stored encrypted password (method 3) successfully. In both cases, you will need to have the public keys set up for your account.

Passing parameters to tb-rsync-vss

Because tb-rsync-vss is a drop-in replacement for rsync, it must take the same parameters as rsync does. This yields a challenge: how to pass parameters to tb-rsync-vss itself? My solution is to hide them in the final parameter to rsync: the path that rsync will back up. For example, if you really want to back up c:\Users, normally rsync in server mode might be called with these parameters (using cygdrive notation): "--server" "." "/cygdrive/c/Users". When using tb-rsync-vss, the last parameter becomes: "/|tmp-drive=x|rsync=c:/opt/cygwin/bin/rsync.exe|cygpath=true|c:/Users/".

tb-rsync-vss reads that last parameter, extracts its information, and translates it to "/cygdrive/x/Users/".

The key-value parameters that tb-rsync-vss understands are:

  • tmp-drive: A single letter. This is the temporary drive letter that the snapshot will be mapped to.
  • rsync: The non-cygdrive path to the cygwin rsync executable.
  • cygpath: If present and true, then the final path passed to rsync will be in cygdrive format (/cygdrive/x/Users/), otherwise it will be in normal DOS format (x:/Users/).
  • logfile: If present, a log file of this name will be opened on the Windows client.
  • verbose: If present and true, then verbose messages are logged to stderr and to the logfile, if one has been specified.
  • rsync-arg-0, rsync-arg-1, etc.: If present, these are added as arguments to rsync before it is executed. They are added as the first arguments to rsync. The original purpose is to allow the --fake-super parameter to be passed in to the local rsync, although any parameter(s) can be added. Example: |rsync-arg-0=--fake-super|.

Log messages

Log messages are always written to stderr. For dirvish, if there's an error, these will show up in the rsync_error file.

If you provide the logfile=<filepath> parameter, the messages will also be written to a file on the Windows computer. If you provide verbose=true, additional messages will be written to the log.

Log messages begin with a timestap, in asctime() format. For example:

Sat Mar 15 08:17:23 2014 startup

Example dirvish configuration

If you're using dirvish, here's a sample configuration file that will use tb-rsync-vss to run cygwin rsync. It assumes that cygwin is installed in c:\opt, adjust it for your particular configuation.

client: Administrator@windows-box.example.com
tree: "/|tmp-drive=x|rsync=c:/opt/cygwin/bin/rsync.exe|cygpath=true|logfile=c:/Users/myuser/tb-rsync-vss.log|c:/Users"
expire-default: +6 weeks
rsync-client: "c:/Program Files/True Blade Systems/tb-rsync-vss-64.exe"
expire-rule:
  wday { sun }               +3 months
  wday { sun } mday { 1-7 }  +1 year
  hour { 3am-1am }           +3 days
numeric-ids: 1

The double quotes used here are important.

Note that dirvish is told that the name of the client rsync executable is tb-rsync-vss-64.exe. This is how tb-rsync-vss gets control. tb-rsync-vss then takes a snapshot of the c: drive, maps it to the x: drive, and finally calls the real rsync as c:/opt/cygwin/bin/rsync.exe, telling it to back up /cygdrive/x/Users. You can either use backslashes or forward slashes for the various path names. Forward slashes make all of the parameter quoting much easier.

Release History

1.3 2015-01-19 Eric V. Smith

  • Remove duplicate "startup" message (issue #4).
  • Added timestamp to log messages (issue #3).
  • Only require one command line argument (issue #5).
  • Fix usage() output (issue #6).

1.2 2014-03-12 Eric V. Smith

  • Added logfile and verbose options.

1.1 2012-11-30 Eric V. Smith

  • Added rsync-arg- arguments to local rsync.
  • Added resource so version number can be seen through Windows Explorer.

1.0 2012-02-26 Eric V. Smith

  • Initial version.