1. Aristarkh Zagorodnikov
  2. s3backup

Overview

HTTPS SSH
This is a single-file utility to do backups to Amazon S3.
It requires AWS SDK for Ruby, see http://aws.amazon.com/sdkforruby/ for details.

Ubuntu pre-build instructions (libxml and libxslt are required for AWS SDK):
apt-get install rubygems libxml2-dev libxslt1-dev

To install dependencies you can either do it manually via gem install or do:
apt-get install ruby-bundler
bundle install

This utility solves a simple yet important task -- it automates synchronization of local files to Amazon S3 for backup purposes.
While there are several similar tools, most of them either have stability problems, have complex dependencies, use excessive system resources, etc.
The operation of this utility is simple -- it takes a source path and a combination of AWS S3 bucket and a prefix, then uploads changed files ("changed" is determined either by size only or size and MD5 checksum) to S3, while optionally removing missing files with age delay.

Typical invocation is as follows:
s3backup /path/to/source bucket/prefix [-c <path_to_aws_config>] [-r <recycling_age_in_days>]
Run "s3backup -h" to discover additional options, it has multiple useful features like dry running, ability to purge bucket before syncronizing, etc.

Please note, that this utility is not intended to synchronize large sets of files in one turn (in terms of files/directories count, not in terms of space) -- attempting to synchronize several millions of files would likely suffer from performance problems.
Also, the largest file it can handle with current settings is about 250GB. If you need it to work with larger files, please create an issue.