preliminarities and whatnot
this is a recipe for creating a close-to-perfect clone of your production VPS box to a virtual machine that you can run locally as a VM guest on your development machine. i did this using a VPS from slicehost (slicehost rocks) running ubuntu server 9.0.4, and VMware fusion on a macbook pro. the recipe i am showing below is totally ubuntu-centric, but i am pretty sure that the general idea could be adapted to accomplish the same ends with a lot of different combinations of production server, server OS, development machine and OS, and virtual machine infrastructure. but your mileage may vary. i know this works with ubuntu, slicehost, OS X, and VMware fusion.
system requirements: a production server, preferably of the VPS flavor (maybe this technique could be adapted to a shared hosting setup, but i think it would be pretty hard); a development machine; software to create and run a virtual machine (VM) (using the same version of the same OS that your production server is running) as a guest on your development machine (i.e. VMware, QEMU, VirtualBox, etc.).
human requirements: you should be comfortable on the *nix command line and have a basic pragmatic understanding of VM concepts and methodologies (i.e. you know how to create and run a virtual machine on your development box).
the guts of the cloning procedure is an adaptation of an apparently no-longer available tutorial by 'defcon', so cred to defcon wherever you are.
there are certainly endless refinements, extensions, and adaptations that could be made and i hope that people who know more than me will add to and correct this guide. to that end, i'm hosting the guide itself in the form of an open source wiki on bitbucket. so edit it, clone it, patch it, use the issue tracker to report my bungles and/or make your own additions. [there is no "source" code for this project other than the wiki source. this wiki is the project.]
wherever it is necessary to edit a system file in the instructions below, i am using nano as my terminal editor of choice. wherever you see 'nano', feel free to substitute vi, vim, ed, emacs, or whatever magic potion you usually use to modify system text files from the command line. and again, the locations of the files and the commands i am showing are for ubuntu v. 9.0.4 server edition, so if that is not your production OS, you may need to adapt and abide.
step 1 : make your VM
start by creating a new virtual machine and doing a basic install of the SAME VERSION of the server OS that you have running on your production VPS box. in my case, my production VPS is a slicehost slice running ubuntu jaunty [9.0.4], so i went to ubuntu.com and downloaded the 9.0.4 server iso and used that to create a VM with VMware Fusion on the mac. the nitty gritty detail on creating a VM is outside the scope of this recipe, but all you need to do is create a basic VM with a vanilla server install. here are a few screenshots of the process on VMware fusion on the mac. if you need more detail than this, consult google and/or the VMware (or other virtual machine infrastructure) documentation.
[note: if you don't see any images below here, you may need to manually add trust for a server cert from 'bytebucket.org'. I guess this is some kind of bitbucket/atlassian issue…]
the one "non-vanilla" thing that i installed on the basic VM clone, was the SSH Server.
you probably don't need to install any of the other packages, and don't spend any time cleaning up the install on the VM. we're about to copy your production machine (which has everything you need and is all up to date, right?) right over most of it.
so now when the VM reboots, log in on the VM console and do the rest of the commands in this recipe from inside your home directory on the VM.
step 2 : save your /etc/fstab
sudo cp /etc/fstab /etc/fstab-orig
step 2a : [optional] insurance against borkage
at this point, just to save the work on the VM install, i shut the VM down
[sudo shutdown -h now] and make a copy of the VM in case something goes wrong in the rest of the process and you need to start over clean. this step will avoid having to repeat step 1 and 2. and it can also be nice to have a clean base install VM lying around to make additional test boxen in the future.
step 3 : tar your shizzle
log into your production VPS (hopefully via SSH) and be in your home directory on the VPS. we're gonna tar up all the stuff on your box that is not hardware specific. for me that means /etc /home /opt /tmp /usr /var. [note: i'm actually not sure whether /lib should or should not be included. i excluded it, and that is why (i believe) i had to install openSSH from the ubuntu install media.] you should take a look in / and see whether there are any other directories off of there that you have created or otherwise believe should be copied to your VM.
here is the tar command [modify as necessary if you need to add or subtract other dirs off /]:
sudo tar -cvzf ~/clone.tgz --exclude ~/clone.tgz /etc /home /opt /tmp /usr /var
[sudo, because there might be stuff that you do not have read access for]
this will take a few minutes to complete as your shizzle whizzes up the terminal.
step 4 : pull your shizzle
after step 3 completes, you should have a file in your home directory called clone.tgz. you want to get that file into your home directory on the VM. it doesn't matter how you do this, but i used scp as follows:
[be in your home directory in the VM]
scp -P XXXX me@my_slice:clone.tgz .
[where me@my_slice is the SSH user@hostname for your production box and XXXX is the port number you have SSH running on. you can omit the "-P XXXX" if you have SSH on the standard port 22. also, if you have your VPS box configured to only accept public key certs instead of password logins, you may need to move some keys around to get this to work]
depending on your bandwidth and the size of your shizzle, this might take anywhere from a few minutes to an hour.
step 5 : untar your shizzle
so you are still in your home directory on the VM. do this:
sudo tar -xvpzf ~/clone.tgz -C /
this will take another few minutes of shizzle whizzing.
step 6 : minor tweakage
there are few little tweaks you need to do before you reboot the VM and it comes back to life as a nearly perfect clone of your production VPS box.
first, restore your VM's fstab:
sudo mv /etc/fstab-orig /etc/fstab
next, change the hostname so you can tell it apart from your production box:
sudo nano /etc/hostname
[change the name to whatever you want your test box to be called]
sudo nano /etc/hosts
[you should have an entry for 127.0.0.1 that is your production box hostname again -- change it to match what you put in hostname]
and though we are not quite done, you can enjoy preliminary rejoicing. when the VM comes back up, whatever user you set up in the VM install of ubuntu will be gone and whatever user(s) were set up on your production box will be there. other than the hostname which you changed, everything should be pretty much the same as your production box at this point which is mostly excellent, except for one thing...
step 7 : dial in your networking
your production box is almost certainly set up with a static IP address on a network which will not work for your VM test clone. so do this:
sudo nano /etc/network/interfaces
at the bottom of the file, you should see something like this [with numbers where i put X's]:
iface eth0 inet static address XXX.XX.XX.XX netmask 255.255.255.0 gateway XXX.XX.XX.X dns-nameservers XXX.XX.XX.X XXX.XX.XX.X
we need to edit that section and replace what's there with numbers that are congruent with the network that your VM is on. there are endless different ways to configure this, depending on your needs and preferences, and the VM infrastructure you are using, and i can't cover them all. VMware gives you three choices for the network: bridged, NAT, and host-only. for me, using NAT for this test box makes the most sense because it lets me access the VM from my macbook on a consistent, static IP address, and it also lets the VM access the internet when necessary. so i configured the VM with a static IP on the VMware NAT network (which VMware usually calls vmnet8). on my install of VMware Fusion 2.05, that network is 172.16.188.xxx so i edited the above block to look like this:
iface eth0 inet static address 172.16.188.100 netmask 255.255.255.0 gateway 172.16.188.2 dns-nameservers 184.108.40.206 220.127.116.11
[those DNS servers, btw, are for openDNS and i recommend that you use them, but i think that you could also use the vmnet gateway as well -- which is your host machine on the VMware NAT network, in my case 172.16.188.2. VMware is configured to run dhcp on the range of addresses from .129 to .254. so i picked a static address outside of that range.]
there are probably a lot of ways to find out what the correct network numbers are for your VM. with VMware, i know they are different for different versions of the product. if you are on OS X like i am, and running fusion v. 2.0.x you can look at /Library/Application Support/VMware Fusion/vmnet8/nat.conf and see what network is configured in there.
a more generic way is to briefly configure your VM to get an address from dhcp and then see what address it gets. to do that, instead of the 5-line block above, just temporarily put in this one line:
iface eth0 inet dhcp
and then save and do this:
sudo /etc/init.d/networking restart
it should acquire a new address from the host dhcp server and tell you what it is. then go back and edit /etc/network/interfaces again and fill in the whole block of static IP config data using the network numbers that the host network gave you. i think that you will definitely want to have a static IP for your test machine.
lastly, you should edit your resolv.conf file and put in the same name servers that you used above. do this:
sudo nano /etc/resolv.conf
mine looks like this (again using the openDNS name servers):
nameserver 18.104.22.168 nameserver 22.214.171.124
after you get all those network numbers configured how you want them, do this:
sudo /etc/init.d/networking restart
and your perfect test box should be ready to roll.
you should be able to SSH into the box now from a terminal on your host machine, using the static IP address that you set up in step 7. depending on your configuration, you may want to go into the web server configs in the VM and change whatever hostname based virtual servers you have to work with new "test" names. and you will probably also want to edit /etc/hosts on your development machine and add some of those new test names pointing at the IP address that you gave to the VM.
that's it then.
use this project's issue tracker for any comments, questions, corrections, etc.