Overview

HTTPS SSH

TestBed

Install the tests

brew install python2.7
virtualenv env
cd src/python/yawn
../../../env/bin/python setup.py develop
cd ../../../

Run the tests

. env/bin/activate
BROWSER=chrome python test.py

or

BROWSER=chrome env/bin/python test.py

If you want to customize the browser being used, or the IP address to connect to the selenium server on you can do this:

SELENIUMIP=192.168.0.5 BROWSER=IE8 env/bin/python test.py

Browser Setups

Firefox

If you don't want a console log available in the tests themselves, just specify BROWSER=Firefox like this:

BROWSER=Firefox env/bin/python test.py

For experimental console.log support see browser/firefox/README.md

Chrome

You'll need chromedriver:

brew install chromedriver

Then you can set the environment variable BROWSER to Chrome.

See also the Headless section below for an alternative approach that allows running Chrome as a headless browser in a VM.

Safari

Safari requires the selenium jar file to be present on your classpath:

cd browser/safari
curl -O -L http://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
cd ../../

You'll also need to install the extension. Load it like this and then click the dialog box:

cd browser/safari
unzip selenium-server-standalone-2.43.1.jar
cd org/openqa/selenium/safari
open SafariDriver.safariextz
cd ../../../../../../

Then you can set the environment variable BROWSER to Safari.

For example:

BROWSER=safari env/bin/python test.py

Internally, the script sets this variable to point to the selenium jar file, but only if it isn't already set elsewhere. It is best to check it isn't set unless you sepcifically want to point to a different Selenium version:

SELENIUM_SERVER_JAR=browser/safari/selenium-server-standalone-2.43.1.jar

Headless

Sometimes you want to run tests on a server rather than locally on a desktop. To do this you need some sort of virtual X11 implementation so that browsers can run. This section describes how to achieve such a set up.

Install

Install like this:

cd browser/headless
vagrant up

You'll see output similar to this:

Bringing machine 'testbed_headless' up with 'virtualbox' provider...
==> default: Importing base box 'precise-server-cloudimg-amd64-vagrant-disk1'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: testbed_headless_1413541867783_38596
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 4444 => 4444 (adapter 1)
    default: 9222 => 9222 (adapter 1)
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 4.1.12
    default: VirtualBox Version: 4.3
==> default: Mounting shared folders...
    default: /vagrant => /home/james/testbed/browser/headless
==> default: Running provisioner: shell...
    default: Running: /tmp/vagrant-shell20141017-9894-r68e4m
==> default: Adding the Google Chrome signing key and apt repo ...
==> default: OK
==> default: done
==> default: Adding the Google Chrome signing key and apt repo ...
==> default: done.
==> default: Updating sources ...
==> default: Installing  libexif12 openjdk-7-jre google-chrome-stable xvfb unzip firefox imagemagick ...
==> default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
==> default: Creating config file /etc/gconf/2/path with new version
==> default: Creating config file /etc/papersize with new version
==> default: done.
==> default: Fetching and installing chromedriver (using /tmp/testbed_install.tXm4) ...
==> default: Latest version is 2.11
==> default: Archive:  chromedriver_linux64.zip
==> default:   inflating: chromedriver            
==> default: done.
==> default: Fetching and installing selenium-server-standalone.jar ... 
==> default: done.
==> default: Adding an init script ...
==> default: done.
==> default: Starting Xvfb ...
==> default: done.
==> default: Starting Google Chrome ...
==> default: done.
==> default: Starting Selenium ...
==> default: Java pid is 7954
==> default: done.
==> default: Success!

You can ignore the lines in red (they don't cause problems):

==> default: dpkg-preconfigure: unable to re-open stdin: No such file or directory
==> default: Creating config file /etc/gconf/2/path with new version
==> default: Creating config file /etc/papersize with new version

Stop

To stop the VM, first make sure you are in the correct directory:

cd browser/headless

Then run:

vagrant halt

Start

To start the VM again, first make sure you are in the correct directory:

cd browser/headless

Then run:

vagrant up

The second time you run this it won't do a full install again, it will just boot the machine.

Running the tests

To run the tests, first make sure the VM is running:

cd browser/headless
vagrant up

Then back in the main directory:

cd ../../

Run a headless chrome like this:

SELENIUMIP=127.0.0.1 BROWSER=chrome env/bin/python test.py

This works by running a headless Chrome with chromedriver in the VM as well as a selenium server on port 4444. This port is forwarded from the VM to localhost so that you don't need to run a local selenium or chromedriver.

You can see the output at http://localhost:4444/wd/hub.

Incidentally, you don't need a chromedriver on the local machine for headless chrome. The chromedriver gets installed on the VM. Any errors you see in headless mode relate to chromedriver on the VM. This can be a bit confusing since they appear in the exception message on the local machine.

To run a headless Firefox run:

SELENIUMIP=127.0.0.1 BROWSER=firefox env/bin/python test.py

This works by connecting to the same Selenium server but this time the server starts a headless Firefox itself before running the tests. (The Chrome browser is still present in the background at the same time as Firefox in this case, but the tests are run in Firefox, not Chrome).

Getting files to and from the VM

You can copy files to and from the VM using SCP. The important thing is that Vagrant can give you the correct SSH options like this:

export SSH_OPTIONS=$( vagrant ssh-config | grep -v "Host " | awk -v ORS=' ' '{print "-o " $1 "=" $2}' )

You can then copy files from the VM like this:

scp $SSH_OPTIONS vagrant@localhost:/path/to/file .

This will copy them to the local directory.

You can copy files to the VM like this:

scp $SSH_OPTIONS local_file vagrant@localhost:/home/vagrant/

Restarting the environment

At any point if you want to re-start the environment (perhaps if things aren't working) you can run:

cd browser/headless
vagrant ssh -c "sudo /etc/init.d/headless restart"

Upgrading Chrome, Chromedriver and Firefox

Sometimes you might want to upgrade to the latest Chrome, Firefox and Chromedriver without destroying the VM and starting from scratch.

You can do this like this:

cd browser/headless
export SSH_OPTIONS=$( vagrant ssh-config | grep -v "Host " | awk -v ORS=' ' '{print "-o " $1 "=" $2}' )
scp $SSH_OPTIONS bootstrap vagrant@localhost:/home/vagrant/
vagrant ssh

Then on the VM run:

sudo bash bootstrap upgrade

This doesn't upgrade Selenium though, it just re-downloads the same version.

Taking Screenshots

You can take screenshots of the browser by using this in your tests:

self.driver.save_screenshot('home1.png')

That's no so interesting though. On the headless box you can also create a screenshot of the whole virtual X11 desktop. First make sure you are in the correct directory:

cd browser/headless

Then you can run this to take a Desktop screenshot:

vagrant ssh -c "DISPLAY=:10 import -window root /home/vagrant/screenshot.png"

And this to copy it to the local directory:

export SSH_OPTIONS=$( vagrant ssh-config | grep -v "Host " | awk -v ORS=' ' '{print "-o " $1 "=" $2}' )
scp $SSH_OPTIONS vagrant@localhost:/home/vagrant/screenshot.png .

Viewing Selenium Logs

The selenium logs are at /var/log/selenium.log on the VM. To view them (and see updates as tests are running), first make sure you are in the correct directory:

cd browser/headless

Then run this command:

vagrant ssh -c "tail -f /var/log/selenium.log"

IE8

Run these commands:

cd ie8
sh fetch.sh

... then follow the instructions.

After the VM boots you'll need to change the Windows path to include E:\. You can do that by clicking the Windows Start icon in the bottom right, right clicking "Computer" and choosing "Properties". Then in the "System" panel that pops up, click "Advanced system settings" from the list on the left. A "System Properties" window pops up. On the "Advanced" tab, click the "Environment Variables" button at the bottom and edit Path to add ;E:\ to the end.

You'll only need to set the path once, but each time you reboot the machine you'll need to start the Selenium server (described in a minute).

Now find out the IP address. Click the start button and then in the "Search programs and files" box enter cmd and press ENTER. At the Command Prompt that appears run ipconfig and you'll see all IP address on the system. You will want to choose the IP next to the "IPv4 Address" that corresponds to the Bridged IP. It will be the one that looks most similar to the IP of the host machine. Make a note of the IP as you'll use it as the value of SELENIUMIP variable on the host machine when you run the tests in a minute.

Finally before running the tests, you need to install Java and start the Selenium server. Click the start button then Computer and under "Network" double click the E: drive share. Within that is an install for Java. Copy it to the Desktop and run it from there.

Once Java is installed, double click on the file called run in the E:\ drive to load the Selenium server.

On the desktop you can then run this command (replacing xxx with the IP you have found above):

SELENIUMIP=xxx BROWSER=ie8 env/bin/python test.py

Build Setup

Inside the build folder is a build.sh script which you can customise to build the latest version of your tool.

Create the VM:

cd build
vagrant up

Run the build script:

vagrant provision

Ordinarily, the build tool will output a package which can be hosted in a repo on the build server to be accessed by the install and upgrade machines.

TODO: Fill out a sample script hosting an apt repo

Install setup

Once you have built a Debian package hosted in an apt repo in the running build VM, you'll want to test an install into an empty envrionment.

A Vagrantfile and install.sh file can be found in the install directory.

You can run an install like this:

cd install
vagrant up

Run the install script:

vagrant provision

Now that you have a running VM with a fresh install of your software, you can run the base isolation test suite against it.

Tip

You can use the vagrant snapshot take and go functionality to restore your VM to an empty state for each test, ensuring your each test works in isolation. Although this appears slower, the increased isolation will cause many fewer problems in the longer term as you get lots and lots of tests. It also means that you can run just a subset of your test suite.

TODO: Fill out the install.sh with an example, fetching from an apt-repo.

Upgrade setup

Most software isn't just about install into a fresh VM, it is important to be able to upgrade it.

It is unlikely you can run the same set of tests on the install box and the upgrade box because the whole point of an upgrade test is that the box already contains user data that isn't necessarily in an expected form.

You have two choices:

  • Set up test data from the old version you are upgrading from and then run an upgrade test from a known state
  • Write tests that make non-destructive changes and are very robust to unexpected data

The first choice is the better option. It can be tempting to try to use a copy of live data, but this can often be tricky due to both the amount of it and the sensitivity of what it contains.

Create the VM:

cd build
vagrant up

Run the upgrade script:

vagrant provision

When to automate

You should automate builds and tests at the very start of your project. The very first sprint should start with a bare-bones build and test system. It is easy to do well at this stage since your code is so small. Even your product just says "Hello World!". Once you have everything working, you can improve and grow your build and test code as your codebase grows.

Weaknesses

  • If you change your network settings (e.g. changing wifi network), you might need to halt and start all the VMs again.