This repository provides base functionality to perform unsupervised or semi-supervised alignment of an unaligned set of images containing examples of the same semantic class such as "cars", "motorbikes" or "dogs". The repository also contains scripts to replicate several experiments published in the scientific works (see References). The code is mainly Matlab code.

CITING If you use this code or parts of it in your publications, reports, Web pages etc. we kindly ask you to cite one of our works (see the references section) to provide us a little "scientific coin".

TABLE OF CONTENTS

Introduction
Installation
Data
Examples
References

Introduction

The principle idea is that you have a set of images containing visual examples of the same class (e.g. "motorbikes"), but the examples are in various poses (translation, scaling, rotation), slightly varying 3D view point and can be visually very different (e.g., "sport bike" vs. "Harley-Davison") and you prefer to have them aligned, i.e. when you compute an average image it would look like a class example itself. For that purpose we need unsupervised image alignment methods.

Installation

Fetch files from the Bitbucket repo:

~$ cd <MYWORKDIR>
~$ git clone git@bitbucket.org:kamarain/image_alignment-code.git

Fetch also the necessary helper function toolbox:

~$ hg clone https://bitbucket.org/kamarain/mvprmatlab

Note: If you will use the "conf.storeResults" option of some functions you need to install the export_fig package from http://www.mathworks.com/matlabcentral/fileexchange/23629-export-fig and make it available in your Matlab path.

Next, install the datasets used in the experiments (at least Caltech-101) and then you're ready to run our examples.

Data

Our examples and experiments are based on the following three datasets:

Caltech-101
r-Caltech-101
ImageNet (ILSVRC2012 classification data)

Instructions to fetch data and necessary annotation is given in the next sections.

Caltech-101

Caltech Computational Vision Group provides the Caltech-101 dataset of 100 different visual classes and a background class. The images are capture from the roughly similar view point and for class example alignment the dataset provides a good benchmark as the problem can be solved accurately using 2D transformations.

Caltech-101 dataset provides

Images organised into category specific directories
Manual annotations of object outlines that describe the object foreground areas rather accurately (no occlusions)

Install these by directly downloading them from the Caltech-101 Web site:

$ cd <MY_DATA_DIR>
$ mkdir caltech101; cd caltech101
$ wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz
$ tar zxfv 101_ObjectCategories.tar.gz
$ wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/Annotations.tar
$ tar -xvf Annotations.tar
$ wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/show_annotation.m

Then, since the Caltech101 data is a bit buggy - or weird, depends on your opinion - you need to do the following subdirectory renaming:

$ cd <MY_DATA_DIR>/caltech101/Annotations
$ cp -r Faces_3 Faces_easy
$ cp -r Airplanes_Side_2 airplanes
$ cp -r Motorbikes_16 motorbikes

In addition to the original data, you need annotated landmarks to benchmark image alignment methods. You may annotate you own landmarks (use, for example, mvpr_anno_imglist.m function in the mvprmatlab helper repo), or you may use the landmarks we have carefully selected and annotated:

landmarks.tar.gz - download and extract to the caltech101 directory.

r-Caltech-101 (randomized Caltech-101)

The construction and some visual object class detection experiments for the randomized Caltech-101 were published in

Making Visual Object Categorization More Challenging: Randomized Caltech-101 Data Set (T. Kinnunen, J.-K. Kamarainen, L. Lensu, J. Lankinen, H. Kälviäinen), In 20th Int. Conf. on Pattern Recognition (ICPR2010), 2010. PDF

The scripts to generate the dataset are available at the project Web page ( http://www2.it.lut.fi/project/visiq/), but they are bit buggy and since the directory structure does not correspond to the original Caltech-101 it is better to download a pre-generated dataset from:

Will be added soon when we find a good storage service for public data

~$ cd <MYDATADIR>
~$ tar zxfv r-caltech101.tar.gz

That will generate directory structure that is similar to Caltech-101, but note that the annotations miss the outline information.

ImageNet

WARNING The dataset is rather huge (approx. 450GB depending a bit on your system) and here we need only a few classes (e.g. acoustic guitar has the ImageNet ID n02676566 so it is sufficient if you download only the images for the classes used in the experiments) - Note that for all experiments you may start with the Caltech-101 images only.

Details can be found from the ImageNet Large Scale Visual Recognition Challenge Web page: http://www.image-net.org/challenges/LSVRC/. The dataset contains millions of images of thousands of object classes (browse examples via http://www.image-net.org/).

How to download full ImageNet data

Register via the challenge Web page and they will provide you a link via which you can download the full image sets (within the next 7 days).

Development KIT

The development kit that is mainly useful to run the evaluation scripts for comparable numbers and is available at the project Web page (code is not very well document and certainly not elegant):

Development kit 2014 (download)

Examples

The following examples can be run for different classes in Caltech-101, r-Caltech-101 and ImageNet (download instructions above) by changing the configuration files. In the beginning it is easiest to start with only Caltech-101 images since they are the most straightforward to download and use.

Supervised alignment using bounding boxes

Virtually all visual class benchmark datasets (Caltech-101, Caltech-256, Pascal VOC, ImageNet) provide bounding boxes for all or a significant number of training images. Class examples can be spatially aligned my mapping the bounding box corners as close as possible to each other. This is demonstrated in the first demo:

#!matlab
~$ ln -s <MYDATADIR>/caltech101/
~$ cp image_alignment_demo1_conf.m.template image_alignment_demo1_conf.m
EDIT (paths etc.) image_alignment_demo1_conf.m
~$ nice matlab -nodesktop
>> addpath <MYWORKDIR>/mvprmatlab/
>> image_alignment_demo1

Unsupervised direct alignment using pre-defined seed

This example demonstrates the unsupervised alignment method in Ref. 1. Note that you need to install the VLFeat Toolbox and for certain detectors/descriptors also OpenCV as described in the MVPRMATLAB Wiki. The demo configuration file contains pre-configured settings for several classes, but you may use it to align any set of images by simply adding your own configuration file. You may also evaluate the alignment accuracy if you provide annotated landmarks or bounding boxes for you images.

#!matlab
~$ cp image_alignment_demo2_conf.m.template image_alignment_demo2_conf.m
EDIT (paths etc.) image_alignment_demo2_conf.m
~$ nice matlab -nodesktop
>> addpath <MYWORKDIR>/mvprmatlab/
>> addpath base/
>> addpath <MYWORKDIR>/vlfeat/toolbox
>> vl_setup
>> addpath <MYWORKDIR>/export_fig        /* or set conf.storeResults=false */
>> image_alignment_demo2

Congealing

Our competitor is the congealing method from University of Massachusetts Vision Lab:

Code : Congealing Complex Images

Compiling congealing code

The code can be cloned from Huang's Bitbucket repo: https://bitbucket.org/gbhuang/congealreal

Their Makefile is crap and you need to replace it with the following:

#!Makefile

IFLAGS = `pkg-config --cflags opencv` -O3
LFLAGS = `pkg-config --libs opencv`

all: congealReal funnelReal

clean:
        rm congealReal funnelReal

congealReal: congealReal.cpp
        g++ $(IFLAGS) -c congealReal.cpp
        g++ congealReal.o $(LFLAGS) -o congealReal

funnelReal: funnelReal.cpp
        g++ $(IFLAGS) -c funnelReal.cpp
        g++ funnelReal.o $(LFLAGS) -o funnelReal

If you have a local installation of the OpenCV the pkg-config may not work and you first need to point where to search for the package information, e.g.: ´´´ export PKG_CONFIG_PATH=/home/kamarain/Work/ext/opencv-2.4.4/build/unix-install/ ´´´ The typing 'make' should work OK!

Testing the code

There is a simple demo in our directory:

~$
[EDIT congealing_demo1_conf.m AND run_congealing.sh]
~$ nice matlab -nodesktop
>> addpath ~/Work/bitbucket/mvprmatlab/
>> congealing_demo1

The demo demonstrates how the congealing improves the alignment of a set of artificially distorted images (left is the input and the right is the output landmarks of geometrically distorted images):

There is also another demo (congealing_demo2.m) that can be used to align and plot results (alignment accuracy and average image) for own image sets. The code is still rather experimental and full of hacks as we implemented it quickly to compare our method to theirs. It however seems to work with our data (see the average image of congealed r-Caltech-101 motorbikes below).

~$
[EDIT congealing_demo2_conf.m
~$ nice matlab -nodesktop
>> addpath ~/Work/bitbucket/mvprmatlab/
>> addpath base
>> congealing_demo2

Similarity Graph Image Alignment

This method is based on our extension of the direct alignment using a similarity graph structure (ref 2.).

Requirements:

Linux
Make sure that you have already done the compilation given in <MYWORKDIR>/mvprmatlab/opencv_descriptors/README
From http://kahlan.eps.surrey.ac.uk/featurespace/web/desc/ make a copy of one of the following files in the directory of <MYWORKDIR>/mvprmatlab/binaries
- compute_descriptors_32bit.ln
- compute_descriptors_64bit.ln

Testing the code

To run the code, you first need to set the parameters in configuration file according to your needs:

#!matlab

~$
[EDIT graphalignment_demo1_conf1.m ]
% Select a dataset and its class
classes = {'r-caltech101-stop_sign',... %1
           'r-caltech101-faces',...     %2
           'r-caltech101-motorbikes',...%3
           'r-caltech101-carside',...   %4
           'caltech101-airplanes',...   %5
           'caltech101-starfish',...    %6
           'ImageNet-meerkat',...       %7
           'LFW5000'};                  %8
classno = 7; % one of above (1-8)

experiment={'StepWise VS Direct',...    %1   Figure.3 in CVPR2015 paper
            'MST vs Dijkstra',...       %2   Figure.4 in CVPR2015 paper
            'Dijkstra VS Direct',...    %3   Figure.5 in CVPR2015 paper      
            'MST VS Direct' };          %4
expNo=1; %one of above (1-4)

Set the directory of images and their landmarks in the configuration file:

#!matlab
~$
[EDIT graphalignment_demo1_conf1.m ]

conf.imgDir = '<MYWORKDIR>/r-caltech101/101_ObjectCategories/';
conf.gtDir  = '<MYWORKDIR>/r-caltech101/landmarks/';

You can also see the constructed graph and Minimum spanning tree:

#!matlab
~$
[EDIT graphalignment_demo1_conf1.m ]

conf.GraphVisualize = false;  % set it true for graph and MST visualisation
                              %Note: Biograph library of matlab is needed.

After finishing the settings, we run the code to extract features and estimates pair-wise visual similarity. Note that this part is very time consuming.

#!matlab
~$
[EDIT graphalignment_demo1_conf1.m ]

conf.showResults = false;    % important for seedMethod all

~$ matlab
>> addpath '<MYWORKDIR>/mvprmatlab/'
>> addpath '<MYWORKDIR>/vlfeat-0.9.16/toolbox/'
>> addpath '<MYWORKDIR>/altmany-export_fig-d966721/'
>> addpath base_CVPR
>> addpath base
>> vl_setup
>> graphalignment_demo1_requirement.m

A folder named "TEMPWORK_" including .mat files should be created on your current directory. Now we can run the experiments of graph alignment.

#!matlab
~$
[EDIT graphalignment_demo1_conf1.m ]

conf.showResults = true;    % important for seedMethod all

~$ matlab
>> graphalignment_demo1.m

Experiment on LFW5000 dataset

References

References to the scientific papers that contains the scientific and technical details of the implemented methods:

Local Feature Based Unsupervised Alignment of Object Class Images (J. Lankinen, J.-K. Kamarainen), In British Machine Vision Conference (BMVC2011), 2011. PDF
Unsupervised Visual Alignment with Similarity Graphs (F. Shokrollahi~Yancheshmeh and K. Chen and J.-K. Kämäräinen), In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015. PDF