Logo replacement using Shape Context feature matching

The aim of this project is to detect a known logo (at arbitrary scale and rotation) in test images and replace it with another logo such that the replacement is not obvious. The most important part is obviously to detect the logo and for this I use Shape context features described in Malik's paper Shape Matching and Object Recognition Using Shape Contexts. The full procedure is described below with relevant screenshots:

1. Extract Shape context features at SIFT keypoints. I modified the context description in the paper to include the scale and rotation information obtained from SIFT keypoints. The radius of the biggest circular bin around the keypoint is proportional to the scale and the angular bins are rotated such that the first bin aligns with the keypoint orientation. This makes the Shape Context descriptors scale and roation invariant. As described in the paper, I extract SC descriptors on Canny edge points. In the following figures, the yellow circles show the radius of the radial bin farthest away from the keypoint and the magenta lines show the orientation of the first angular bin.

2. Match SC features using the Hungarian minimum-cost assignment method (with dummy nodes for capturing outliers) as mentioned in the paper.

3. Use RANSAC to select matches that provide a consistent homography.

4. Estimate a full Thin Plate Spline transform from reference logo image to logo instance in test image (also as described in the paper) using the matches and get a bounding box by transforming the four corners of the logo reference image.

5. Erase the logo instance in the bounding box by assuming that the top-left corner of the bounding box is background (and anything that is different from it in an SSD sense is foreground).

6. Paste the replacement logo on the image obtained from previous step. Care is taken to paste just the foreground of the replacement logo.

Some other results are shown below (detecting Penn logo and replacing with Starbucks logo):