EE5175-Lab 2 Occlusion Detection & Image Mosaicking Solved

Occlusion Detection

In this problem, we are going to revisit the Occlusion Detection task from Lab-1. The task is to estimate the homography matrix using SIFT based features, assuming that you don’t know the underlying transformations and the point correspondences.

1.1      Steps
•    Taking IMG2.pgm as the reference image, determine homography H between I2 = IMG2.pgm and I1 = IMG1.pgm such that I1 = H × I2.

•    Transform I2 using the estimated homography H (target-to-source mapping) and determine the changes between the two images.

Verify that the underlying transformations consist of in-plane rotation and translation. And also that the values matches with those computed in Lab-1.

1.2            Determining homography between two images
1.   Determine SIFT features of the two images and determine correspondences between them.File sift_corresp.m returns the SIFT correspondences between two images (see Section 1.4).

Now to find H such that:

 0   xi        xi

yi0 ∼ H yi

                                                                                                 

                                                                                     1                 1

2.   Run RANSAC on matched points (correspondences) to remove outliers (wrong matches), andfind the homography between the two images.

(a)     Input: Matched points (xi,yi) and (x0i,yi0) with i ∈ M.

(b)    Randomly pick four correspondences (so that we can form eight equations), i.e. (xi,yi) and   with i ∈ R ⊂ M and |R| = 4, where | · | denotes the cardinality of the set.

(c)     Calculate the homography H using the above four correspondences (see Section 1.3).

(d)    For each of the remaining correspondences (xi,yi) and (x0i,yi0) with i ∈ P = M\R, check whether they satisfy the homography (within an error bound). If yes, add the index of that correspondence to the consensus set.

 00   xi          xi

 y00 ← H yi, and normalize so that, i

                                                                                   

00

                                                                    zi                                      1

i.e. x00i ← x00i /zi00 and yi00 ← yi00/zi00

If  , then update consensus set C ← C ∪ {i}.

(e) If the consensus set is large enough i.e. if |C| d(= 0.8|P|), then return this homography

H, else go to step (b).

(f) Output: Homography H.

1.3         Calculating homography

1. Consider a correspondence (x,y) and (x0,y0),
 
 
 0 

                                                                             x             h1

 0 = h4 y 

  

                                                                             z0               h7
h2 h5 h8
 

h3               x

h6y.

 

h9               1
Upon normalizing z0,

x0 = h1x + h2y + h3/h7x + h8y + h9, y0 = h4x + h5y + h6/h7x + h8y + h9.

Form two equations for each correspondence (corresponding to two rows of matrix A).

(x)h1+(y)h2+(1)h3+(0)h4+(0)h5+(0)h6

− (x0x)h7 − (x0y)h8 − (x0)h9 = 0

(0)h1+(0)h2+(0)h3+(x)h4+(y)h5+(1)h6

− (y0x)h7 − (y0y)h8 − (y0)h9 = 0

2.   Solve the system,

  h1

h2



A8×9 ...  = 0





  h9

i.e., find the null space of A.

3.   Homography matrix



h1

H = h4



h7
h2 h5 h8
 h3

h6.



h9
1.4         Using SIFT
Files needed:

1.   sift_corresp.m

2.   sift.m and

3.   sift (for GNU/Linux) or siftWin32.exe (for Windows).

Copy the above three files to the working directory. In GNU/Linux, check that the file sift has executable permission. If not, run chmod +x sift in a terminal.

Usage: [corresp1, corresp2] = sift_corresp(‘img1.pgm’,‘img2.pgm’)

2        Image mosaicing

2.1         Problem Statement
Image mosaicing is the alignment and stitching of a collection of images having overlapping regions into a single image. In this assignment, you have been given three images which were captured by panning the scene left to right. These images (mosaic1.pgm, mosaic2.pgm and mosaic3.pgm) capture overlapping regions of the same scene from different viewpoints. The task is to determine the geometric transformations (homographies) between these images and stitch them into a single image.

2.2      Steps
1.   Take mosaic2.pgm as the reference image.

2.   Determine homography H21 between I2 = mosaic2.pgm and I1 = mosaic1.pgm such that I1 = H21I2.

3.   Determine homography H23 between I2 = mosaic2.pgm and I3 = mosaic3.pgm such that I3 = H23I2.

4.   Create an empty canvas. For every pixel in the canvas, find corresponding points in I1, I2 and I3 using H21, identity matrix and H23 respectively (target-to-source mapping). Blend the three values by averaging them. Employ the values in blending only if it falls within the corresponding image bounds. Choose the origin so as to get a full mosaic.

2.2.1          Creating the mosaic

Pseudo-code:

canvas = zeros(NumCanvasRows,NumCanvasColumns); for jj = 1:NumCanvasColumns

for ii = 1:NumCanvasRows

i = ii - OffsetRow; j = jj - OffsetColumn;

tmp = H21 * [i;j;1]; i1 = tmp(1) / tmp(3); j1 = tmp(2) / tmp(3);

tmp = H23 * [i;j;1]; i3 = tmp(1) / tmp(3); j3 = tmp(2) / tmp(3);

v1 = BilinearInterp(i1,j1,I1); v2 = BilinearInterp(i,j,I2); v3 = BilinearInterp(i3,j3,I3);

canvas(ii,jj) = BlendValues(v1,v2,v3);

end

end

where (OffsetRow, OffsetColumn) can be chosen such that the final mosaic will fit within the canvas.

2.3          Capturing your own data
Use your mobile phone camera to capture images (three or more), and run your code to generate the mosaic. Ensure that there is adequate overlap between successive images, and the camera is imaging a far-away scene (think why). Bring down the resolution of the input images such that the width < 1000 pixels, and convert them to grayscale before using them for mosaicing. (In MATLAB, use ‘imresize’ to reduce the image resolution, and ‘rgb2gray’ for conversion to grayscale)