CV1- Assignment 4: Image Alignment and Stitching Solved

General guidelines

Your    source   code    and    report    must    be    handed    in    together    in    a    zip    file

(ID1 ID2 ID3 ID4.zip) before the deadline by sending it to Canvas Lab 4 Assignment. For full credit, make sure your report follows these guidelines:

•    Include an introduction and a conclusion to your report. • The maximum number of pages is 10 (single-column, including tables and figures). Please express your thoughts concisely. The number of words does not necessarily correlate with how well you understand the concepts. • Answer all given questions (in green boxes). Briefly describe what you implemented. Blue boxes are there to give you hints to answer questions.

•    Try to understand the problem as much as you can. When answering a question, give evidences (qualitative and/or quantitative results, references to papers, etc.) to support your arguments.

•    Analyze your results and discuss them, e.g. why algorithm A works better than algorithm B in a certain problem.

•    Tables and figures must be accompanied by a brief description. Do not forget to add a number, a title, and if applicable name and unit of variables in a table, name and unit of axes and legends in a figure.

.
1             Image Alignment
In this practice, you will write a function that takes two images as input and computes the affine transformation between them. You will work with supplied boat images. The overall scheme is as follows:

Detect interest points in each image.
Characterize the local appearance of the regions around interest points.
Get the set of supposed matches between region descriptors in each image.
Perform RANSAC to discover the best transformation between images.
Hint
The first three steps can be performed using David Lowe’s SIFT. Check out the Docs of SIFT related function for further information in the following link: https://docs.opencv.org/master/da/df5/tutorial_py_sift_intro.html and https://docs.opencv.org/3.4.9/d5/d3c/classcv_1_1xfeatures2d_1_1SIFT. html

For some patent issue, the newest version of OpenCV does not contain SIFT related function. However, you can install a old version (for example: opencv-python==3.4.2.17 and opencv-contrib-python==3.4.2.17)

The final stage, running RANSAC, should be performed as follows:

Repeat N times:
Pick P matches at random from the total set of matches T.
Construct a matrix A and vector b using the P pairs of points and find transformation parameters (m1,m2,m3,m4,t1,t2) by solving the equation Ax = b.
Using the transformation parameters, transform the locations of all T points in image1. If the transformation is correct, they should lie close to their counterparts in image2. Plot the two images side by side with a line connecting the original T points in image1 and transformed T points over image2.
Count the number of inliers, where inliers are defined as the number of transformed points from image1 that lie within a radius of 10 pixels of their pair in image2.
(1)

It can be rewritten as

(2)

or

(3)

Figure 1: Affine transformation. • If this count exceeds the best total so far, save the transformation parameters and the set of inliers.

End repeat.
Finally, transform image1 using this final set of transformation parameters. If you display this image, you should find that the pose of the object in the scene should correspond to its pose in image2. To transform the image, implement your own function based on nearest-neighbor interpolation. Then use the OpenCV built-in function warpAffine and compare your results.
Hint
Nearest neighbors does not mean you have to classify points. The problem is that if you have a transformation, then the transformed points may not be at perfect pixels (e.g., 0.3px). Instead of linear interpolation, which requires more work to implement, we can just use nearest neighbors which means simply rounding the coordinates.

Question - 1 (55-pts)
Create a function that takes image pairs pgm and boat2.pgm as input, and return the keypoint matchings between the two images. Name your script as keypoint matching.py.
Take a random subset (with set size set to 10) of all matching points, and plot on the image. Connect matching pairs with lines. You can assign a random color to each line to make them easier to distinguish.
Create a function that performs the RANSAC algorithm as explained above. The function should return the best transformation found. For visualization, show the transformations from image1 to image2 and from image2 to image1. Name your script as py.
Include a demo function to run your code.

Question - 2 (5-pts)
How many matches do we need to solve an affine transformation which can be formulated as shown in Figure 1?
How many iterations in average are needed to find good transformation parameters?
2             Image Stitching
In this part, you will write a function that takes two images as input and stitch them together. The method described in the previous section will be used to stitch two images together by transforming one of them to the coordinate space of the other. You will work with supplied images left.jpg and right.jpg. The overall scheme can be summarized as follows:

As in previous task you should first find the best transformation between input images.
Then you should estimate the size of the stitched image.
Finally, combine the jpg with the transformed right.jpg in one image.