Starting from:
$30
CSE527 Homework2-SIFT features for scene matching and object recognition Solved
In this homework you will experiment with SIFT features for scene matching and object recognition. You will work with the SIFT tutorial and code from the University of Toronto. In the compressed homework file, you will find the tutorial document (tutSIFT04.pdf) and a paper from the International Journal of Computer Vision (ijcv04.pdf) describing SIFT and object recognition. Although the tutorial document assumes matlab implemention, you should still be able to follow the technical details in it. In addition, you are STRONGLY encouraged to read this paper unless you’re already quite familiar with matching and recognition using SIFT.
There are 3 problems in this homework with a total of 100 points. Two bonus questions with extra 5 and 15 points are provided under problem 1 and 2 respectively. The maximum points you may earn from this homework is 100 + 20 = 120 points. Be sure to read Submission Guidelines below. They are important.
Using SIFT in OpenCV 3.x.x in Local Machine
Feature descriptors like SIFT and SURF are no longer included in OpenCV since version 3. This section provides instructions on how to use SIFT for those who use OpenCV 3.x.x. If you are using OpenCV 2.x.x then you are all set, please skip this section. Read this if you are curious about why SIFT is removed https://www.pyimagesearch.com/2015/07/16/where-did-sift-and-surf-go-in-opencv-3/ (https://www.pyimagesearch.com/2015/07/16/where-did-sift-and-surf-go-in-opencv-3/).
We strongly recommend you to use SIFT methods in Colab for this homework, the details will be described in the next section.
However, if you want to use SIFT in your local machine, one simple way to use the OpenCV in-built function SIFT is to switch back to version 2.x.x, but if you want to keep using OpenCV 3.x.x, do the following:
1. uninstall your original OpenCV package
2. install opencv-contrib-python using pip (pip is a Python tool for installing packages written in Python), please find detailed instructions at https://pypi.python.org/pypi/opencv-contrib-python
(https://pypi.python.org/pypi/opencv-contrib-python)
After you have your OpenCV set up, you should be able to use cv2.xfeatures2d.SIFT_create() to create a
SIFT object, whose functions are listed at http://docs.opencv.org/3.0beta/modules/xfeatures2d/doc/nonfree_features.html (http://docs.opencv.org/3.0beta/modules/xfeatures2d/doc/nonfree_features.html)
Using SIFT in OpenCV 3.x.x in Colab (RECOMMENDED)
The default version of OpenCV in Colab is 3.4.3. If we use SIFT method directly, typically we will get this error message:
error: OpenCV(3.4.3) /io/opencv_contrib/modules/xfeatures2d/src/sift.cpp:1207: erro r: (-213:The function/feature is not implemented) This algorithm is patented and is excluded in this configuration; Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function 'create'
One simple way to use the OpenCV in-built function SIFT in Colab is to switch the version to the one from 'contrib'. Below is an example of switching OpenCV version:
1. Run the following command in one section in Colab, which has already been included in this assignment:
pip install opencv-contrib-python==3.4.2.16
2. Restart runtime by
Runtime - Restart Runtime
Then you should be able to use use cv2.xfeatures2d.SIFT_create() to create a SIFT object, whose functions are listed at http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html (http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html)
Some Resources
In addition to the tutorial document, the following resources can definitely help you in this homework:
http://opencv-pythontutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html (http://opencvpython-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html) http://docs.opencv.org/3.1.0/da/df5/tutorial_py_sift_intro.html (http://docs.opencv.org/3.1.0/da/df5/tutorial_py_sift_intro.html) http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html?highlight=sift#cv2.SIFT (http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html?highlight=sift#cv2.SIFT) http://docs.opencv.org/3.0beta/doc/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html
(http://docs.opencv.org/3.0beta/doc/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html)
Problem 1: Match transformed images using SIFT features
{40 points + bonus 5} You will transform a given image, and match it back to the original image using SIFT keypoints.
Step 1 (5pt). Use the function from SIFT class to detect keypoints from the given image. Plot the image with keypoints scale and orientation overlaid.
Step 2 (10pt). Rotate your image clockwise by 60 degrees with the cv2.warpAffine function. Extract SIFT keypoints for this rotated image and plot the rotated picture with keypoints scale and orientation overlaid just as in step 1.
Step 3 (15pt). Match the SIFT keypoints of the original image and the rotated imag using the knnMatch function in the cv2.BFMatcher class. Discard bad matches using the ratio test proposed by D.Lowe in the SIFT paper. Use 0.1 as the ratio in this homework. Note that this is for display purpose only. Draw the filtered good keypoint matches on the image and display it. The image you draw should have two images side by side with matching lines across them.
Step 4 (10pt). Use the RANSAC algorithm to find the affine transformation from the rotated image to the original image. You are not required to implement the RANSAC algorithm yourself, instead you could use the cv2.findHomography function (set the 3rd parameter method to cv2.RANSAC ) to compute the transformation matrix. Transform the rotated image back using this matrix and the cv2.warpPerspective function. Display the recovered image.
Bonus (5pt). You might have noticed that the rotated image from step 2 is cropped. Rotate the image without any cropping and you will be awarded an extra 5 points.
Hints: In case of too many matches in the output image, use the ratio of 0.1 to filter matches
Problem 1: Match transformed images using SIFT features
{40 points + bonus 5} You will transform a given image, and match it back to the original image using SIFT keypoints.
Step 1 (5pt). Use the function from SIFT class to detect keypoints from the given image. Plot the image with keypoints scale and orientation overlaid.
Step 2 (10pt). Rotate your image clockwise by 60 degrees with the cv2.warpAffine function. Extract SIFT keypoints for this rotated image and plot the rotated picture with keypoints scale and orientation overlaid just as in step 1.
Step 3 (15pt). Match the SIFT keypoints of the original image and the rotated imag using the knnMatch function in the cv2.BFMatcher class. Discard bad matches using the ratio test proposed by D.Lowe in the SIFT paper. Use 0.1 as the ratio in this homework. Note that this is for display purpose only. Draw the filtered good keypoint matches on the image and display it. The image you draw should have two images side by side with matching lines across them.
Step 4 (10pt). Use the RANSAC algorithm to find the affine transformation from the rotated image to the original image. You are not required to implement the RANSAC algorithm yourself, instead you could use the cv2.findHomography function (set the 3rd parameter method to cv2.RANSAC ) to compute the transformation matrix. Transform the rotated image back using this matrix and the cv2.warpPerspective function. Display the recovered image.
Bonus (5pt). You might have noticed that the rotated image from step 2 is cropped. Rotate the image without any cropping and you will be awarded an extra 5 points.
Hints: In case of too many matches in the output image, use the ratio of 0.1 to filter matches
Problem 2: Scene stitching with SIFT features
{30 points + 15 bonus} You will match and align between different views of a scene with SIFT features.
Use cv2.copyMakeBorder function to pad the center image with zeros into a larger size. Hint: the final output image should be of size 1608 × 1312. Extract SIFT features for all images and go through the same procedures as you did in problem 1. Your goal is to find the affine transformation between the two images and then align one of your images to the other using cv2.warpPerspective . Use the cv2.addWeighted function to blend the aligned images and show the stitched result. Examples can be found at http://docs.opencv.org/trunk/d0/d86/tutorial_py_image_arithmetics.html
(http://docs.opencv.org/trunk/d0/d86/tutorial_py_image_arithmetics.html). Use parameters 0.5 and 0.5 for alpha blending.
Step 1 (15pt). Compute the transformation from the right image to the center image. Warp the right image with the computed transformation. Stitch the center and right images with alpha blending. Display the SIFT feature matching between the center and right images like you did in problem 1. Display the stitched result (center and right image).
Step 2 (15pt) Compute the transformation from the left image to the stitched image from step 1. Warp the left image with the computed transformation. Stich the left and result images from step 1 with alpha blending. Display the SIFT feature matching between the result image from step 1 and the left image like what you did in problem 1. Display the final stitched result (all three images).
Bonus (15pt). Instead of using cv2.addWeighted to do the blending, implement Laplacian Pyramids to blend the two aligned images. Tutorials can be found at http://opencv-pythontutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_pyramids/py_pyramids.html (http://opencvpython-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_pyramids/py_pyramids.html). Display the stitched result (center and right image) and the final stitched result (all three images) with laplacian blending instead of alpha blending.
Note that for the resultant stitched image, some might have different intensity in the overlapping and other regions, namely the overlapping region looks brighter or darker than others. To get full credit, the final image should have uniform illumination.
Hints: You need to find the warping matrix between images with the same mechanism from problem 1. You will need as many reliable matches as possible to find a good homography so DO NOT use 0.1 here. A suggested value would be 0.75 in this case.
When you warp the image with cv2.warpPerspective, an important trick is to pass in the correct parameters so that the warped image has the same size with the padded_center image. Once you have two images with the same size, find the overlapping part and do the blending.
There are 3 problems in this homework with a total of 100 points. Two bonus questions with extra 5 and 15 points are provided under problem 1 and 2 respectively. The maximum points you may earn from this homework is 100 + 20 = 120 points. Be sure to read Submission Guidelines below. They are important.
Using SIFT in OpenCV 3.x.x in Local Machine
Feature descriptors like SIFT and SURF are no longer included in OpenCV since version 3. This section provides instructions on how to use SIFT for those who use OpenCV 3.x.x. If you are using OpenCV 2.x.x then you are all set, please skip this section. Read this if you are curious about why SIFT is removed https://www.pyimagesearch.com/2015/07/16/where-did-sift-and-surf-go-in-opencv-3/ (https://www.pyimagesearch.com/2015/07/16/where-did-sift-and-surf-go-in-opencv-3/).
We strongly recommend you to use SIFT methods in Colab for this homework, the details will be described in the next section.
However, if you want to use SIFT in your local machine, one simple way to use the OpenCV in-built function SIFT is to switch back to version 2.x.x, but if you want to keep using OpenCV 3.x.x, do the following:
1. uninstall your original OpenCV package
2. install opencv-contrib-python using pip (pip is a Python tool for installing packages written in Python), please find detailed instructions at https://pypi.python.org/pypi/opencv-contrib-python
(https://pypi.python.org/pypi/opencv-contrib-python)
After you have your OpenCV set up, you should be able to use cv2.xfeatures2d.SIFT_create() to create a
SIFT object, whose functions are listed at http://docs.opencv.org/3.0beta/modules/xfeatures2d/doc/nonfree_features.html (http://docs.opencv.org/3.0beta/modules/xfeatures2d/doc/nonfree_features.html)
Using SIFT in OpenCV 3.x.x in Colab (RECOMMENDED)
The default version of OpenCV in Colab is 3.4.3. If we use SIFT method directly, typically we will get this error message:
error: OpenCV(3.4.3) /io/opencv_contrib/modules/xfeatures2d/src/sift.cpp:1207: erro r: (-213:The function/feature is not implemented) This algorithm is patented and is excluded in this configuration; Set OPENCV_ENABLE_NONFREE CMake option and rebuild the library in function 'create'
One simple way to use the OpenCV in-built function SIFT in Colab is to switch the version to the one from 'contrib'. Below is an example of switching OpenCV version:
1. Run the following command in one section in Colab, which has already been included in this assignment:
pip install opencv-contrib-python==3.4.2.16
2. Restart runtime by
Runtime - Restart Runtime
Then you should be able to use use cv2.xfeatures2d.SIFT_create() to create a SIFT object, whose functions are listed at http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html (http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html)
Some Resources
In addition to the tutorial document, the following resources can definitely help you in this homework:
http://opencv-pythontutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html (http://opencvpython-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html) http://docs.opencv.org/3.1.0/da/df5/tutorial_py_sift_intro.html (http://docs.opencv.org/3.1.0/da/df5/tutorial_py_sift_intro.html) http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html?highlight=sift#cv2.SIFT (http://docs.opencv.org/3.0-beta/modules/xfeatures2d/doc/nonfree_features.html?highlight=sift#cv2.SIFT) http://docs.opencv.org/3.0beta/doc/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html
(http://docs.opencv.org/3.0beta/doc/py_tutorials/py_imgproc/py_geometric_transformations/py_geometric_transformations.html)
Problem 1: Match transformed images using SIFT features
{40 points + bonus 5} You will transform a given image, and match it back to the original image using SIFT keypoints.
Step 1 (5pt). Use the function from SIFT class to detect keypoints from the given image. Plot the image with keypoints scale and orientation overlaid.
Step 2 (10pt). Rotate your image clockwise by 60 degrees with the cv2.warpAffine function. Extract SIFT keypoints for this rotated image and plot the rotated picture with keypoints scale and orientation overlaid just as in step 1.
Step 3 (15pt). Match the SIFT keypoints of the original image and the rotated imag using the knnMatch function in the cv2.BFMatcher class. Discard bad matches using the ratio test proposed by D.Lowe in the SIFT paper. Use 0.1 as the ratio in this homework. Note that this is for display purpose only. Draw the filtered good keypoint matches on the image and display it. The image you draw should have two images side by side with matching lines across them.
Step 4 (10pt). Use the RANSAC algorithm to find the affine transformation from the rotated image to the original image. You are not required to implement the RANSAC algorithm yourself, instead you could use the cv2.findHomography function (set the 3rd parameter method to cv2.RANSAC ) to compute the transformation matrix. Transform the rotated image back using this matrix and the cv2.warpPerspective function. Display the recovered image.
Bonus (5pt). You might have noticed that the rotated image from step 2 is cropped. Rotate the image without any cropping and you will be awarded an extra 5 points.
Hints: In case of too many matches in the output image, use the ratio of 0.1 to filter matches
Problem 1: Match transformed images using SIFT features
{40 points + bonus 5} You will transform a given image, and match it back to the original image using SIFT keypoints.
Step 1 (5pt). Use the function from SIFT class to detect keypoints from the given image. Plot the image with keypoints scale and orientation overlaid.
Step 2 (10pt). Rotate your image clockwise by 60 degrees with the cv2.warpAffine function. Extract SIFT keypoints for this rotated image and plot the rotated picture with keypoints scale and orientation overlaid just as in step 1.
Step 3 (15pt). Match the SIFT keypoints of the original image and the rotated imag using the knnMatch function in the cv2.BFMatcher class. Discard bad matches using the ratio test proposed by D.Lowe in the SIFT paper. Use 0.1 as the ratio in this homework. Note that this is for display purpose only. Draw the filtered good keypoint matches on the image and display it. The image you draw should have two images side by side with matching lines across them.
Step 4 (10pt). Use the RANSAC algorithm to find the affine transformation from the rotated image to the original image. You are not required to implement the RANSAC algorithm yourself, instead you could use the cv2.findHomography function (set the 3rd parameter method to cv2.RANSAC ) to compute the transformation matrix. Transform the rotated image back using this matrix and the cv2.warpPerspective function. Display the recovered image.
Bonus (5pt). You might have noticed that the rotated image from step 2 is cropped. Rotate the image without any cropping and you will be awarded an extra 5 points.
Hints: In case of too many matches in the output image, use the ratio of 0.1 to filter matches
Problem 2: Scene stitching with SIFT features
{30 points + 15 bonus} You will match and align between different views of a scene with SIFT features.
Use cv2.copyMakeBorder function to pad the center image with zeros into a larger size. Hint: the final output image should be of size 1608 × 1312. Extract SIFT features for all images and go through the same procedures as you did in problem 1. Your goal is to find the affine transformation between the two images and then align one of your images to the other using cv2.warpPerspective . Use the cv2.addWeighted function to blend the aligned images and show the stitched result. Examples can be found at http://docs.opencv.org/trunk/d0/d86/tutorial_py_image_arithmetics.html
(http://docs.opencv.org/trunk/d0/d86/tutorial_py_image_arithmetics.html). Use parameters 0.5 and 0.5 for alpha blending.
Step 1 (15pt). Compute the transformation from the right image to the center image. Warp the right image with the computed transformation. Stitch the center and right images with alpha blending. Display the SIFT feature matching between the center and right images like you did in problem 1. Display the stitched result (center and right image).
Step 2 (15pt) Compute the transformation from the left image to the stitched image from step 1. Warp the left image with the computed transformation. Stich the left and result images from step 1 with alpha blending. Display the SIFT feature matching between the result image from step 1 and the left image like what you did in problem 1. Display the final stitched result (all three images).
Bonus (15pt). Instead of using cv2.addWeighted to do the blending, implement Laplacian Pyramids to blend the two aligned images. Tutorials can be found at http://opencv-pythontutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_pyramids/py_pyramids.html (http://opencvpython-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_pyramids/py_pyramids.html). Display the stitched result (center and right image) and the final stitched result (all three images) with laplacian blending instead of alpha blending.
Note that for the resultant stitched image, some might have different intensity in the overlapping and other regions, namely the overlapping region looks brighter or darker than others. To get full credit, the final image should have uniform illumination.
Hints: You need to find the warping matrix between images with the same mechanism from problem 1. You will need as many reliable matches as possible to find a good homography so DO NOT use 0.1 here. A suggested value would be 0.75 in this case.
When you warp the image with cv2.warpPerspective, an important trick is to pass in the correct parameters so that the warped image has the same size with the padded_center image. Once you have two images with the same size, find the overlapping part and do the blending.
1 file (2.7MB)
