CMU15463-Assignment 3 Solved

The purpose of this assignment is to explore gradient-domain processing, in the context of two different applications: image blending [3] and flash/no-flash photography [1]. As we discussed in class, Poisson image blending is a technique that allows seamlessly blending an object or texture from a source image into a target image. Flash/no-flash photography is a technique that allows to combine the neutral and pleasant illumination effects of photographs where there is only ambient light, with the low-noise characteristics of photographs captured with a flash light. While the assignment and the class slides describe most of the steps you need to perform, it is highly recommended that you read the two associated papers. There is a “Hints and Information” section at the end of this document that is likely to help.

1.     Poisson Blending 
As we discussed in class, the main insight behind Poisson blending is that people often care much more about the gradient of an image than the overall intensity. So, we can try to do blending by finding values for the area to be blended that maximally preserve the gradient of the source region without changing any of the background pixels of the target image. Note that we are making a deliberate decision here to ignore the overall intensity! So a green hat could turn red, but it will still look like a hat.

For an example of this, consider the picture of the bear and swimmers being pasted into a pool of water, shown in Figure 1. Let’s ignore the bear for a moment (it’s a non-threatening bear) and consider the swimmers. We will denote by s the source image, that is, the original image the swimmers were cut out of. That image isn’t even shown, because we are only interested in the cut-out of the swimmers, which we denote as the region S. S includes the swimmers and a bit of light blue background. You can clearly see the region S in the left image, as the cutout blends very poorly into the pool of water. The pool of water, before things were rudely pasted into it, is the target image t.

How do we blend in the swimmers? We construct a new image v whose gradients inside the region S are similar to the gradients of the cutout we’re trying to paste in (swimmers and a little bit of background). Outside S, v will match the pool. We won’t even bother computing the gradients of the pool outside S; we’ll just copy those pixels directly.

We can formulate the above as a least squares problem, as follows:

                                                    v = argmin X ((vi − vj) − (si − sj))2 + X ((vi − tj) − (si − sj))2 .                                               (1)

v

                                                i∈S, j∈Ni∩S                                                                                      i∈S, j∈Ni∩¬S

In the first summation of Equation (1), we set the gradients of v inside S. We loop over all the pixels inside the region S, and require that the new image v have the same gradients as the swimmers. The summation is over every pixel i in S; j is the 4 neighbors of i (left, right, up, and down), giving us both horizontal and vertical gradients. The second summation takes care of the boundary around S. In this case, i is inside S, but j is outside. In this case we are not solving for a pixel intensity vj, since j is not inside S. So we just insert the intensity value tj from the target image, which we know since we are not modifying that part of the image. The gradients will not end up matching exactly: the least squares solver will take any hard edges of the cutout at the boundary and smooth them by spreading the error over the gradients inside S.

Toy Problem . The implementation for gradient domain processing is not complicated, but it is easy to make mistakes, so you will start with a toy example using the image toy  problem.png in the ./data directory of the homework ZIP archive. In this example, you will compute the x and y gradients of an image s, then use these the gradients, plus one pixel intensity, to reconstruct an image v. If all is done correctly, the reconstructed image should match the original one.

In particular, denote the intensity of the source image at (x,y) as s(x,y), and the values of the image to solve for as v (x,y). For each pixel, then, we have two objectives:

 

Figure 1: Pool example for Poisson blending.

1. Minimize the difference between the x-gradients of v and the x-gradients of s, that is,

((v (x + 1,y) − v (x,y)) − (s(x + 1,y) − s(x,y)))2 .

2. Minimize the difference between the y-gradients of v and the y-gradients of s, that is,
(2)
((v (x,y + 1) − v (x,y)) − (s(x,y + 1) − s(x,y)))2 .
(3)
Note that the above objectives could be solved while adding any constant value to v, so you will need to add one more objective:

3. Minimize the difference between the colors of the top left corners of the two images,

                                                                                             (v (1,1) − s(1,1))2 .                                                                           (4)

Combine all of the above objectives into a single least-squares problem, expressed in matrix form:

                                                                                                (Av − b)2 ,                                                                                         (5)

where A is a sparse matrix, v are the variables to be solved, and b is a known vector. Then, solve this problem using one of Matlab’s solvers and compare the result to the original image. (See help for the \ operator)

Poisson blending. It is now time to implement Poisson blending. For this, you will use the images hiking.png, penguin.png, and penguin-chick.png included in the ./data directory of the homework ZIP archive. You will insert each of the two penguin images on top of the snow in the hiking image. You can select where exactly the insertion region will be. Given that the images are RGB, you should process each color channel separately. The file starter.m, provided in the ./src directory of the homework ZIP archive, contains some starter code for this part. Its use is optional. We can split implementation into three steps.

1.    Select the boundaries of a region in the source image and specify a location in the target image where it should be blended. Then, transform (e.g., translate) the source image so that indices of pixels in the source and target regions match. For help with this, you can use the functions getMask.m and alignSource.m provided in the ./src directory of the homework ZIP archive. You may want to augment the code to allow rotation or resizing into the target region. You can be a bit sloppy about selecting the source region: The penguin selection can be done very crudely, with lots of room around them—just make sure that you are kind to the penguins and you do not chop off parts of them.

2.    Formulate and solve the blending constraints expressed in Equation (1).

3.    Copy the solved values vi into your target image.

Blending with mixed gradients. Follow the same steps as Poisson blending, but use the gradient in source or target with the larger magnitude as the guide, rather than the source gradient. That is, solve the least squares problem:

v = argmin X ((vi − vj) − dij)2 + X ((vi − tj) − dij)2 ,

v

                                                          i∈S, j∈Ni∩S                                                                      i∈S, j∈Ni∩¬S

where dij is whichever of the source or the target gradients has the largest magnitude,
(6)
(

                                                                                   si − sj                  if |si − sj| > |ti − tj|,

dij = ti − tj, otherwise.
(7)
Your own examples (20 points.) Apply Poisson blending and the mixed-gradients variant to your own examples. Show at least three results for each. Explain any failure cases (e.g., weird colors, blurred boundaries, etc.). We discussed some failure cases in class, so you can try to reproduce those. For mixed gradients, one possible example is blending a picture of writing on a plain background onto another image.

The total number of points you will get from this question will depend on how visually compelling your blending results are.

2.     Bonus: Other types of gradient-domain processing (up to 30 points)
Gradient-domain processing is a general image processing paradigm that can be used for a broad set of applications, including blending as above, but also tone-mapping, colorization, converting to grayscale, edge enhancement, image abstraction and non-photorealistic rendering, and so on.

GradientShop [2] is a paper and associated app that provide a unifying framework for expressing all these tasks as gradient-domain processing algorithms. Read this paper, and implement whichever of the applications described within appeal to you the most (the instructor’s personal favorite is non-photorealistic rendering, but it won’t reflect badly on your grade if you go for something else).

Your implementation does not need to follow the paper exactly. For instance, instead of implementing the “long-edge detector” described in the paper, you can come up with your own detector of continuous edges. We also recommend not trying the video-based applications.

In your submitted report, you should make sure to describe in detail exactly what you did. This includes what task you chose to implement, where you followed and where you deviated from GradientShop, and in the latter case what you did differently. You should also show at least three examples showcasing the effects of your gradient-domain processing pipeline.

3.     Flash/no-flash photography 
Getting the lighting in your scene right is one of the most important considerations one needs to take care of when taking a photograph. This can be particularly challenging when the scene you are trying to photograph has a large dynamic range, including both very dark and very bright objects. Previously, we saw a passive technique for photographing such difficult scenes, using HDR imaging. Here we will investigate an alternative active technique, where we insert new light into the scene using our camera’s flash.

Consider the scene shown in Figure 2(a). The background painting is nicely illuminated from the ambient environment lighting. However, the person in the foreground is very dimly lit. We can attempt to address this using the camera’s flash, which results in the photograph in Figure 2(b). Now the person in the foreground is well-illuminated, but our photograph has a different problem: The sharp lighting of our flash has created a specular highlight on the painting. This is a common problem when using simple point flash lights, e.g., the one on your DLSR or at the back of your cell phone. A photographer with access to some more advanced equipment would use either a pointable flash that they can bounce off of the ceiling, or they would use a diffuse display to make the light be smooth. However, for the purposes of this assignment, we can assume we do not have access to such equipment.

Instead, we can attempt to produce an artifact-free image by fusing the pair of images taken without and with flash. Techniques that use such pairs are often referred to as flas/no-flash photography. There are

 

Figure 2: Combining photographs taken under ambient lighting (left) and flash lighting (middle), to produce a new image that combines the best of both worlds (right).

many different ways to perform the fusion of the pair into a single photograph, and here we will focus on a technique that uses gradient-domain processing, first proposed by Agrawal et al. [1]. As usual, you are strongly encouraged to read through the paper, to find many implementation details and insights about the various parameters. The end result of the fusion is shown in Figure 2(c).

Implement the flash/no-flash fusion . For this implementation, you will use the images museum ambient.png and museum flash.png included in the ./data directory of the homework ZIP archive. Following Agrawal et al. [1], we will denote the first image (ambient light) as a, the second image (flash light) as Φ0, and the fused image as Φ∗. Your fusion algorithm should follow the procedure described in the paper, which consists of the following steps:

1.    Compute the gradient orientation coherency map M using Equation (5) in the paper.

2.    Compute the saturation weight map ws from the flash image Φ0 using Equation (11) in the paper.

3.    Compute a new gradient field for the fused image Φ∗ using Equation (12) in the paper.

4.    Synthesize the fused image Φ∗ by integrating the above integral field.

For the integration, you can use the Poisson solver you developed in Part 1. Experiment with different values for the various parameters involved (e.g., the parameters in the saturation weight map), and show the resulting fused image you obtain.

Capture and fuse your own flash/no-flash pairs . Now it is time to apply what you implemented above to your own pictures. To create results which are clearly better than any single exposure, you should take pictures of a scene where flash affects some parts of the image, while leaving others relatively unaffected. Good examples include dark scenes that have both matte and specular objects. You can look at the paper by Agrawal et al. [1] for inspiration.

4.     Bonus: Reflection removal in flash/no-flash photography
A different potential problem when using flash is that it can create reflections off of glass surfaces. See Figure 6 in Agrawal et al. [1]. Implement the fusion algorithm described in Section 4.1 (“Observation Deck”) of the paper, and then use it on a flash/no-flash pair you capture with your own camera. We suggest using a scene with a window for demonstrating this part, as in the paper.