Starting from:
$30
EE569-Homework 1 Image Demosaicing and Histogram Manipulation an Image Denoising Solved
Problem 1: Image Demosaicing and Histogram Manipulation
(a) Bilinear Demosaicing
To capture color images, digital camera sensors are usually arranged in form of a color filter array (CFA), called the Bayer array, as shown in Figure 1. Since each sensor at a pixel location only captures one of the three primary colors (R, G, B), the other two colors have to be re-constructed based on their neighbor pixel values to obtain the full color. Demosaicing is the process of translating this Bayer array of primary colors into a color image that contains the R, G, B values at each pixel.
Figure 1: (a) Single CCD sensor covered by a CFA and (b) Bayer pattern [1].
Implement the simplest demosaicing method based on bilinear interpolation. Exemplary demosaicing results are given in Figure 2. With this method, the missing color value at each pixel is approximated by bilinear interpolation using the average of its two or four adjacent pixels of the same color. To give an example, the missing blue and green values at pixel 𝑅3,4 are estimated as:
As for pixel 𝐺3,3, the blue and red values are calculated as:
(a) (b)
Figure 2: (a) The original image and (b) the demosaiced image by bilinear interpolation.
Figure 3: The dog image with the CFA sensor input.
(1) Apply the bilinear demosaicing to the Dog image in Figure 3 and show your results.
(2) Compare your demosaiced image with the ground truth color image Dog_ori. Do you observe any artifacts? If yes, explain the cause of the artifacts and provide your ideas to improve the demosaicing performance.
(b) Malvar-He-Cutler (MHC) Demosaicing (20%)
Malvar et al. [2] proposed an improved linear interpolation demosaicing algorithm. It yields a higher quality demosaicing result by adding a 2nd-order cross-channel correction term to the basic bilinear demosaicing result. Both the bilinear and the MHC demosaicing results of the Fruit_Shop image are shown in Figure 4.
Figure 4: Demosacing results of Fruit_Shop image: the CFA input (left), the bilinear demosaicing result (middle) and the MHC demosaicing result (right).
The MHC algorithm is stated below.
To estimate a green component at a red pixel location, we have
where 𝐺̂𝑏𝑙 is the bilinear interpolation result and the 2nd term is a correction term. For the 2nd term, 𝛼 is a weight factor, and Δ𝑅 is the discrete 5-point Laplacian of the red channel:
To estimate a red component at a green pixel location, we have
where 𝛥𝐺 is a discrete 9-point Laplacian of the green channel.
To estimate a red component at a blue pixel location,
where 𝛥𝐵 is a discrete 9-point Laplacian of the blue channel. The weights control how much correction is applied, and their default values are:
The above formulas can be generalized to missing color components at each sensor location. Consequently, the MHC demosaicing can be implemented by convolution with a set of linear filters. There are eight different filters for interpolating the different color components at different locations as illustrated in Figure 5 [2].
Figure 5: Filter coefficients.
(1) Implement the MHC linear demosaicing algorithm and apply it to the Dog image in Figure 3. Show your results.
(2) Compare the MHC and the bilinear demosaicing results and explain the performance differences between these two algorithms in your own words.
(c) Histogram Manipulation
Implement two histogram equalization techniques:
• Method A: the transfer-function-based histogram equalization method,
• Method B: the cumulative-probability-based histogram equalization method to enhance the contrast of the Toy image in Figure 6 below.
(1) Plot the histograms of the red, green and blue channels of the original image. The figure should have the intensity value as the x-axis and the number of pixels as the y-axis.
(2) Apply Method A to the original image and show the enhanced image. Plot the transfer function for each channel.
(3) Apply Method B to the original image and show the enhanced image. Plot the cumulative histogram for each channel.
(4) Discuss your observations on these two enhancement results. Do you have any idea to improve the current result?
Note that MATLAB users CANNOT use functions from the Image Processing Toolbox except displaying function like imshow().
Figure 6: Toy image
Problem 2: Image Denoising
In this problem, you will implement a set of denoising algorithms to improve image quality. You can use the PSNR (peak-signal-to-noise-ratio) quality metric to assess the performance of your denoising algorithm. The PSNR value for R, G, B channels can be, respectively, calculated as follows:
Remove noise in the image in Figure 7(b), compare it with the original image in Figure 7(a), and answer the following questions:
(a) Basic denoising methods
(1) What is the type of embedded noise in Figure 7(b)?
(2) Apply a linear filter of size N by N to the noisy image. Compare the performance of two choices of the filter parameters – the uniform weight function and the Gaussian weight function.
Figure 7: The original and noisy corn images.
(b) Bilateral Filtering
In most low-pass linear filters, we often see degradation of edges. However, using some nonlinear filters, we can preserve the edges. Bilateral filters are one such kind of filters. A discrete bilateral filter is given by:
where (𝑘, 𝑙) is the neighboring pixel location within the window centered around (𝑖, 𝑗), 𝐼 is the image with noise, 𝑌 is the filtered image. 𝜎𝑐 and 𝜎𝑠 are the spread parameters.
(1) Implement the bilateral denoising filter and apply it to the noisy image.
(2) Explain the roles of 𝜎𝑐 and 𝜎𝑠. Discuss the change in filter's performance with respect to the values of 𝜎𝑐 and 𝜎𝑠.
(3) Does this filter perform better than linear filters you implemented in Problem 3(a)? Justify your answer in words.
(c) Non-Local Means (NLM) Filtering
The non-local mean filter utilizes the pixel value from a larger region rather the mean of a local window centered around the target pixel. A discrete non-local mean filter with Gaussian weighting function is as follows:
where 𝐼, 𝑌 are the noisy and filtered images respectively, 𝑁𝑥,𝑦 is the window centered around location (𝑥, 𝑦), and ℎ is the filtering parameter, 𝑁′ ≤ 𝑁 and 𝑀′ < 𝑀 denote the window size of your choice.
The Gaussian weighted Euclidian distance between window 𝐼(𝑁𝑖,𝑗)and 𝐼(𝑁𝑘,𝑙) is defined as:
2 2
‖𝐼(𝑁𝑖,𝑗) − 𝐼(𝑁𝑘,𝑙)‖2,𝑎 = ∑ 𝐺𝑎(𝑛1, 𝑛2)(𝐼(𝑖 − 𝑛1, 𝑗 − 𝑛2) − 𝐼(𝑘 − 𝑛1, 𝑙 − 𝑛2))
𝑛1,𝑛2∈ℵ
where ℵ denotes the local neighborhood centered at the origin, 𝑛1, 𝑛2 ∈ ℵ denotes the relative position in the neighborhood window. 𝑎 0 is the standard deviation of the Gaussian kernel.
(1) Implement the NLM filter and apply it to the noisy image. Try several filter parameters and discuss their effect on filtering process. Clearly state your final choice of parameters in your report.
(2) Compare the performance of NLM with filters used in Problem 3(a) and Problem 3(b).
(d) Block matching and 3-D (BM3D) transform filter
In this part, you will get familiar with another state-of-the-art denoising algorithm proposed in [3].
(1) Please explain the BM3D algorithm in your own words.
(2) Implement the BM3D filter (Write your own code or use any available online source code but include the source in your reference) to denoise the noisy image House in Figure 7. Show the final noise-removed image, and compare the PSNR value and visual quality with those in (a).
Note: It is recommended that you use the code provided by the authors on their website [4]. Their code is written in MATLAB.
(e) Mixed noises in color image
Figure 8 (b) is a noisy color image corrupted with mixed types of noises. Please identify noise types in the image and answer the following questions:
(1) What types of noises are there?
(2) Should you perform filtering on individual channels separately for both noise types?
(3) What filters would you like use to remove mixed noise? Can you cascade these filters in any order? Justify your answer.
(a) (b)
Figure 8: (a) the original pepper image (b) the pepper image with mixed noises.
Appendix:
Problem 1: Image Demosaicing and Histogram Manipulation
Dog.raw 600x532
8-bit
gray
Dog_ori.raw 600x532
24-bit
color(RGB)
Toy.raw 560x400
Problem 2: Image Denoising
24-bit
color(RGB)
Corn_gray.raw 320x320
8-bit
gray
Corn_noisy.raw 320x320
8-bit
gray
(a) Bilinear Demosaicing
To capture color images, digital camera sensors are usually arranged in form of a color filter array (CFA), called the Bayer array, as shown in Figure 1. Since each sensor at a pixel location only captures one of the three primary colors (R, G, B), the other two colors have to be re-constructed based on their neighbor pixel values to obtain the full color. Demosaicing is the process of translating this Bayer array of primary colors into a color image that contains the R, G, B values at each pixel.
Figure 1: (a) Single CCD sensor covered by a CFA and (b) Bayer pattern [1].
Implement the simplest demosaicing method based on bilinear interpolation. Exemplary demosaicing results are given in Figure 2. With this method, the missing color value at each pixel is approximated by bilinear interpolation using the average of its two or four adjacent pixels of the same color. To give an example, the missing blue and green values at pixel 𝑅3,4 are estimated as:
As for pixel 𝐺3,3, the blue and red values are calculated as:
(a) (b)
Figure 2: (a) The original image and (b) the demosaiced image by bilinear interpolation.
Figure 3: The dog image with the CFA sensor input.
(1) Apply the bilinear demosaicing to the Dog image in Figure 3 and show your results.
(2) Compare your demosaiced image with the ground truth color image Dog_ori. Do you observe any artifacts? If yes, explain the cause of the artifacts and provide your ideas to improve the demosaicing performance.
(b) Malvar-He-Cutler (MHC) Demosaicing (20%)
Malvar et al. [2] proposed an improved linear interpolation demosaicing algorithm. It yields a higher quality demosaicing result by adding a 2nd-order cross-channel correction term to the basic bilinear demosaicing result. Both the bilinear and the MHC demosaicing results of the Fruit_Shop image are shown in Figure 4.
Figure 4: Demosacing results of Fruit_Shop image: the CFA input (left), the bilinear demosaicing result (middle) and the MHC demosaicing result (right).
The MHC algorithm is stated below.
To estimate a green component at a red pixel location, we have
where 𝐺̂𝑏𝑙 is the bilinear interpolation result and the 2nd term is a correction term. For the 2nd term, 𝛼 is a weight factor, and Δ𝑅 is the discrete 5-point Laplacian of the red channel:
To estimate a red component at a green pixel location, we have
where 𝛥𝐺 is a discrete 9-point Laplacian of the green channel.
To estimate a red component at a blue pixel location,
where 𝛥𝐵 is a discrete 9-point Laplacian of the blue channel. The weights control how much correction is applied, and their default values are:
The above formulas can be generalized to missing color components at each sensor location. Consequently, the MHC demosaicing can be implemented by convolution with a set of linear filters. There are eight different filters for interpolating the different color components at different locations as illustrated in Figure 5 [2].
Figure 5: Filter coefficients.
(1) Implement the MHC linear demosaicing algorithm and apply it to the Dog image in Figure 3. Show your results.
(2) Compare the MHC and the bilinear demosaicing results and explain the performance differences between these two algorithms in your own words.
(c) Histogram Manipulation
Implement two histogram equalization techniques:
• Method A: the transfer-function-based histogram equalization method,
• Method B: the cumulative-probability-based histogram equalization method to enhance the contrast of the Toy image in Figure 6 below.
(1) Plot the histograms of the red, green and blue channels of the original image. The figure should have the intensity value as the x-axis and the number of pixels as the y-axis.
(2) Apply Method A to the original image and show the enhanced image. Plot the transfer function for each channel.
(3) Apply Method B to the original image and show the enhanced image. Plot the cumulative histogram for each channel.
(4) Discuss your observations on these two enhancement results. Do you have any idea to improve the current result?
Note that MATLAB users CANNOT use functions from the Image Processing Toolbox except displaying function like imshow().
Figure 6: Toy image
Problem 2: Image Denoising
In this problem, you will implement a set of denoising algorithms to improve image quality. You can use the PSNR (peak-signal-to-noise-ratio) quality metric to assess the performance of your denoising algorithm. The PSNR value for R, G, B channels can be, respectively, calculated as follows:
Remove noise in the image in Figure 7(b), compare it with the original image in Figure 7(a), and answer the following questions:
(a) Basic denoising methods
(1) What is the type of embedded noise in Figure 7(b)?
(2) Apply a linear filter of size N by N to the noisy image. Compare the performance of two choices of the filter parameters – the uniform weight function and the Gaussian weight function.
Figure 7: The original and noisy corn images.
(b) Bilateral Filtering
In most low-pass linear filters, we often see degradation of edges. However, using some nonlinear filters, we can preserve the edges. Bilateral filters are one such kind of filters. A discrete bilateral filter is given by:
where (𝑘, 𝑙) is the neighboring pixel location within the window centered around (𝑖, 𝑗), 𝐼 is the image with noise, 𝑌 is the filtered image. 𝜎𝑐 and 𝜎𝑠 are the spread parameters.
(1) Implement the bilateral denoising filter and apply it to the noisy image.
(2) Explain the roles of 𝜎𝑐 and 𝜎𝑠. Discuss the change in filter's performance with respect to the values of 𝜎𝑐 and 𝜎𝑠.
(3) Does this filter perform better than linear filters you implemented in Problem 3(a)? Justify your answer in words.
(c) Non-Local Means (NLM) Filtering
The non-local mean filter utilizes the pixel value from a larger region rather the mean of a local window centered around the target pixel. A discrete non-local mean filter with Gaussian weighting function is as follows:
where 𝐼, 𝑌 are the noisy and filtered images respectively, 𝑁𝑥,𝑦 is the window centered around location (𝑥, 𝑦), and ℎ is the filtering parameter, 𝑁′ ≤ 𝑁 and 𝑀′ < 𝑀 denote the window size of your choice.
The Gaussian weighted Euclidian distance between window 𝐼(𝑁𝑖,𝑗)and 𝐼(𝑁𝑘,𝑙) is defined as:
2 2
‖𝐼(𝑁𝑖,𝑗) − 𝐼(𝑁𝑘,𝑙)‖2,𝑎 = ∑ 𝐺𝑎(𝑛1, 𝑛2)(𝐼(𝑖 − 𝑛1, 𝑗 − 𝑛2) − 𝐼(𝑘 − 𝑛1, 𝑙 − 𝑛2))
𝑛1,𝑛2∈ℵ
where ℵ denotes the local neighborhood centered at the origin, 𝑛1, 𝑛2 ∈ ℵ denotes the relative position in the neighborhood window. 𝑎 0 is the standard deviation of the Gaussian kernel.
(1) Implement the NLM filter and apply it to the noisy image. Try several filter parameters and discuss their effect on filtering process. Clearly state your final choice of parameters in your report.
(2) Compare the performance of NLM with filters used in Problem 3(a) and Problem 3(b).
(d) Block matching and 3-D (BM3D) transform filter
In this part, you will get familiar with another state-of-the-art denoising algorithm proposed in [3].
(1) Please explain the BM3D algorithm in your own words.
(2) Implement the BM3D filter (Write your own code or use any available online source code but include the source in your reference) to denoise the noisy image House in Figure 7. Show the final noise-removed image, and compare the PSNR value and visual quality with those in (a).
Note: It is recommended that you use the code provided by the authors on their website [4]. Their code is written in MATLAB.
(e) Mixed noises in color image
Figure 8 (b) is a noisy color image corrupted with mixed types of noises. Please identify noise types in the image and answer the following questions:
(1) What types of noises are there?
(2) Should you perform filtering on individual channels separately for both noise types?
(3) What filters would you like use to remove mixed noise? Can you cascade these filters in any order? Justify your answer.
(a) (b)
Figure 8: (a) the original pepper image (b) the pepper image with mixed noises.
Appendix:
Problem 1: Image Demosaicing and Histogram Manipulation
Dog.raw 600x532
8-bit
gray
Dog_ori.raw 600x532
24-bit
color(RGB)
Toy.raw 560x400
Problem 2: Image Denoising
24-bit
color(RGB)
Corn_gray.raw 320x320
8-bit
gray
Corn_noisy.raw 320x320
8-bit
gray
1 file (4.8MB)
