16-385- Assignment 4: Physics-based vision Solved

Instructions
Integrity and collaboration: Students are encouraged to work in groups but each student must submit their own work. If you work as a group, include the names of your collaborators in your write up. Code should NOT be shared or copied. Please DO NOT use external code unless permitted. Plagiarism is strongly prohibited and will lead to failure of this course.
Start early! Especially if you are not familiar with MATLAB.
Questions: If you have any question, please look at piazza first. Other students may have encountered the same problem, and is solved already. If not, post your question on the discussion board. TAs will respond as soon as possible.
Write-up: Items to be included in the writeup are mentioned in each question, and summarized in the Writeup section. Please note that we DO NOT accept handwritten scans for your write-up in this assignment. Please type your answers to theory questions and discussions for experiments electronically.
Handout: The handout zip file contains 3 items. pdf is the assignment handout. data contains normals for the Stanford bunny images, photometric stereo data, an image of a fruitbowl, and the spectral radiance of bananas. matlab contains a script that you will make use of in this homework.
1             Overview
The purpose of this homework is to go over the topics of radiometry, reflectance, photometric stereo, and color. Given the short nature of this homework, and unlike previous assignments, it consists of a few smaller and independent problems, rather than a sequence of related problems.

1.1           Lambertian albedo      
For Lambertian surfaces, the BRDF is a constant function of the input and output directions. For such a material, we often describe the reflectance in terms of its albedo, which is given the symbol ρ. For a Lambertian surface, the BRDF and albedo are related by fr(vˆi,vˆo) = ρ/π. Using conservation of energy, prove that 0 ≤ ρ ≤ 1.

1.2           Foreshortening  
A small Lambertian source dA is centered at P and emits radiance L. The orientation of this patch is the same as that of a plane containing two points, X1 and X2. The point X1 is the point on this plane that is closest to P, and the distance from P to X1 is D as shown.

Calculate the solid angle subtended by dA at points X1 and X2.
Calculate the irradiance E incident on the plane at points X1 and X2, and calculate the ratio E(X1)/E(X2).
1.3           Simple rendering          
Obtain the file bunny.mat from the data folder of this assignment and load it into Matlab. There is a single variable in this file; the variable N is an h×w ×3 array of surface normals. N(i,j,1), N(i,j,2), and N(i,j,3) are the x, y, and z components of the surface normal at the ijth surface point, as observed by an orthographic camera with view direction (0,0,1).

Use quiver to display the x-y components of the normal field and print the result.
Compute and display the radiance emitted from each point assuming Lambertian reflectance and constant albedo (equal to one), with a distant point light source in direction ˆs = (0, 0, 1), by typing imshow(N(:,:,3)). Explain why this works.
Compute, display and print the emitted radiance for three different light source directions which are rotated i) 45◦ up, ii) 45◦ right, and iii) 75◦ right from the frontal direction ˆs = (0, 0, 1). Can you spot errors in the field of surface normals? What are the illumination effects being ignored in this calculation of scene radiance?
Figure 1: A few bunny renderings.

In your write-up: Please include figures for the normal visualizations and the three renderings of the bunny, as well as answers to the associated questions.

1.4           Photometric stereo     
The data folder of this assignment contains a set of seven photometric stereo images along with a MAT file containing the light source vectors. You can load the images into Matlab and convert them to double, grayscale format.

Estimate the surface normal and grayscale albedo for each pixel. Submit your Matlab code as well as the results (using imshow for the albedo values and quiver for the surface normals.)
Note the poor estimates of the albedo (and surface normal) in the area surrounding the nostrils. What is the source of this error? Describe one method for finding a better estimate of this information from these seven images.
Use the recovered surface information to predict what the person would look like (in grayscale) if illuminated from direction ˆs = (0.58,−0.58,−0.58) and from direction ˆs = (−0.58,−0.58,−0.58). Submit your results and your code.
The function integrate m in the matlab folder can be used to recover a surface z(x,y) from your surface normals nˆ(x,y), and the surface can be displayed in Matlab. Display and submit two views of the recovered shape.
Figure 2: Two views of the recovered face shape.

In your write-up: Include figures for the recovered normals and albedo, the re-rendered face images, and the final reconstructed surface, as well as answers to all associated questions.

1.5           Dichromatic reflectance      
A reasonable reflectance model for dielectric (non-conducting) surfaces is the so-called dichromatic model, according to which the spectral BRDF is written as a linear combination of a Lambertian diffuse component fd and a wavelength-independent specular component fs:

f(λ,ωˆi,ωˆo) = fd(λ) + fs(ωˆi,ωˆo).

When we image a scene consisting of such surfaces under a directional light source with spectral power distribution I(λ) and direction ˆl = (lx,ly,lz), the RGB values

(CR(~u),CG(~u),CB(~u)) recorded at pixel ~u = (u,v) can be expressed as

where h·,·i is the inner product operation, ˆn(~u) is the surface normal at the scene point imaged by pixel ~u, and gs(~u) is a function that depends non-linearly on ˆn(~u) (as well as view and lighting directions) through the specular component of the BRDF.

Assuming that the spectral sensitivities of a camera’s three filters are (cR(λ),cG(λ),cB(λ)) and that the BRDF at the surface point imaged at pixel ~u is f(λ,ωˆi(~u),ωˆo(~u)) = fd(λ,~u) + fs(ωˆi(~u),ωˆo(~u)), write expressions for the elements of the diffuse color vectors d~(~u) and the source color vector ~s.
Suppose you are given two unit-length three-vectors ˆr1 and ˆr2 that are orthogonal to
~s. Show that the two-channel image[1] given by the per-pixel inner products J~(~u) =

does not depend on the specular components of the BRDFs, fs(ωˆi(~u),ωˆo(~u)).
depends linearly on the surface normals, ˆn(~u).
Show that the two properties from part (b) are also satisfied by the single-channel
(grayscale) image.

Write a function imout=makeLambertian(im,s) that takes an RGB image im and a source color vector s and computes the grayscale image J(~u) from part (c). Run this function on the image png from the data folder using s=[0.6257 0.5678 0.5349] and submit the results. Explain in three sentences or less why imout might be more useful than im to a computer vision system. Note that a simple way to numerically obtain a set of N − 1 unit-length vectors that are orthogonal to N-vector is to use the null command in Matlab.
Figure 3: Left: Original color image. Right: Image converted to grayscale with specularities removed.

1.6           Color metamers (Extra Credit)     
Two distinct spectral distributions are metameric if they induce the same responses in the cones of a human retina. Said another way, metamers are distinct spectral distributions that map to the same tristimulus vector (in CIE XYZ or any other linear color space). If you own a grocery store that sells bananas, it is in your interest to choose your light source so that overripe bananas look that same as those in their prime. That is, you want the light reflected from ripe and overripe bananas to be metameric.

Let f~ and ~g be N×1 vectors that come from discretizing the spectral reflectance functions f(λ) and g(λ) of ripe and overripe bananas, respectively. (For example, if we sampled from 400nm to 700nm in increments of 10nm, we’d have N = 31.) Let R be the 3 × N matrix whose rows are obtained by discretizing the CIE XYZ color matching functions.

Show that the tristimulus vector ) for a ripe banana f(λ) under illuminant spectrum l(λ) can be written aswhere Lf is a 3 × N matrix and
~l is the discretization of l(λ).

Given f~ and ~g, a good way to choose ~l is to minimize the distance, in a least-squares sense, of the two resulting tristimulus vectors. Write an expression for the Euclidean distance between the two tristimulus vectors in terms of Lf, its counterpart Lg, and
Write a Matlab function T=metamericLight(f,g) that finds the temperature T in the closed interval T ∈ [2500K, 10,000K], with precision ±50K, of the blackbody radiator that minimizes the distance derived in part (b). The spectral power distribution for a blackbody radiator (see FP Sect. 6.1.2) with temperature T is
where a = 1.4388 × 10−2(m · K).

The file mat from the data folder contains two spectral reflectance functions, ripe and overripe, which are sampled in 10nm increments from 400nm to 700nm and correspond roughly to the data from Day 5 and Day 7 of Fig. 4. Use your function to find the optimal metamer-inducing blackbody temperature for these materials, and plot the distance as a function of temperature, and the spectral power distribution of the best illuminant. You will need to use the file ciexyz64 1.csv from the data folder, which contains tabulated data for the CIE XYZ color matching functions. In Matlab, you can read data from a CSV file using the csvread command. Remember to normalize the spectral power distributions of blackbody radiators in a way that is meaningful for the minimization criterion of part (b).
Figure 4: Relative spectral reflectance curves for different colors of bananas during ripening. Day five is the first day of shelf life. (Taken without permission

from Morita et al., “ Evaluation of Change in Quality of Ripening Bananas Using

Light Reflectance Technique”, in Memoirs of the Faculty of Agriculture,

Kagoshima University, 1992.)

[1] This image can have pixels with negative values, and these negative values are just as useful as the positive ones.

1             Overview
The purpose of this homework is to go over the topics of radiometry, reflectance, photometric stereo, and color. Given the short nature of this homework, and unlike previous assignments, it consists of a few smaller and independent problems, rather than a sequence of related problems.

1.1           Lambertian albedo      
For Lambertian surfaces, the BRDF is a constant function of the input and output directions. For such a material, we often describe the reflectance in terms of its albedo, which is given the symbol ρ. For a Lambertian surface, the BRDF and albedo are related by fr(vˆi,vˆo) = ρ/π. Using conservation of energy, prove that 0 ≤ ρ ≤ 1.

1.2           Foreshortening  
A small Lambertian source dA is centered at P and emits radiance L. The orientation of this patch is the same as that of a plane containing two points, X1 and X2. The point X1 is the point on this plane that is closest to P, and the distance from P to X1 is D as shown.

Calculate the solid angle subtended by dA at points X1 and X2.
Calculate the irradiance E incident on the plane at points X1 and X2, and calculate the ratio E(X1)/E(X2).
1.3           Simple rendering          
Obtain the file bunny.mat from the data folder of this assignment and load it into Matlab. There is a single variable in this file; the variable N is an h×w ×3 array of surface normals. N(i,j,1), N(i,j,2), and N(i,j,3) are the x, y, and z components of the surface normal at the ijth surface point, as observed by an orthographic camera with view direction (0,0,1).

Use quiver to display the x-y components of the normal field and print the result.
Compute and display the radiance emitted from each point assuming Lambertian reflectance and constant albedo (equal to one), with a distant point light source in direction ˆs = (0, 0, 1), by typing imshow(N(:,:,3)). Explain why this works.
Compute, display and print the emitted radiance for three different light source directions which are rotated i) 45◦ up, ii) 45◦ right, and iii) 75◦ right from the frontal direction ˆs = (0, 0, 1). Can you spot errors in the field of surface normals? What are the illumination effects being ignored in this calculation of scene radiance?
Figure 1: A few bunny renderings.

In your write-up: Please include figures for the normal visualizations and the three renderings of the bunny, as well as answers to the associated questions.

1.4           Photometric stereo     
The data folder of this assignment contains a set of seven photometric stereo images along with a MAT file containing the light source vectors. You can load the images into Matlab and convert them to double, grayscale format.

Estimate the surface normal and grayscale albedo for each pixel. Submit your Matlab code as well as the results (using imshow for the albedo values and quiver for the surface normals.)
Note the poor estimates of the albedo (and surface normal) in the area surrounding the nostrils. What is the source of this error? Describe one method for finding a better estimate of this information from these seven images.
Use the recovered surface information to predict what the person would look like (in grayscale) if illuminated from direction ˆs = (0.58,−0.58,−0.58) and from direction ˆs = (−0.58,−0.58,−0.58). Submit your results and your code.
The function integrate m in the matlab folder can be used to recover a surface z(x,y) from your surface normals nˆ(x,y), and the surface can be displayed in Matlab. Display and submit two views of the recovered shape.
Figure 2: Two views of the recovered face shape.

In your write-up: Include figures for the recovered normals and albedo, the re-rendered face images, and the final reconstructed surface, as well as answers to all associated questions.

1.5           Dichromatic reflectance       
A reasonable reflectance model for dielectric (non-conducting) surfaces is the so-called dichromatic model, according to which the spectral BRDF is written as a linear combination of a Lambertian diffuse component fd and a wavelength-independent specular component fs:

f(λ,ωˆi,ωˆo) = fd(λ) + fs(ωˆi,ωˆo).

When we image a scene consisting of such surfaces under a directional light source with spectral power distribution I(λ) and direction ˆl = (lx,ly,lz), the RGB values

(CR(~u),CG(~u),CB(~u)) recorded at pixel ~u = (u,v) can be expressed as

where h·,·i is the inner product operation, ˆn(~u) is the surface normal at the scene point imaged by pixel ~u, and gs(~u) is a function that depends non-linearly on ˆn(~u) (as well as view and lighting directions) through the specular component of the BRDF.

Assuming that the spectral sensitivities of a camera’s three filters are (cR(λ),cG(λ),cB(λ)) and that the BRDF at the surface point imaged at pixel ~u is f(λ,ωˆi(~u),ωˆo(~u)) = fd(λ,~u) + fs(ωˆi(~u),ωˆo(~u)), write expressions for the elements of the diffuse color vectors d~(~u) and the source color vector ~s.
Suppose you are given two unit-length three-vectors ˆr1 and ˆr2 that are orthogonal to
~s. Show that the two-channel image[1] given by the per-pixel inner products J~(~u) =

does not depend on the specular components of the BRDFs, fs(ωˆi(~u),ωˆo(~u)).
depends linearly on the surface normals, ˆn(~u).
Show that the two properties from part (b) are also satisfied by the single-channel
(grayscale) image.

Write a function imout=makeLambertian(im,s) that takes an RGB image im and a source color vector s and computes the grayscale image J(~u) from part (c). Run this function on the image png from the data folder using s=[0.6257 0.5678 0.5349] and submit the results. Explain in three sentences or less why imout might be more useful than im to a computer vision system. Note that a simple way to numerically obtain a set of N − 1 unit-length vectors that are orthogonal to N-vector is to use the null command in Matlab.
Figure 3: Left: Original color image. Right: Image converted to grayscale with specularities removed.

1.6           Color metamers (Extra Credit)    
Two distinct spectral distributions are metameric if they induce the same responses in the cones of a human retina. Said another way, metamers are distinct spectral distributions that map to the same tristimulus vector (in CIE XYZ or any other linear color space). If you own a grocery store that sells bananas, it is in your interest to choose your light source so that overripe bananas look that same as those in their prime. That is, you want the light reflected from ripe and overripe bananas to be metameric.

Let f~ and ~g be N×1 vectors that come from discretizing the spectral reflectance functions f(λ) and g(λ) of ripe and overripe bananas, respectively. (For example, if we sampled from 400nm to 700nm in increments of 10nm, we’d have N = 31.) Let R be the 3 × N matrix whose rows are obtained by discretizing the CIE XYZ color matching functions.

Show that the tristimulus vector ) for a ripe banana f(λ) under illuminant spectrum l(λ) can be written aswhere Lf is a 3 × N matrix and
~l is the discretization of l(λ).

Given f~ and ~g, a good way to choose ~l is to minimize the distance, in a least-squares sense, of the two resulting tristimulus vectors. Write an expression for the Euclidean distance between the two tristimulus vectors in terms of Lf, its counterpart Lg, and
Write a Matlab function T=metamericLight(f,g) that finds the temperature T in the closed interval T ∈ [2500K, 10,000K], with precision ±50K, of the blackbody radiator that minimizes the distance derived in part (b). The spectral power distribution for a blackbody radiator (see FP Sect. 6.1.2) with temperature T is
where a = 1.4388 × 10−2(m · K).

The file mat from the data folder contains two spectral reflectance functions, ripe and overripe, which are sampled in 10nm increments from 400nm to 700nm and correspond roughly to the data from Day 5 and Day 7 of Fig. 4. Use your function to find the optimal metamer-inducing blackbody temperature for these materials, and plot the distance as a function of temperature, and the spectral power distribution of the best illuminant. You will need to use the file ciexyz64 1.csv from the data folder, which contains tabulated data for the CIE XYZ color matching functions. In Matlab, you can read data from a CSV file using the csvread command. Remember to normalize the spectral power distributions of blackbody radiators in a way that is meaningful for the minimization criterion of part (b).
Figure 4: Relative spectral reflectance curves for different colors of bananas during ripening. Day five is the first day of shelf life. (Taken without permission

[1] This image can have pixels with negative values, and these negative values are just as useful as the positive ones.