Starting from:
$30
COMS4771- Homework 0 Solved
1.1 [Probability and Statistics] Let X and Y be discrete random variables, and consider the joint distribution (X,Y ) given by
Y=1
Y=2
Y=3
X=1
0.1
0.2
0.3
X=2
0.2
0.1
0.1
(i) What is the marginal distribution of X?
(ii) What is Pr[Y = 1|X = 2]?
(iii) Let f : x 7→ x2. What is E[f(X)|Y = 1]?
1.2 Fix some θ 0, and consider the function gθ : [0,∞) → R defined as .
(i) Verify that gθ is a probability distribution.
(ii) Let X be a random variable distributed as gθ. What is E[X]?
(iii) What is Variance(X)?
1.3 You live with your cat in an apartment in a relatively safe neighborhood with a crime rate of 5%. Being a cautious person, you have invested in a good quality burglar alarm that is rated to be 99% effective at the time of a break-in. Over time you have observed that your cat likes to play with the alarm causing it to trip 10% of the time. One day while you are at work your neighbor calls you telling that the alarm is ringing. What is the probability that there was a break-in?
2.1 [Linear Algebra] Consider the subspace S spanned by vectors .
(i) What is the dimension of the subspace S.
(ii) Compute the orthogonal linear projection of the point , onto the subspace S.
2.2 Prove that for any m × n real matrix A and any ρ 0, the matrix ATA + ρI is invertible (where I is the n × n identity matrix). (Hint: show that all eigenvalues of ATA + ρI are real and analyze the smallest eigenvalue.)
3.1 [Calculus and optimization] Let A ∈ Rd×D be a real matrix, and b ∈ Rd be a real vector. Define the function
f :RD → R
x 7→ kAx − bk2 + kxk2.
(i) What is ∇f(x)?
(ii) What value of x minimizes f, that is, find argminx f(x)? (Hint: compute the stationary points of f.)
4.1 [Programming practice] Download the Matlab data file hw0data.mat (instructions on Piazza on where to download the file). Write a script that does the following.
Special note for those who are not using Matlab: Python users can use scipy to read in the mat file, R users can use R.matlab package to read in the mat file, Julia users can use JuliaIO/MAT.jl. Octave users should be able to load the file directly.
(i) Load the data in hw0data.mat. It contains one matrix variable is called M.
(ii) Print the dimensions of M.
(iii) Print the 4th row and 5th column entry of M.
(iv) Print the mean value of the 5th column of M.
(v) Compute the histogram of the 4th row of M and show the figure.
(vi) Compute and print the top three eigenvalues of the matrix MTM.
4.2 We will try to understand the geometry of eigenvectors and eigenvalues of a matrix via exper-
imentation. Let L be a 2 × 2 matrix. To understand eigenvectors and
eigenvalues, we will study the action of L on random vectors and relate it to eigenvectors and eigenvalues. Write a script that does the following.
(i) Create the 2 × 2 matrix L (as defined above).
(ii) Create 500 random, unit length, two-dimensional vectors. (Hint: to generate a random d-dimensional unit length vector, draw d independent samples from the Gaussian distribution N(0,1) and assign each sample as one component of the vector. Now, normalize the vector to have length one.) Let R be the set of these 500 random 2-dimensional unit vectors.
(iii) For each vector r ∈ R, compute how the matrix L “distorts” r, that is, compute ˜r := Lr.
(iv) Compute the eigenvalues of L. Let λmax and λmin denote the maximum and the minimum eigenvalue respectively.
(v) For each distorted vector ˜r, compute the length k˜rk.
(vi) Create a histogram of values of k˜rk (use 50 bins) and compare it to λmax and λmin.
(vii) What relationship can you infer between k˜rk, λmax and λmin?
(viii) Now, compute the eigenvectors of L. Let vmax denote the eigenvector corresponding to the maximum eigenvalue λmax.
(ix) Make a two-dimensional plot of all the distorted vectors ˜r (in blue color) and the eigenvector Lvmax (in red color). (make sure that the x- and the y-axis are displayed at the same scale).
(x) What can you infer about the vmax from studying this plot?
Y=1
Y=2
Y=3
X=1
0.1
0.2
0.3
X=2
0.2
0.1
0.1
(i) What is the marginal distribution of X?
(ii) What is Pr[Y = 1|X = 2]?
(iii) Let f : x 7→ x2. What is E[f(X)|Y = 1]?
1.2 Fix some θ 0, and consider the function gθ : [0,∞) → R defined as .
(i) Verify that gθ is a probability distribution.
(ii) Let X be a random variable distributed as gθ. What is E[X]?
(iii) What is Variance(X)?
1.3 You live with your cat in an apartment in a relatively safe neighborhood with a crime rate of 5%. Being a cautious person, you have invested in a good quality burglar alarm that is rated to be 99% effective at the time of a break-in. Over time you have observed that your cat likes to play with the alarm causing it to trip 10% of the time. One day while you are at work your neighbor calls you telling that the alarm is ringing. What is the probability that there was a break-in?
2.1 [Linear Algebra] Consider the subspace S spanned by vectors .
(i) What is the dimension of the subspace S.
(ii) Compute the orthogonal linear projection of the point , onto the subspace S.
2.2 Prove that for any m × n real matrix A and any ρ 0, the matrix ATA + ρI is invertible (where I is the n × n identity matrix). (Hint: show that all eigenvalues of ATA + ρI are real and analyze the smallest eigenvalue.)
3.1 [Calculus and optimization] Let A ∈ Rd×D be a real matrix, and b ∈ Rd be a real vector. Define the function
f :RD → R
x 7→ kAx − bk2 + kxk2.
(i) What is ∇f(x)?
(ii) What value of x minimizes f, that is, find argminx f(x)? (Hint: compute the stationary points of f.)
4.1 [Programming practice] Download the Matlab data file hw0data.mat (instructions on Piazza on where to download the file). Write a script that does the following.
Special note for those who are not using Matlab: Python users can use scipy to read in the mat file, R users can use R.matlab package to read in the mat file, Julia users can use JuliaIO/MAT.jl. Octave users should be able to load the file directly.
(i) Load the data in hw0data.mat. It contains one matrix variable is called M.
(ii) Print the dimensions of M.
(iii) Print the 4th row and 5th column entry of M.
(iv) Print the mean value of the 5th column of M.
(v) Compute the histogram of the 4th row of M and show the figure.
(vi) Compute and print the top three eigenvalues of the matrix MTM.
4.2 We will try to understand the geometry of eigenvectors and eigenvalues of a matrix via exper-
imentation. Let L be a 2 × 2 matrix. To understand eigenvectors and
eigenvalues, we will study the action of L on random vectors and relate it to eigenvectors and eigenvalues. Write a script that does the following.
(i) Create the 2 × 2 matrix L (as defined above).
(ii) Create 500 random, unit length, two-dimensional vectors. (Hint: to generate a random d-dimensional unit length vector, draw d independent samples from the Gaussian distribution N(0,1) and assign each sample as one component of the vector. Now, normalize the vector to have length one.) Let R be the set of these 500 random 2-dimensional unit vectors.
(iii) For each vector r ∈ R, compute how the matrix L “distorts” r, that is, compute ˜r := Lr.
(iv) Compute the eigenvalues of L. Let λmax and λmin denote the maximum and the minimum eigenvalue respectively.
(v) For each distorted vector ˜r, compute the length k˜rk.
(vi) Create a histogram of values of k˜rk (use 50 bins) and compare it to λmax and λmin.
(vii) What relationship can you infer between k˜rk, λmax and λmin?
(viii) Now, compute the eigenvectors of L. Let vmax denote the eigenvector corresponding to the maximum eigenvalue λmax.
(ix) Make a two-dimensional plot of all the distorted vectors ˜r (in blue color) and the eigenvector Lvmax (in red color). (make sure that the x- and the y-axis are displayed at the same scale).
(x) What can you infer about the vmax from studying this plot?
1 file (297.0KB)
