EECS491-Assignment 4 Solved

Exercise 1. Multivariate Gaussians 
1.1 (5 pts) Consider the 2D normal distribution
Define three separate 2D covariance matrices for each of the following cases: and are uncorrelated; and are
correlated; and and are anti-correlated. Plot samples from these distributions to show these properties. Use a
different mean for each. Make sure your plots show the density.
1.2  Compute the principal axes for each of these distributions, i.e. the eigenvectors of the covariance matrices.
Use can use a linear algebra package. Plot the samples again, but this time overlay the 1, 2, and 3-sigma contours and
with the scaled eigenvectors.
Exercise 2. Linear Gaussian Models 
Consider two independent multi-dimensional Gaussian random vector variables
Now consider a third variable that is the sum of the first two:
2.1  What is the expression for the distribution ?
2.2  What is the expression for the condidtional distribution ?
2.3 (Write code to illustrate the result in Q2.1. Show both the components of and that the sampling
from the analytic result is the same as adding two samples.
Exercise 3. Dimensionality Reduction and PCA 
In this quesiton you will use principal component analysis to reduce the dimensionality of your data and analyze the
results.
3.1  Find a set of high dimensional data. It should be continuous and have at least 6 dimensions, e.g. stats for
sports teams, small sound segments or images patches also work. Note that if the dimensionality of the data is too
large, you might run into computational efficiency problems using standard methods. Describe the data and illustrate it,
if appropriate.
3.2  Compute the principal components of the data. Plot a few of the largest eigenvectors and interpret them in
terms of how there are modeling the structure of the data.
3.3  Plot, in decreasing order, the cumulative percentage of variance each eigenvector accounts for as a function
of the eigenvector number. These values should be in decreasing order of the eigenvalues. Interpret these results.
3.4  Plot the original data projected into the space of the two principal eigenvectors (i.e. the eigenvectors with
the largest two eigenvalues). Be sure to either plot relative to the mean, or subtract the mean when you do this. Interpret
your results. What insights can you draw? Interpret the dimensions of the two largest principal components. Which
dimensions of the data are correlated? Or anti-correlated?
Exercise 4. Gaussian Mixture Models 
4.1 ( Use the EM equations for multivariate Gaussian mixture model to write a program that implements the
Gaussian Mixture Model to estimates from an ensemble of data the means, covariance matrices, and class probabilities.
Choose reasonable values for your initial values and a reasonable stopping criterion. Explain your code and the steps of
the algorithm. Do not assume a diagonal or isotropic covariance matrices.
4.2  Write code to plot the 3-sigma contours of each Gaussian overlayed on the data (try to find a library function
to plot ellipses). Illustrate with an example.
4.3 )Define a two-model Gaussian mixture test case, synthesize the data, and verify that your algorithm infers the
(approximately) correct values based on training data sampled from the model and plotting the results.
4.4  Apply your model to the Old Faithful dataset (supplied with the assignment files). Run the algorithm for the
cases , , and . For each case, plot the progression of the solutions at the beginning, middle, and
final steps in the learning. For each your plots (you should have 9 total), you should also print out the corresponding
values of the mean, covariance, and class probabilities.
Exploration 
Like in previous assignments, in this exercise you have more lattiude and are meant to do creative exploration. The
intention is for you to teach yourself about a topic beyond what's been covered above.