40.319-Homework 3 Solved

Problem 1
In this problem, we will implement the EM algorithm for clustering. Start by importing the required packages and preparing the dataset.

import numpy as np import matplotlib . pyplot as plt

from numpy import linalg as LA from matplotlib . patches import Ellipse from sklearn . datasets . samples  generator import make  blobs from scipy . stats import multivariate  normal

K = 3

NUMDATAPTS = 150

X, y = make  blobs ( n  samples=NUMDATAPTS, centers=K, shuffle=False , random  state=0, cluster  std =0.6)

g1 = np. asarray ([[2.0 , 0] , [ −0.9 , 1]]) g2 = np. asarray ([[1.4 , 0] , [0.5 , 0 . 7 ] ] ) mean1 = np.mean(X[ : int (NUMDATAPTS/K)])

mean2 = np.mean(X[ int (NUMDATAPTS/K):2∗ int (NUMDATAPTS/K)])

X[ : int (NUMDATAPTS/K)] = np. einsum( ’nj , ij −ni ’ ,

X[ : int (NUMDATAPTS/K)] − mean1 , g1) + mean1

X[ int (NUMDATAPTS/K):2∗ int (NUMDATAPTS/K)] = np. einsum( ’nj , ij −ni ’ ,

X[ int (NUMDATAPTS/K):2∗ int (NUMDATAPTS/K)] − mean2 , g2) + mean2

X[ : , 1 ] −= 4

(a)    Randomly initialize a numpy array mu of shape (K, 2) to represent the mean of the clusters, and initialize an array cov of shape (K, 2, 2) such that cov[k] is the identity matrix for each k. cov will be used to represent the covariance matrices of the clusters. Finally, set π to be the uniform distribution at the start of the program.

(b)    Write a function to perform the E-step:

def E  step ():

gamma = np. zeros ((NUMDATAPTS, K))

. . .

. . .

return gamma

(c)     Write a function to perform the M-step:

def M  step(gamma):

. . . . . .

(d)    Now write a loop that iterates through the E and M steps, and terminates after the change inlog-likelihood is below some threshold. At each iteration, print out the log-likelihood, and use the following function to plot the progress of the algorithm:

def plot  result (gamma=None ):

ax = plt . subplot (111 , aspect=’ equal ’ ) ax . setxlim ([−5 , 5]) ax . set  ylim ([−5 , 5]) ax . scatter (X[: , 0] , X[: , 1] , c=gamma, s=50, cmap=None)

for k in range(K):

l , v = LA. eig (cov [ k ]) theta = np. arctan (v [1 ,0]/ v [0 ,0])

e = Ellipse ((mu[k , 0] , mu[k , 1]) , 6∗ l [0] , 6∗ l [1] , theta ∗ 180 / np. pi )

 e . set alpha (0.5) ax . add artist (e) plt . show()

(e)    Use sklearn’s KMeans module

https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html to perform K-means clustering on the dataset, and compare both clustering results.

Problem 2
Let p and q be distributions on {1,2,3,4,5} such that , and

 .

(a)    Compute the cross-entropy H(p,q) in bits. Is H(q,p) = H(p,q)?

(b)    Compute the entropies H(p) and H(q) in bits.

(c)     Compute the KL-divergence DKL(p|q) in bits.

Show all working and leave your answers in fractions.

Problem 3
(a)    Perform singular value decomposition (SVD) on the following matrix

 .

(b)    For a general design matrix X, why are the columns of the transformed matrix T = XV orthogonal?

Problem 4
In this problem, we will perform principal component analysis (PCA) on sklearn’s diabetes dataset. Start by importing the required packages and load the dataset.

import numpy as np from sklearn import decomposition from sklearn import datasets

X = datasets . load  diabetes (). data

You can find out more on how to use sklearn’s PCA module from:

For this problem, make sure the design matrix is first normalized to have zero mean and unit standard deviation for each column.

(a)    Write code to print the matrix V that will be used to transform the dataset, and print all the singular values.

(b)    Now perform PCA on the dataset and print out the 3 most important components for the first10 data-points.

Problem 5
An AR(2) model assumes the form

rt = φ0 + φ1rt−1 + φ2rt−2 + at,

where at is a white noise sequence. Show that if the model is stationary, then

  (assume φ1 + φ2 6= 1);

(b) the ACF is given by

 .