CSCI4390-6390 - Assign3 - Non-Linear Dimensionality Reduction - Linear Regression - Solved

Both Part I and II have to be done by all sections. Differences have been specified by CSCI4390 and CSCI6390 labels.

You will use the Appliances energy prediction data set. You should ignore the first attribute, which is a date-time variable, and you should also remove the last attribute, which is a duplicate of the previous one. For Kernel PCA, you will use all remaining attributes. For Linear Regression, use the first attribute (after removing the date-time variable), which denotes the Appliances Energy Use, as the response variables, with the remaining attributes as predictor variables.

Part I: Kernel Principal Components Analysis (50 points)

You will implement the Kernel PCA (KPCA) algorithm as described in Algorithm 7.2 (Chapter 7, page 208). You need to compute the kernel and then center it, followed by extracting the dominant eigenvectors, which will give you the components of the directions in feature space. Next you will project and visualize the data. To compute the principal components (PCs) from the kernel matrix, you may use the inbuilt numpy function eigh.

Using the linear kernel (i.e., polynomial kernel with degree and      ; in other words           ), how many dimensions are required to

q = 1 c = 0       K(x, y) = x y

T

capture fraction of the total variance? For the same linear kernel, compute the projected points along the first two kernel PCs, and create a

α = 0.95

scatter plot of the projected points.

Next, use the covariance matrix for the original data to compute the regular principal components, i.e., for the covariance matrix. Project the data onto the first two PCs. How do the eigenvalues and the projection compare with that obtained via Kernel PCA with linear kernel?

Finally, use the gaussian kernel and repeat the exercise for kernel PCA. Project the points onto the first two PCs and plot the scatter plot. For the variance 
or spread of the gaussian kernel, namely , read it from the command line. Try different values and submit your plot for the value that makes most sense

σ2

to you (observe the projected plots for various spread values and then decide).

Part II: Linear Regression via QR Factorization (50 points)

You will implement the linear regression algorithm via QR factorization, namely Algorithm 23.1 on page 602 in Chapter 23.

CSCI4390: You can use the numpy.linalg.qr function to compute the QR factorization.

CSCI6390: You must implement QR factorization on your own, as described in Section 23.3.1 (you cannot use numpy.linalg.qr or similar function).

Next, for both sections, using the and the      matrices, you must solve for the augmented weight vector using backsubstitution. See Example 23.4

Q         R         w

on how backsolve works. You cannot use numpy.linalg.inv in your solution (but it can be used to verify your answer).

After you have computed the weight vector , you should compute the SSE value for the predictions, and also the       statistic, defined as:

w         R2

TSS−SSE

R= 2

TSS

where TSS is the total scatter of the response variable

TSS = ∑i=1 (y −µ )

n          i           Y 2

What to submit

 

Write two python scripts named as Assign3-kpca.py and Assign3-lr.py, for each of the parts, respectively.

 

For part1, read the filename from the command line, assume it is in the local directory. Your script will be run as

Assign3-kpca.py FILENAME ALPHA SPREAD. FILENAME is the datafile name, ALPHA is the approximation threshold and SPREAD is the

α          σ2

parameter for the Gaussian kernel. In other words, your script must compute and return the correct number of components to capture fraction of

α

total variance. You should also plot the projected points on the first two Kernel PCs.

Note that KPCA on the full dataset with about 20K points will be expensive and it will take around 12GB of memory if you use np.float32 for all data. If you cannot run on the full dataset, run it on the first 10K points. If that is also too much, then you should you use at least 5K points.

 

For part2, the script will be run as Assign3-lr.py FILENAME, where FILENAME is the input data file. Use 70% of the points for the training dataset, and 30% of the points for the testing dataset. Your script MUST print the following (where MSE is mean squared error):

1. the weight vector

w

 

2.           the           norm of the weight vector

L2

3.           the SSE, MSE and          values on the training data

R2

4.           the SSE, MSE and          values on the testing data

R2