CSCE633-Homework 3 Solved

Question 1: Activation Function in Neural Networks
Consider a two-layer fully-connected network with D input features, one hidden layer with M nodes, and one output layer. The activation function of each node is a sigmoid function of the form sigmoid(

Provide a schematic representation of the network. Define the variables for the input, theoutput, and the weights.
Express the output as a function of the input.
Calculate the number of parameters that need to be inferred.
Show that there exists an equivalent network with hidden unit activation functions givenby the hyperbolic tangent, which computes exactly the same function, where the hyperbolic tangent is given by
Hint: First find the relation between sigmoid(α) and tanh(α), then show that the parameters of the two networks differ by linear transformations.

Question 2: Image processing for human faces
In this problem, we will process face images coming from the Yale Face Dataset: http:

//vision.ucsd.edu/content/yale-face-database. This dataset contains images of the faces of 15 individuals. For each individual there are 11 images taken under a variety of conditions e.g., the person makes a happy expression, wears glasses etc.

1

Download the dataset from the above URL. Implement Principal Component Analysis (PCA) on the input images. Assume that the input vector of PCA contains all rows of an image stacked one on top of the other. You can use available libraries that calculate eigenvalues and eigenvectors of a matrix. Hint: Don’t forget to normalize the data.
Plot a curve displaying the first k eigenvalues λ1,...,λK, i.e. the energy of the first K principal components. How many components do we need to capture 50% of the energy?
Plot the top 10 eigenfaces, i.e. the eigenvectors uk, k = 1,...,10 obtained by PCA.
Select a couple of images from the data. Use the first k eigenfaces as a basis to reconstruct the images. Visualize the reconstructed images using 1, 10, 20, 30, 40, 50 components. How many components do we need to achieve a visually good result?
Hint: Reconstruction of an input vector x based on the eigenvectors u1,...,uK is given by the following expression xuk, where ck = uTkx is the projection of the input image to the kth eigenvector.

Perform face recognition: Split the input data into training and testing making sure that every person is included in each set. Use any of the classification methods that we have learnt so far to recognize a person’s identity. Use as input features the transformed feature space that resulted from PCA. Experiment with different number of PCA components through a 5-fold cross-validation. Report the recognition accuracy on the test set.
Bonus: Data augmentation: Data augmentation is a way to increase the size of our dataset and reduce overfitting, especially when we use complicated models with many parameters to learn. Using any available toolbox or your own code, implement some of these techniques and augment the original Yale Face Dataset.
Bonus: Face recognition with CNNs: Use CNNs to perform face recognition in the augmented Yale Face Dataset. Use the same split for the train and test set as in Question 2e. Experiment with different CNN parameters, e.g. filter size, stride size, activation function, dropout, pool size, etc.