Lab 8
Neural Networks and Deep Learning
Advanced Machine Learning DATA 442/642
Exercise 1
Consider a two-dimensional class problem that involves two classes ω1(+1) and ω2(−1). Each one of them is modeled by a mixture of equiprobable Gaussian distributions. Specifically, the means of the Gaussians associated with ω1 are [−5,5]> and [5,−5]>, while the means of the Gaussians associated with ω2 are [−5,−5]>,[0,0]>, and [−5,−5]>. The covariances of all Gaussians are σ2I, where σ2 = 1.
(a) Generate and plot a data set X1 (training set) containing 100 points from ω1 (50 points from each associated Guassian) and 150 points from ω2 (again 50 points from each associated Gaussian). In the same way, generate an additional set X2 (test set).
(b) Based on the training set X1, train a train a two-layer neural network with two nodes in the hidden layer, each one having the rectified linear activation function or ReLU and a single output node with linear activation function using the standard backpropagation algorithm for 6000 iterations and step size equal to 0.01. Compute the training and test errors, based on X1 and X2 respectively. Also, plot the test points as well as the decision lines formed by the network.
(c) Repeat step (b) for step size equal to 0.0001 and comment on the results.
(d) Repeat step (b) for step size equal to 0.0001 and k = 1,4,20,50 hidden layer nodes and comment on the results.
