Task 3: Run the experiments and report
The goal of the experiments in this part is to have you explore the consequences of important decisions made while training neural networks. Split the dataset into training and test sets. Use test set to estimate performance in all of the experiments after training the model with training set. Evaluate the performance using accuracy. You are welcome to perform any experiments and analyses you see fit (e.g., the effect of data augmentation / dropout / number of hidden layers / ... on accuracy), but at a minimum you must complete the following experiments in the order stated below:
1. First of all, create three different models: (1) an MLP with no hidden layers, i.e., it directly maps the inputsto outputs, (2) an MLP with a single hidden layer having 128 units and ReLU activations, (3) an MLP with 2 hidden layers each having 128 units with ReLU activations. It should be noted that since we want to perform classification, all of these models should have a softmax layer at the end. After training, compare the test accuracy of these three models on the MNIST dataset. Comment on how non-linearity and network depth effects the accuracy. Are the results that you obtain expected?
2. Take the last model above, the one with 2 hidden layers, and create two different copies of it in which the activations are now sigmoid and tanh. After training these two models compare their test accuracies with model having ReLU activations. Comment on the performances of these models: which one is better and why? Are certain activations better than others? If the results are not as you expected, what could be the reason?
3. Create an MLP with 2 hidden layers each having 128 units with ReLU activations as above. However, this time, add L2 regularization (weight decay) to the cost and train the MLP in this way. How does this affect the accuracy?
4. Create an MLP with 2 hidden layers each having 128 units with ReLU activations as above. However, this time, train it with unnormalized images. How does this affect the accuracy?
5. You can report your findings either in the form of a table or a plot in the write-up. However, include in yourcolab notebooks the plots of the test and train performance of the MLPs as a function of training epochs. This will allow you to see how much the network should be trained before it starts to overfit to the training data.
