The purpose of this homework is to allow you to obtain deeper understanding of the underlying working mechanisms and theory behind neural networks. The homework consists of a series of tasks which allow you to understand, develop or re-implement some of the features of the neural networks.
Task 1: Read and fully understand the article Nothing but NumPy: Understanding & Creating Neural Networks with Computational Graphs from Scratch Link. If you have not programmed in Python before, please type yourself all the code provided at the end of the article. This will help you get a feel of Python programming and help you understand what is going on. (Even if you programmed in Python before I strongly recommend you do not skip this step.)
Task 2: Get the program from Task 1 to work on the MNIST database Link. Evaluate the performance (classification error) of your program in comparison to others Link. Use minimum 3 internal layers with minimum 20 activation nodes in each internal layer.
Save your program and the corresponding results as the version 1 (V1).
Task 3:
Change the following in your program:
Replace batch gradient descent with mini-batch gradient descent Link to train the network. Re-run your program on the MNIST dataset and compare the performance (execution speed due to mini-batch change and the classification error). Save this program as version 2 (V2).
Change the activation function to another based on your choice Link. Note that changing the activation function will require you to change backpropagation derivatives. Re-run your program on the MNIST dataset and compare the performance (execution speed due to the activation function change and the classification error). Save this program as version 3 (V3).
Task 4: Summarize and explain the observed performance changes in all 3 versions of your program in a short presentation (not more than 10 slides).
