MLP-Coursework 2 Solved

Machine Learning Practical 2020/21: Coursework 2

1          Introduction
The aim of this coursework is to explore the classification of images using convolutional neural networks on a different dataset, CIFAR100 (pronounced as “see far 100”). CIFAR100 consists of 60,000 32 × 32 colour images in 100 classes, with 600 images per class. The first part of the coursework concerns with debugging a “broken” neural network and quantitatively analyzing the problem behind this broken network. The second part involves exploring the solutions in literature for fixing this “broken” neural network, and then subsequently implementing one of these solutions to improve the performance and training of this network.

In order to support your experiments, we have acquired Google Cloud Platform credits which allow the use of the Google Compute Engine infrastructure. You will need this to run all the tasks of the coursework (which will be carried out in PyTorch). Each student enrolled on the MLP course will receive a $50 Google Cloud credit coupon which is enough to carry out the experiments required for this coursework. You will receive an email

which will give you the URL you will need to access in order to request your Google Cloud Platform coupon.

As in the previous coursework, you will need to submit your python code and a report. The detailed submission instructions are given in Section 8.2 – please follow these instructions carefully.

2          Github branch mlp2020-21/coursework_2
The provided code and setup information for this coursework is available on the course Github repository on a branch mlp2020-21/coursework_2. To create a local working copy of this branch in your local repository you need to do the following.

1.    Make sure all modified files on the branch you are currently have been committed (see notes/gettingstarted-in-a-lab.md if you are unsure how to do this).

2.    Fetch changes to the upstream origin repository by running git fetch origin

3.    Checkout a new local branch from the fetched branch using git checkout -b coursework_2 origin/mlp2020-21/coursework_2

You will now have a new branch in your local repository with everything you need to carry out the coursework.

Before continuing, remember to run bash install.sh to install some additional dependencies required. This assumes that you have already installed your environment as explained in Lab1, under notes/environment-set-up.md.

This branch includes the following additions to your setup:

•    For the PyTorch Framework:

–    A Jupyter notebook, Coursework_2_Pytorch_Introduction.ipynb, which introduces Pytorch, and provides further resources on tutorials, documentation and debugging.

–    A directory pytorch_mlp_framework/, which includes tooling and ready to run scripts to enable straightforward experimentation on GPU. Documentation on this is included in notes/pytorch-experiment-framework.md

–    A note on how to use the Google Cloud Platform (using your student credits), notes/google_cloud_setup.md.

•    For the report:

–    A directory called report which contains the LaTeX template and style files for your report. You should copy all these files into the directory which will contain your report.

3          Tasks
This coursework has three tasks and objectives:

•    Diagnose a deep CNN that cannot be properly trained due to optimization issues. Discuss what the problem is, why it happens in a deep CNN and quantitatively analyze the problem.

•    Discuss solutions present in the literature to address this problem.

•    Propose in detail and implement one of the solutions to the problem. Since the aim of this is to improve the training behaviour of the network, the solutions should then be evaluated in terms of convergence speed and per-epoch training accuracy/loss and gradient flows.

Carrying out larger convolutional network experiments using the MLP numpy framework is computationally inefficient because (1) it runs on CPU and not on GPU, and (2) a default implementation of a convolutional layer is unlikely to be computationally efficient. For these reasons in this coursework, which involves running convolutional network experiments we will use GPU computing (on the Google Compute Engine) and the highly efficient PyTorch framework.

4          Task 1: Identifying training problems of a deep CNN
Time budget[1]: This section should take about 20% of the time you have allocated for your coursework.

This part of the coursework involves debugging and tuning deep CNNs using PyTorch and the Google Compute Engine.

Identifying and resolving any optimization-related problems that might prevent your model from fitting the training set are critical skills when working with deep neural networks. Being able to work around this problem is key to building high performance deep neural networks.

4.1        Introducing our broken CNN
Note: This section’s objective is to test your knowledge, understanding and research skill. It’s not meant to be an implementation-oriented section. The solutions only require you to add at most 4-5 lines of code.

Using the Pytorch-based research framework that can be found in the pytorch_mlp_framework folder, we have built, trained and evaluated 2 deep CNNs. One consisting of a total of 7 convolutional layers + one fully connected, and another consisting of 37 convolutional layers + one fully connected layer. Figure 1 illustrates the

 

Figure 1: Training and validation plots of Healthy (VGG_08) vs Broken (VGG_38) in terms of error and accuracy.

training/val loss performance of the two models. One can clearly see that the 37 layer CNN (VGG_38) was unable to minimize its loss, unlike the healthy 7 layer CNN (VGG_08) which converges to a low error. Given that we know that extra layers means more abstraction power, parameters and capacity, one would expect the deeper model to be doing better at learning than the shallow one, however this is simply not the case.

Identifying the problem. Construct a hypothesis as to what is causing the issue in Figure 1. You can reproduce the figures by running the notebooks/Plot_Results.ipynb notebook. This file takes as input the collected metrics in folders VGG_08 and VGG_38.

Quantitative Analysis of the problem. Support your position of the problem using arguments based on quantitative observations. For instance, Figure 2 shows how the gradient flows in the healthy network, VGG_08 across layers. These are the gradients with respect to the weights of the model. Visualize and discuss how gradient flows for the broken network affects/does not affect the loss curves, training and convergence of the 37 layer CNN.

4.1.1     Implementation Guidelines

The curves shown in Figure 1 can be reproduced by running the bash scripts to train each model from scratch with the default settings given in run_vgg_08_default.sh and run_vgg_38_default.sh.

For reproducing Figure 2 and visualizing the gradient flows for the broken network, implement the function plot_grad_flows() within pytorch_mlp_framework/experiment_builder.py. This function takes as input the model parameters during training, accumulates the absolute mean of the gradients in all_grads and the layer names in layers. The matplotlib function plt plots gradient values for each layer and the function plot_grad_flows() returns this final plot.


5          Task 2: Background Literature
Time budget1: This section should take about 20% of the time you have allocated for your coursework.

There exist at least several methods that can improve the training performance of the broken 37 layer CNN introduced in Section 4.1. Discuss, in your own words and in no any three out of the four research papers given below.

•    Batch normalization: Accelerating deep network training by reducing internal covariate shift, Ioffe and Szegedy [2015].

Gradient flow

 

Figure 2: Gradient Flow in each layer for the VGG_08 network.

•    Deep residual learning for image recognition, He et al. [2016].

•    Densely connected convolutional networks, Huang et al. [2017]

•    Deeply-supervised nets, Lee et al. [2015]

This discussion should outline what problems the papers address, are these related to the problem you analyze in

Task 1, what/which methods they adopt to solve it, brief details of the method and the final conclusion from the papers. Summary of each paper should be limited to 200-300 words.


6          Task 3: Solution and Experiments
Time budget1: This section should take about 30% of the time you have allocated for your coursework.

Solution Overview. The recommended solutions are in the four papers listed in section 5. Pick and discuss any one solution in detail in this section. Discuss theoretically and intuitively why you chose this solution and how it addresses/improves the training performance of the VGG_38 model.

Experiments. The recommended solutions are described in Ioffe and Szegedy [2015], He et al. [2016]. You can also find alternative ones in Huang et al. [2017], Lee et al. [2015] which can be more challenging to implement. We have written the MLP Pytorch framework in a way that allows you to implement said solutions with minimal effort. Each solution will require at approximately 8-10 lines of code.

Important Components to Note

1.    The classes ConvolutionalProcessingBlock and ConvolutionalDimensionalityReductionBlock located within pytorch_mlp_framework/model_architectures.py. The former class implements a

basic cascade of 2 convolutional layers, each followed by a Leaky ReLU activation function, while the latter implements a basic cascade of 2 convolutional layers, with an average pooling layer in the middle,

which effectively halves the height and width dimensions of the tensor volume passing through it. The former is used as the basic building block of the model (repeated num_blocks_per_stage times),

while the latter is used only as a dimensionality reduction layer, usually once after each network stage. A network stage is considered a cascade of convolutional layers, followed by a dimensionality reduction function such as average pooling.

2.    Lines 43-50 in pytorch_mlp_framework/train_evaluate_image_classification_system.py showcase how one can create an if-else structure that can read arguments from the command line in order to choose which processing and dim_reduction blocks will be used.

3.    Lines 17-27 in the same file showcase how some simple data-augmentation strategies can be applied to the system (useful in improving the generalization of the model.)

Implementing a proposed solution

1.    Copy the classes ConvolutionalProcessingBlock and ConvolutionalDimensionalityReductionBlock located within pytorch_mlp_framework/model_architectures.py, and provide a name referring to your proposed solution. Then change the two blocks to implement your proposed solution.

2.    To create a new block ensure that you rewrite the build_module() and forward() methods.

3.    Add a new clause in the if-else structure in lines 43-50 of pytorch_mlp_framework/train_evaluate _image_classification_system.py to add a choice for a configuration using your new blocks.

4.    Write small unit-tests for your blocks to ensure that the layers can fprop some data without throwing errors related to Pytorch.

5.    Once tested, you can use the argument parser to easily run experiments with your new modules. Hint: Have a look at the block_type argument under pytorch_mlp_framework/arg_extractor.py, to get an idea on how to easily pass module names as an argument.

Note: Some potential solutions might not require modification of the above, but those are considered more advanced. If you choose that path, you will also bear the responsibility of figuring out the design of your code. That being said, most of known methods can be implemented using the above 3 steps.

To break this down:

1.    Select and implement one method to address the problem in Task 1.

2.    Design and run experiments to investigate the performance of the method. Make sure to run your experiments on multiple seeds.

3.    Build a final model using the knowledge you have gained and evaluate its test performance.

4.    Quantitatively and intuitively analyze how this solution addressed the problem.

5.    Write about what you did and why in your report.

Important thing to note about experiments:

1.    The purpose of these experiments is NOT to report results with the best combination of hyper-parameters but finding the ones so that your solution addresses the problem discussed in Task 1 effectively. You will have to do a valid hyper-parameter search over learning rates, batch size, weight decay coefficients, etc. such that your model achieves a stable training and testing accuracy with the proposed solution. Make sure you choose the values wisely and not exhaust your google cloud credits.

2.    Designing experiments so that you are in a position to draw conclusions from your experiments is more important than doing as many experiments as possible.

3.    When reporting the results of experiments, make sure that the comparison/contrast you are exploring is clear in the way you present the results.

When writing the report on this task you should include the following:

•    Outline and explain the research questions you are investigating. Provide citations if appropriate.

•    Explain the methodology used – in this case the approach/method picked to solve the problem. Provide citations if appropriate. • Describe carefully the experiments carried out, providing enough information to make them reproducible. Present your results clearly and concisely. Graphs and tables should be constructed to make clear the contrasts and comparisons you are interested in based on the research questions. For instance, some interesting evaluation criteria that can be used to compare different strategies are classification accuracy, learning speed of your network and gradients.

•    Discuss your results, with reference to the research questions, and if appropriate with reference to the literature. What conclusions can you draw from your experiments and are they consistent with the literature?

Using Google Compute Engine

1.    You will receive an email containing the URL you will need to access in order to request a Google Cloud Platform coupon, and information about how to do this.

2.    In the coursework_2 branch of the GitHub, notes/google_cloud_setup.md gives the instructions you should follow to set up a Google Compute Engine instance to carry out this coursework.

3.    The PyTorch experimental framework that is used for this coursework is described in notes/pytorch-experiment-framework.md and in notebooks/Coursework_2_Pytorch_experiment_framework.ipynb


Abstract and Introduction:

Discussion and Conclusion: 

The directory coursework_2/report contains a template for your report (mlp-cw2-template.tex); the generated pdf file (mlp-cw2-template.pdf) is also provided, and you should read this file carefully as it contains some useful information about the required structure and content. The template is written in LaTeX, and we strongly recommend that you write your own report using LaTeX, using the supplied document style mlp2020 (as in the template).

You should copy the files in the report directory to the directory containing the LaTeX file of your report, as pdflatex will need to access these files when building the pdf document from the LaTeX source file. The coursework 1 spec outlines how to create a pdf file from a LaTeX source file.

Your report should be in a 2-column format, based on the document format used for the ICML conference. The report should be a maximum of 6 pages long, not counting the references. We will not read or assess any parts of the report beyond this limit.

As discussed in the coursework 1 spec, all figures should ideally be included in your report file as vector graphics files, rather than raster files as this will make sure all detail in the plot is visible.

If you make use of any any books, articles, web pages or other resources you should appropriately cite these in your report. You do not need to cite material from the course lecture slides or lab notebooks.