CS451 Assignment # 3 – Reinforcement Learning and Self Organizing Maps Solution

Assignment # 3 – Reinforcement Learning and Self Organizing Maps

Objective: The objective of this assignment is to enable students to implement a basic reinforcement learning agent. The assignment also trains them to use a specialized form of artificial neural networks (called Self Organizing Maps) for clustering and visualization.

Question 1 [25 points]
You have to train your reinforcement learning based agent that navigates in a N * N grid. The grid world offers reward (+100) at different points. Also, there are certain obstacles in the grid that restrict movement of the agent. Your agent is moving around (via Up, Down, Left, Right actions) to maximize its reward.

A sample grid is shown above where obstacles are represented as red cells (-100) and rewards as green cell(s) (+100). You want to minimize unnecessary movement of agent and make them take optimal actions.
Your task is to learn the value function via episodic learning. The agent will run multiple episodes in the grid and will update the value of each state encountered based on the outcome of the episode. An episode will end either when the agent reaches an absorbing state or max number of steps are performed. During the episodes, actions will be selected using Boltzman distribution. You have to try with different configurations of grid and ensure that your agent is generic and can work fine with any configuration of grid.
Grading:
The grading will be based on the following components:
Correct Implementation of algorithm 45%
Adjustable grid configuration 15
Convergence of value function 15%
Visualization of learned policy 25%

Question 2 [25 points] Clustering world data using SOM
In this question, you will build Self Organizing Maps (SOM) on a world dataset of your choice (such as world poverty data, country-wise environmental data, COVID19 data etc.) to build an effective visualization of that dataset. You have to make sure that there are multiple attributes in that dataset that make the cluster analysis meaningful.
SOM will result in the following type of clustering:
(Note: The sample is taken from World Poverty Map showing clustering based on poverty related dataset)

where each color represents a cluster and its varying shades denotes nearby/similar clusters.
Your implementation of the algorithm and the dataset integration should be generic that support updated data files with more rows and/or columns.
Bonus:
The representative color of each country will be used to color that country on world map resulting in a visualization similar to the following:

You can further enhance your SOM visualization by making use of timeseries data to show changes/progression of the underlying factors over time.

Grading:
The grading will be based on the following components:
Component Weight
Correct Implementation of algorithm 50%
Generic dataset integration 20%
Visualization of SOM grid 20%
Dataset selection (relevance, uniqueness, comprehensiveness of dataset) 10%
Bonus: Visualization on map + datewise Progression 20%

Submission: You will submit a zip file containing separate folders for the source code of two questions and a combined report talking about your problem formulation and results/visualization.