CS7642 Project #3 Solution

Problem
Description
As you encountered in the first project, replication of previously published results can be an interesting and challenging task. You learned that researchers often leave out important details that cause you to perform extra experimentation to produce the right results.

For this project, you will be reading “Correlated Q-Learning” by Amy Greenwald and Keith Hall. You are then asked to replicate the results found in Figure 3(parts a-d). You can use any programming language and libraries you choose.
Procedure
● Read the paper.
● Develop a system to replicate the experiment found in section "5. Soccer Game"
○ This will include the soccer game environment
○ This will include agents capable of Correlated-𝖰, Foe-𝖰, Friend-𝖰, and 𝖰-learning
● Run the experiment found in section "5. Soccer Game"
○ Collect data necessary to reproduce all the graphs in Figure 3
● Create graphs demonstrating
○ The 𝖰-value difference for all agents
○ The quality of the code is not graded. You don’t have to spend countless hours adding comments, etc. But, it will be examined by the TAs.
○ Make sure to include a README.md file for your repository
■ Include thorough and detailed instructions on how to run your source code in the README.md
○ You will be penalized by 50 points if you:
■ Do not have any code or do not submit your full code to the GitHub repository
■ Do not include the git hash for your last commit in your paper
● Write a paper describing your agents and the experiments you ran
○ Include the hash for your last commit to the GitHub repository in the paper’s header.
○ The rubric includes a few points for formatting. Make sure your graphs are legible and you cite sources properly. While it is not required, we recommend you use a conference paper format. Just pick any one.
○ 5 pages maximum -- really, you will lose points for longer papers.
○ Describe the game
○ Describe the experiments/algorithms replicated: implementation/outcome/etc
○ Explain your experiments
○ The paper should include your graphs
■ And, discussions regarding them
○ Discuss your results
■ How well do they match?
■ Significant differences?
○ Describe any problems/pitfalls you encountered (e.g. unclear parameters, contradictory descriptions of the procedure to follow, results that differ wildly from the published results)
■ What steps did you take to overcome them
■ What assumptions you made
● Justifications for such assumptions
○ Save this paper in PDF format
○ Submit!
● Celebrate your mastery of Reinforcement Learning!
Your grade will largely be based upon your report and analysis.
Resources
The concepts explored in this homework are covered by:
● Lectures
○ Game Theory (all of them)
● Readings
○ Greenwald-Hall (2003)
Submission Details
The submission consists of:
● Your written report in PDF format (Make sure to include the git hash of your last commit)
To complete the assignment, submit your written report to Project 3 under your Assignments on Canvas: https://gatech.instructure.com
Note: Late is late. It does not matter if you are 1 second, 1 minute, or 1 hour late. If Canvas marks your assignment as late, you will be penalized. Additionally, if you resubmit your project and your last submission is late, you will incur the penalty corresponding to the time of your last submission.
Grading and Regrading
When your assignments, projects, and exams are graded, you will receive feedback explaining your errors (and your successes!) in some level of detail. This feedback is for your benefit, both on this assignment and for future assignments. It is considered a part of your learning goals to internalize this feedback. This is one of many learning goals for this course, such as: understanding game theory, random variables, and noise.
It is important to note that because we consider your ability to internalize feedback a learning goal, we also assess it. This ability is considered 10% of each assignment. We default to assigning you full credit. If you request a regrade and do not receive at least 5 points as a result of the request, you will lose those 10 points.