CS433-Homework 1-Tweet analysis with MapReduce Solved

Data Intensive Computing 

 



 

In this homework, you’ll write a MapReduce algorithm to analyze sample twitter dataset containing approximately 3.8 million tweets.

 

            •           Install Hadoop to your own server or use cs433.cse.unr.edu.

                        •           You need to use jump host to access cs433.cse.unr.edu from outside of UNR campus. So, you can first login to nxlogin.engr.unr.edu and from there to cs433.cse.unr.edu

            •           Download ZIP file in here. Its size is around 405 MB. The files are already uploaded to HDFS in cs433.cse.unr.edu under “/” directory. Check by running “Hadoop dfs -ls /homework1/” 

            •           Unzip the file and upload “training_set_tweets.txt” (tweets) and “training_set_users.txt” (users) files to HDFS

 

Once your Hadoop cluster is up and running do the following tasks:

            •           Show HDFS daemons (hint: search for processes called namenode, datanode) (5 pts)

            •           Show how many blocks created in HDFS for “tweets” file, either through command line or namenode web ui (5 pts)

            •           Show how many map tasks are created when you try to process “tweets” file in HDFS (10pts)

            •           Set the number of reduce tasks to 3 and show that Hadoop created 3 reduce tasks  (10 pts)

            •           Write a MapReduce code to count the number of hash tags occurrences and find the most repeated 10 hashtags. (20 pts)

            •           Write a MapReduce code find the most tweeted 10 days. (Tweets are associated with time stamps so you need to count all the tweets posted in same days) (20 pts)

            •           Write a MapReduce code to find the most tweeted 10 cities along with the number of tweets (“training_set_users.txt” file has user_id  city relation to extract city information) (30 pts)

Important Notes

            •           It is NOT allowed to use global variables in Q5 and Q6 as they are easy to implement with single MR job. 

            •           Although it is not an ideal solution, you can use a global variable in Q7 to keep the solution simple. However, I offer 10pt bonus points if you implement without using a global variable. You'll need to write multiple jobs in one application and use reduce-side join to implement this way.

 

 

What to deliver

Create following files/folders and compress them in a single zip file with name <LASTNAME>_<NAME>_HW1.zip and submit on WebCampus

 

            •           Take screenshots for Question 1-4 to a file answers1-4.pdf

            •           Copy the most repeated 30 hashtags along with number of occurrences to a file called “popular_tweets.txt” file

            •           Copy the most tweeted 20 days along with number of tweets to a file called “most_tweeted_days.txt” file

            •           Copy the most tweeted 10 cities along with number of tweets to a file called “most_tweeted_citites.txt” file

            •           Create three directories Q5, Q6, and Q7 and copy your source code for question 5, 6, and 7 into those directories.

            •           [Important] Create README file that shows how to run compile and run your code 

            •           [Important] Do not include input files in your final submission