$25
SOFT8033 Assignment 2- Spark Algorithms Solved
It’s been already a few weeks since you started your short-term internship in the Data Analytics Department of the start-up OptimiseYourJourney, which will enter the market next year with a clear goal in mind: “leverage Big Data technologies for improving the user experience in transportation”. Your contribution in Assignment 1 has proven the potential OptimiseYourJourney can obtain by applying MapReduce to analyse large-scale public transportation datasets as the one in the New York City Bike Sharing System: https://www.citibikenyc.com/
In the department meeting that has just finished your boss was particularly happy, again.
• The very same dataset from Assignment 1 (let’s call it my_dataset_1) provides an opportunity to leverage other large-scale data analysis libraries, such as Spark SQL.
• The graph structure of the dataset allows you to explore the potential of Spark GraphFrames, a small library of Spark specialised in the parallel execution of classical graph algorithms. To do so, two small graph examples (let’s call them my_dataset_2 and my_dataset_3) are provided to explore the classical algorithms of: o Dijkstra – for finding the shortest path from a source node to the remaining nodes.
o PageRank – for assigning a value to each node based on its neighbourhood.
DATASET 1:
This dataset occupies ~80MB and contains 73 files. Each file contains all the trips registered the CitiBike system for a concrete day:
• 2019_05_01.csv => All trips registered on the 1st of May of 2019.
• 2019_05_02.csv => All trips registered on the 2nd of May of 2019.
• …
• 2019_07_12.csv => All trips registered on the 12th of July of 2019.
Altogether, the files contain 444,110 rows. Each row contains the following fields:
start_time , stop_time , trip_duration , start_station_id , start_station_name , start_station_latitude , start_station_longitude , stop_station_id , stop_station_name , stop_station_latitude , stop_station_longitude , bike_id , user_type , birth_year , gender , trip_id
• (00) start_time
! A String representing the time the trip started at.
<%Y/%m/%d %H:%M:%S>
! Example: “2019/05/02 10:05:00”
• (01) stop_time
! A String representing the time the trip finished at.
<%Y/%m/%d %H:%M:%S>
! Example: “2019/05/02 10:10:00”
• (02) trip_duration
! An Integer representing the duration of the trip.
! Example: 300
• (03) start_station_id
! An Integer representing the ID of the CityBike station the trip started from.
! Example: 150
• (04) start_station_name
! A String representing the name of the CitiBike station the trip started from.
! Example: “E 2 St &; Avenue C”.
• (05) start_station_latitude
! A Float representing the latitude of the CitiBike station the trip started from.
! Example: 40.7208736
• (06) start_station_longitude
! A Float representing the longitude of the CitiBike station the trip started from.
! Example: -73.98085795
• (07) stop_station_id
! An Integer representing the ID of the CityBike station the trip stopped at.
! Example: 150
• (08) stop_station_name
! A String representing the name of the CitiBike station the trip stopped at.
! Example: “E 2 St &; Avenue C”.
• (09) stop_station_latitude
! A Float representing the latitude of the CitiBike station the trip stopped at.
! Example: 40.7208736
• (10) stop_station_longitude
! A Float representing the longitude of the CitiBike station the trip stopped at.
! Example: -73.98085795
• (11) bike_id
! An Integer representing the id of the bike used in the trip.
! Example: 33882.
• (12) user_type
! A String representing the type of user using the bike (it can be either “Subscriber” or “Customer”).
! Example: “Subscriber”.
• (13) birth_year
! An Integer representing the birth year of the user using the bike.
! Example: 1990.
• (14) gender
! An Integer representing the gender of the user using the bike (it can be either 0 => Unknown; 1 => male; 2 => female).
! Example: 2.
• (15) trip_id
! An Integer representing the id of the trip.
! Example: 190.
DATASET 2:
This dataset consists in the file tiny_graph.txt, which contains 26 edges (indeed, 13 edges, one on each direction) in a graph with 8 nodes.
DATASET 3:
This dataset consists in the file tiny_graph.txt, which contains 18 edges (indeed, 9 edges, one on each direction) in a graph with 6 nodes.
TASKS / EXERCISES.
The tasks / exercises to be completed as part of the assignment are described in the next pages:
• The following exercises are placed in the folder my_code:
1. A02_Part1/A02_Part1.py
2. A02_Part2/A02_Part2.py 3. A02_Part3/A02_Part3.py
4. A02_Part4/A02_Part4.py
5. A02_Part5/A02_Part5.py
Marks are as follows:
1. A02_Part1/A02_Part1.py => 18 marks
2. A02_Part2/A02_Part2.py => 18 marks 3. A02_Part3/A02_Part3.py => 18 marks
4. A02_Part4/A02_Part4.py => 18 marks
5. A02_Part5/A02_Part5.py => 28 marks
Tasks:
1. A02_Part1/A02_Part1.py
2. A02_Part2/A02_Part2.py
3. A02_Part3/A02_Part3.py
Complete the function my_main of the Python program.
Do not modify the name of the function nor the parameters it receives.
The entire work must be done within Spark SQL:
§ The function my_main must start with the creation operation 'read' above loading the dataset to Spark SQL.
§ The function my_main must finish with an action operation 'collect', gathering and printing by the screen the result of the Spark SQL job.
§ The function my_main must not contain any other action operation 'collect' other than the one appearing at the very end of the function.
§ The resVAL iterator returned by 'collect' must be printed straight away, you cannot edit it to alter its format for printing.
4. A02_Part4/A02_Part4.py
Complete the function compute_page_rank of the Python program.
Do not modify the name of the function nor the parameters it receives.
The function must return a dictionary with (key, value) pairs, where:
§ Each key represents a node id.
§ Each value represents the pagerank value computed for this node id.
5. A02_Part5/A02_Part5.py
Complete the function my_main of the Python program.
Do not modify the name of the function nor the parameters it receives.
The entire work must be done within Spark SQL:
§ The function my_main must start with the creation operation 'read' above loading the dataset to Spark SQL.
§ The function my_main must finish with an action operation 'collect', gathering and printing by the screen the result of the Spark SQL job.
§ The function my_main must not contain any other action operation 'collect' other than the one appearing at the very end of the function.
§ The resVAL iterator returned by 'collect' must be printed straight away, you cannot edit it to alter its format for printing.
