Tutorial Meeting #2

Data Science and Machine Learning

Map-Reduce and
the New Software Stack
DAMA60 Group
School of Science and Technology
Hellenic Open University
Introduction

• Once upon a time….

• There used to be just a single computer
• It used to perform calculations all by itself

• Later in the years to come….

• Data sources started proliferating
• Size of the data sources that are used in calculations exploded
• Advanced programming techniques started to appear

• That’s when advanced science came in and gave us “distributed

computation”
• MapReduce is a representative of such technology

2
MapReduce (section 2)

• Challenges:
• How to distribute computation?
• Distributed/parallel programming is hard

• Map-reduce addresses all of the above

• Google’s computational/data manipulation model
• Elegant way to work with big data

3
Single Node Architecture (section 2.1)

CPU

{ Memory
Machine Learning, Statistics

“Classical” Data Mining

Disk

Main needs:
a storage infrastructure and a programming paradigm (for analysis)

4
Motivation: Google Example

• 20+ billion web pages x 20KB = 400+ TB

• 1 computer reads 30-35 MB/sec from disk
• ~4 months to read the web
• ~1,000 hard drives to store the web
• Takes even more to do something useful
with the data, both in terms of storing it as well as the time needed
to analyze it !!
• Today, a standard architecture for such problems is emerging:
• Cluster of commodity Linux nodes
• Commodity network (ethernet) to connect them

5
Big picture of the new paradigm

• Split the data into chunks

• Have multiple disks and CPUs
• Distribute the data chunks across multiple disks
• Process the data in parallel across CPUs

• Example:
• Time to read data: 4million secs (46+ days)
• If we had 1000 CPUs we can do the task in
• 4 million / 1000 = 4000 seconds (about 1 hour)

6
Cluster Architecture (Section 2.1.1)
2-10 Gbps backbone between racks
Switch
1 Gbps between
any pair of nodes
Each rack contains 16-64 nodes
in a rack
Switch Switch

CPU CPU CPU CPU

Mem … Mem Mem … Mem

Disk Disk Disk Disk

It has been estimated that Google has 1M machines, http://bit.ly/Shh0RO

7
8
Large-scale Computing
• Large-scale computing for data mining
problems on commodity hardware
• Challenges:
• How do you distribute computation?
• How can we make it easy to write distributed programs?
• Machines fail:
• One server may stay up 3 years (1,000 days)
• If you have 1,000 servers, expect to lose 1/day
• If Google has ~1M machines, then
• 1,000 machines fail every day!

9
Large-scale Computing (2)

• Problem #1: If nodes fail at such a rate, how do we make sure data is
stored PERSISTENTLY
• i.e., we are guaranteed to have data (or its copies) if machines that store it
fail.

• What if nodes fail during a long-running process what do we do ?

• E.g., do we restart the process all over again ?

• Solution: a new infrastructure that ”hides” all these failures

10
Large-scale Computing (3)

• Problem #2: Network Bottleneck

• If bandwidth = 1 Gbps, moving 10TB takes ~1 day
• In typical applications data moves around to be analyzed
• In our case we should be moving enormous amounts of data into thousands
of servers and that can be prohibitive

• Solution: a new framework that doesn’t move data around !

• In my opinion: the big beauty of MapReduce !!

11
Map-Reduce (Section 2.2)

• It addresses previously mentioned problems in the following ways:

• Stores data redundantly on multiple nodes (computers) for persistence and

availability

• Move computation close to the data to minimize data movement. This is a

simple yet powerful idea.

• Simple programming model that even non-savvy programmers can

implement

12
Idea and Solution
• Issue: Copying data over a network takes time
• Idea:
• Bring computation close to the data
• Store files multiple times for reliability
• Map-reduce addresses these problems
• Google’s computational/data manipulation model
• Elegant way to work with big data
• Storage Infrastructure – File system
• Google: GFS. Hadoop: HDFS
• Programming model
• Map-Reduce

13
Storage Infrastructure

• Problem:
• If nodes fail, how to store data persistently?
• Answer:
• Distributed File System (redundant storage):
• Provides global file namespace
• Implementations: Google GFS; Hadoop HDFS;
• Stores each data piece multiple times across servers
• Typical usage pattern
• Huge files (100s of GB to TB)
• Data is rarely updated in place (e.g. storing URLs)
• Reads and appends are common

14
Distributed File System

• Reliable distributed file system

• Data kept in “chunks” spread across machines
• Each chunk replicated on different machines
• Seamless recovery from disk or machine failure

C0 C1 D0 C1 C2 C5 C0 C5

C5 C2 C5 C3 D0 D1 … D0 C2
Chunk server 1 Chunk server 2 Chunk server 3 Chunk server N

Bring computation directly to the data!

Chunk servers also serve as compute servers 15
Distributed File System

• Chunk servers
• File is split into contiguous chunks
• Typically each chunk is 16-64MB
• Each chunk replicated (usually 2x or 3x)
• Try to keep replicas in different racks
• Master node
• a.k.a. Name Node in Hadoop’s HDFS
• Stores information about where files are stored
• Might be replicated
• Client library for file access
• Talks to master to find chunk servers
• Connects directly to chunk servers to access data

16
Coordination: Master

• Master node takes care of coordination:

• Task status: (idle, in-progress, completed)
• Idle tasks get scheduled as workers become available
• When a map task completes, it sends the master the location and sizes of its
R intermediate files, one for each reducer
• Master pushes this info to reducers

• Master pings workers periodically to detect failures

17
Dealing with Failures

• Map worker failure

• Map tasks completed or in-progress at
worker are reset to idle
• Reduce workers are notified when task is rescheduled on another worker
• Reduce worker failure
• Only in-progress tasks are reset to idle
• Reduce task is restarted
• Master failure
• MapReduce task is aborted and client is notified

18
Fault tolerance:
Handled via re-execution
• On worker failure:
• Detect failure via periodic heartbeats
• Re-execute completed and in-progress map tasks
• Task completion committed through master

• Robust: Google lost 1600 of 1800 machines, but kept running fine
Programming Model: MapReduce

Warm-up task:
• We have a huge text document
• Maybe 10-20TB of size

• Count the number of times each

distinct word appears in the file

• Sample application:
• Analyze web server logs to find popular URLs
• Keyword statistic for search

20
Task: Word Count

Case 1:
• File too large for memory, but all <word, count> pairs fit in memory <-
simple program !
Case 2:
• Count occurrences of words:
• words(doc.txt) | sort | uniq -c
• where words takes a file and outputs the words in it, one per a line
• Case 2 captures the essence of MapReduce
• Great thing is that it is naturally parallelizable

21
MapReduce: Overview

words(doc.txt) | sort

• Map
• Scan input file one record at a time
• Extract something you care about from each record (keys)

• Group by key
• Groups all the keys with the same value
• Sort and shuffle

• Reduce
• Run a function
• Aggregate, summarize, filter or transform
• Write the answer

22
MapReduce: The Map Step

Input Intermediate
key-value pairs key-value pairs

k v
map
k v
k v
map
k v
k v

… …

k v k v

24
MapReduce: The Reduce Step

Output
Intermediate Key-value groups key-value pairs
key-value pairs
reduce
k v k v v v k v
reduce
Group
k v k v v k v
by key

k v
… …
…

k v k v k v

25
More Specifically
• Input: a set of key-value pairs

• Programmer specifies two methods:

• Map(k, v) → <k’, v’>*
• Takes a key-value pair and outputs a set of key-value pairs
• E.g., key is the filename, value is a single line in the file
• There is one Map call for every (k,v) pair
• Reduce(k’, <v’>*) → <k’, v’’>*
• All values v’ with same key k’ are reduced together
and processed in v’ order
• There is one Reduce function call per unique key k’

• Note: * means a set

26
Map-Reduce: Environment

Map-Reduce environment takes care of:

• Partitioning the input data
• Scheduling the program’s execution across a
set of machines
• Performing the group by key step
• Handling machine failures
• Managing required inter-machine communication

27
 Map-Reduce: A diagram (centralized)
Big document
MAP:
Read input and
produces a set of
key-value pairs

Group by key:
Collect all pairs with
same key Map function
(Hash merge, Shuffle,
Chunk
Sort, Partition)

Reduce:
Collect all values
belonging to the key
and output Reduce
function
28
 Map-Reduce: In Parallel

Node

Ensures all
list with same
key are sent to
the same reduce
node
All phases are distributed with many tasks doing the work 29
 Map-Reduce
• Programmer specifies:
• Map and Reduce and input files Input 0 Input 1 Input 2

• Workflow:
• Read inputs as a set of key-value-pairs
• Map transforms input kv-pairs into a new set Map 0 Map 1 Map 2
of k'v'-pairs
• Sorts & Shuffles the k'v'-pairs to output nodes
• All k’v’-pairs with a given k’ are sent to the Shuffle
same reduce
• Reduce processes all k'v'-pairs grouped by key
into new k''v''-pairs Reduce 0 Reduce 1
• Write the resulting pairs to files

• All phases are distributed with many tasks

doing the work Out 0 Out 1

30
Data Flow

• Input and final output are stored on a distributed file system (FS):
• Scheduler tries to schedule map tasks “close” to physical storage location of
input data

• Intermediate results are stored on local FS of Map and Reduce

workers

• Output is often input to another MapReduce task

31
 MapReduce: Word Counting
Provided by the Provided by the
Key appears programmer programmer
1 time
MAP: Group by key: Reduce:
Read input and Collect all values
Collect all pairs
produces a set of belonging to the
with same key
key-value pairs key and output

data
The crew of the space

reads
shuttle Endeavor recently
(The, 1) (crew, 1)

read the
returned to Earth as (crew, 1) (crew, 1)
ambassadors, harbingers of (crew, 2)
a new era of space (of, 1) (space, 1)
(space, 1)

sequential
exploration. Scientists at (the, 1) (the, 1)
NASA are saying that the (the, 3)
(space, 1) (the, 1)

Sequentially
recent assembly of the
Dextre bot is the first step in (shuttle, 1)
a long-term space-based (shuttle, 1) (the, 1)
man/mache partnership.
(recently, 1)
(Endeavor, 1) (shuttle, 1)
'"The work we're doing now …

Only
-- the robotics we're doing - (recently, 1) (recently, 1)
- is what we're going to
need ……………………..
…. …
Big document (key, value) (key, value) (key, value)
32
Word Count Using MapReduce

map(key, value):
// key: document name; value: text of the document
for each word w in value:
emit(w, 1)

reduce(key, values):
// key: a word; values: an iterator over counts
result = 0
for each count v in values:
result += v
emit(key, result)

33
Refinement: Partition Function
• Want to control how keys get partitioned
• Inputs to map tasks are created by contiguous splits of input file
• Reduce needs to ensure that records with the same intermediate key end up
at the same worker

• System uses a default partition function:

• hash(key) mod R

• Sometimes useful to override the hash function:

• E.g., hash(hostname(URL)) mod R ensures URLs from a host
end up in the same output file

36
Selection by Map-Reduce
• Given a table R compute the operation 𝜎𝑐𝑜𝑛𝑑𝑖𝑡𝑖𝑜𝑛 𝑅 , where
condition is a logical expression. Selection chooses the tuples from R
that satisfy the condition.
• Alternatively, in SQL, the operation is
SELECT attributes
FROM R
WHERE “condition is true”

A B
𝑹 a1: b1 A B
𝜎𝐵=𝑏1 (𝑅)
a2 b1 a1: b1

a3 b2 a2 b1

a4 b3

37
Selection by Map-Reduce

• Mapper
• Emit only tuples that satisfy the selection condition

• The (key,value) pairs are of the form (t,t), where t is a tuples satisfying the
condition.
• In the previous example
((a1,b1),(a1,b1))
((a2,b1),(a2,b1))

• Reducer
• No reducer is required

38
GROUP BY and AGGREGATION by Map-Reduce
• Compute the query, Q:
SELECT A,B,SUM(C)
FROM R
GROUP BY A,B
• R(A,B,C) is stored in a file

A B C

A1 b1 1
𝛾 Α, B, SUM(C) (R) A B C

A1 b1 2 A1 b1 3

A3 b2 3 A3 b2 7

A3 b2 4

39
GROUP BY and AGGREGATION by Map-Reduce

• Mapper
• The (Key, Value) pairs are of the form:
• Key =<attributes in GROUP BY>
• Value =<attributes in AGGREGATION.
• In the previous example
((A1,B1),1)
((A1,B1),2)
((A3,B2),3)
((A3,B2),4)

• Reducer
• Applies the aggregation operation on the list of values and outputs results
• In the previous example
(A1,B1,3)
(A3,B2,7)

40
Natural Join By Map-Reduce
• Compute the natural join R(A,B) ⋈ S(B,C)
• R and S are each stored in files
• Tuples are pairs (a,b) or (b,c)

A B
B C A B C
a1 b1
b2 c1 R⋈S a3 b2 c1
a2

a3
b1

b2
⋈ b2 c2 = a3 b2 c2

b3 c3 a4 b3 c3
a4 b3
S
R

41
Map-Reduce Join
• Use a hash function h from B-values to 1...k

• A Map process turns:

• Each input tuple R(a,b) into key-value pair (b,(a,R))
• Each input tuple S(b,c) into (b,(c,S))

• Map processes send each key-value pair with key b to Reduce process h(b)
• Hadoop does this automatically; just tell it what k is.

• Each Reduce process matches all the pairs (b,(a,R)) with all (b,(c,S)) and
outputs (a,b,c).

42
Hands – on Activity
 Alex,Georgia
Hands – on Activity Apostolos,Alex
Anastasios,Apostolos
Anastasios,Alexandros
Georgia,Anastasios

• Graph data of a social network that we want to analyze.

• On each line in the file there is a pair of usernames which
represents a two-way friendship relationship between them.
• Users Alex and Georgia are friends with each other.
• Goal: find the number of friends each user of the social
network has.
• As the number of users is very large, we wish to implement
the solution using Map-Reduce and run our program on
Apache Hadoop.
Hands – on Activity: Alex,Georgia
Solution Apostolos,Alex
Anastasios,Apostolos
Anastasios,Alexandros
Georgia,Anastasios
Matrix-Vector Multiplication with Map/Reduce

A B C
1 2 0 0 20

0 3 4 X = 110

5 0 6 20 120

Task: Compute product C = A·B

Computing Sparse Matrix-Vector Product
A B
k 〈1, 1, 1, A〉 j
1 2 0 〈1, 2, 2, A〉 0
〈2, 1, 10, B〉
〈2, 2, 3, A〉
i 0 3 4 〈2, 3, 4, A〉
5 0 6 〈3, 1, 5, A〉
X k

20
〈3, 1, 20, B〉

〈3, 3, 6, A〉

● Represent matrix as list of nonzero entries 〈row, col, value, matrixID〉

● Strategy
○ Phase 1: Compute all products ai,k · bk,j
○ Phase 2: Sum products for each entry i,j
○ Each phase involves a Map/Reduce
 Phase 1: Map of Matrix-Vector Multiplication
Mapping of Initial representation
Intermediate Keys
to intermediate keys
Key = 1
〈row, col, value, matrixID〉→ (col, (matrixID, row, value))
(1, (A, 1, 1))
A 〈1, 1, 1, A〉→ (1, (A, 1, 1)) (1, (A, 3, 5))
1 2 0 〈1, 2, 2, A〉→ (2, (A, 1, 2))
〈2, 2, 3, A〉→ (2, (A, 2, 3)) Key = 2
0 3 4 〈2, 3, 4, A〉→ (3, (A, 2, 4)) (2, (A, 1, 2)) (2, (B, 1, 10))
5 0 6 〈3, 1, 5, A〉→ (1, (A, 3, 5)) (2, (A, 2, 3))
〈3, 3, 6, A〉→ (3, (A, 3, 6))
Key = col
B Key = 3
0 (3, (A, 2, 4)) (3, (B, 1, 20))
〈2, 1, 10, B〉→ (2, (B, 1, 10)) (3, (A, 3, 6))
〈3, 1, 20, B〉→ (3, (B, 1, 20))
Key = row
20

Group values ai,k and bk,j according to key k

Phase 1: Reduce of Matrix-Vector Multiplication

Key = 1
(1, (A, 1, 1))
(1, (A, 3, 5))X
A B
1 2 0 0 Key = 2
(2, (A, 1, 2)) 〈1, 1, 2x10=20, C〉
0 3 4 X X
(2, (A, 2, 3)) (2, (B, 1, 10)) 〈2, 1, 3x10=30, C〉
5 0 6 20

Key = 3
〈2, 1, 4x20=80, C〉
(3, (A, 2, 4))
(3, (A, 3, 6))X (3, (B, 1, 20)) 〈3, 1, 6x20=120, C〉

Generate all products ai,k · bk,j

 Phase 2: Map of Matrix-Vector Multiplication
Mapping of intermediate representation to intermediate keys

〈row, col, value, matrixID〉→ ((row, col), (matrixID, value))

〈1, 1, 20, C〉→ ((1, 1), (C, 20))
〈2, 1, 30, C〉→ ((2, 1), (C, 30)) Key = 1,1 ((1, 1), (C, 20))
〈2, 1, 80, C〉→ ((2, 1), (C, 80))
((2, 1), (C, 30))
〈3, 1, 120, C〉→ ((3, 1), (C, 120)) Key = 2,1 ((2, 1), (C, 80))
Key = (row,col)
Key = 3,1 ((3, 1), (C, 120))

A B
1 2 0 0

0 3 4 X
5 0 6 20

Group products ai,k · bk,j with matching values of i and j

Phase 2: Reduce of Matrix-Vector Multiplication

Key = 1,1 ((1, 1), (C, 20)) + C

20
〈1, 1, 20, C〉
((2, 1), (C, 30))
Key = 2,1 + 〈2, 1, 110, C〉 110
((2, 1), (C, 80))
〈3, 1, 120, C〉
120
Key = 3,1 ((3, 1), (C, 120)) +

A B
1 2 0 0

0 3 4 X
5 0 6 20

Sum products to get final entries

 Extensions to Map Reduce
• Workflow systems extend MapReduce from the simple two-step
workflow (the Map function feeds the Reduce function) to any
acyclic network of functions, where each function is implemented
by a collection of tasks.

• An arc a → b represents that function a’s output is an input to function b.

Examples
1. Apache Spark
2. Google’s TensorFlow.
52
Brief Overview of Apache Spark
• Open-source distributed computing system
• Built in Scala
• APIs in Java, Python, R
• Machine Learning libraries
• Graph Databases support
• SQL support
• Basic abstraction: Resilient Distributed Dataset (RDD).
• Data are split in partitions within a computing cluster
• These partitions altogether comprise an RDD
• RDDs are immutable: They cannot be altered
• But they can be transformed into other RDDs
• RDDs are resilient
• Its Lineage is maintained: If a computation fails, an RDD is recomputed from its
predecessor RDDs
• Computation methodology: Lazy Evaluation
• The programmer declares chains of “Transformations” on RDDs
• These are not executed, until an “Action” is executed.
• This way, computations are limited only to the ones required by the specified
Action

53
Brief Overview of Tensorflow
• Open-source machine learning framework developed by Google
• Widely used for deep learning applications
• Flexible architecture for deployment on CPUs, GPUs, and TPUs
• TensorFlow uses tensors; a tensor is simply a multidimensional matrix.

54