MapReduce Examples
CSE532: Theory of Database Systems
Fusheng Wang
Department of Biomedical Informatics
Department of Computer Science
Word Count Execution
Input
the
quick
brown
fox
the fox
ate the
mouse
how now
brown
cow
Execution
Map
Map
Shuffle & Sort
Reduce
Output
Reduce
brown, 2
fox, 2
how, 1
now, 1
the, 3
Reduce
ate, 1
cow, 1
mouse, 1
quick, 1
brown, 1
fox, 1
the, 1
fox, 1
the, 1
the, 1
Map
brown, 1
how, 1
now, 1
Map
quick, 1
ate, 1
mouse, 1
cow, 1
Total Order Sorting by Mapper Output Keys
The output key-value pairs from the Mapper are sorted
by keys before they reach the reducers
The sort order for keys is controlled by RawComparator
mapreduce.job.output.key.comparator.class
Keys are a subclass of WritableComparable
Or the RawComparator: compare records read from a
stream without deserializing them into objects
Partitioner (a customizable hashing function) will decide
how the keys are split into reducers
Each reducer will merge the keys from multiple reducers
and preserve the order
Across reducers, there is no total order
A single reducer will generate a total order of keys but will
be too slow
Sorting
Idea
Idea: produce a set of sorted files that, if
concatenated, would form a globally sorted file
The secret: use a partitioner that respects the total
order of the output
e.g.: sort the weather dataset by temperature
Reducer
<-10C
-10C - 0C
-0C - 10C
>= 10C
Sorting
Total Order Partioner
HashPartitioner (default) hashes a records key to
determine which partition/reducer the record belongs in
Goals of a total order partitioner:
The number of partitions equals to the number of reducers
The size of each partition should be balanced
Sampling the key space to estimate the distribution
and generate partitioning boundaries for partitioning
The ImputSampler runs on client limiting splits for sampling
The InputSampler writes a partition file to share with the tasks
running on the cluster with Distributed Cache
Distributed Cache is a facility provided by the Map-Reduce
framework to cache files (text, archives, jars etc.) needed by
applications
Sorting
Overview of Total Order Sorting
Sorting
Example Code
Sorting
Joins
Repartition joinA reduce-side join for situations
where you are joining two or more large datasets
Replication joinA map-side join that works in
situations where one of the datasets is small enough
to cache
Semi-joinAnother map-side
join where one dataset is initially
too large to fit into memory, but
after some filtering can be
reduced down to a size that can
fit in memory
Repartition Join
A repartition join is a reduce-side join implemented as
a single MapReduce job, and supports multi-way join
The map phase reads the data from multiple datasets,
determining the join value for each record, and
emitting that join value as the output key
A (key, value) B (key, value)
(key, value(value, tag) ): tag annotates the table name
The output value contains needed for combining datasets in the
reducer to produce the job output
A reducer receives all of the values for a join key
emitted by the map function, and partition them based
on data sources
The reducer performs a Cartesian product across all
partitions and emits the results of each join
Repartition Join
Repartition Join
Example
Join Customers (CID, Name,
Phone) with Orders (CID,
OrderID, Price Date):
Find orders for each customer
Mapper: same key (CID) for
both inputs; value is customer
info for Customers, order info
for Orders, PLUS a tag on
data source
Repartition Join
(CID, Name, Phone)
(CID, OrderID, Price, Date)
The Reducer Side of Repartioned Join
For a given join key,
the reduce task
performs a full crossproduct of values
from different sources
Repartition Join
Replicated Joins
Repartioned join happens late at Reducer phase, major
overhead on moving data to reducer nodes
Replicated join: join operation between one large and
many small data sets that can be performed on the map
side
Completely eliminates the need to shuffle any data to the
reduce phase
All the data sets except the very large one are essentially
read into memory during the setup phase of each map task
Join is done entirely in the map phase, with the very large
data set being the input for the MapReduce job
Restriction: a replicated join is really useful only for an
inner or a left outer join where the large data set is the
left data set
Replicated Join
Replicated Joins
Replicated Joins
// Read cached table into Hashtable
Replicated Join
References
[1] J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters.
Commun. ACM, 51(1):107113, 2008.
[2] Fuhui Wu et al. Comparison & Performance Analysis of Join Approach in MapReduce
ISCTCS 2012, CCIS 320, pp. 629636
[3] Marko Lali et al. Comparison of a Sequential & a MapReduce Approach to Joining Large
Datasets MIPRO 2013, pp.1289-1291
[4] Spyros, B., Jignesh, M.P., Vuk, E., Jun, R., Eugene, J., Yuanyuan, T.: A Comparison of
Join Algorithms for Log Processing in MapReduce. In: SIGMOD 2010, June 611. ACM,
Indianapolis (2010)
[5] Foto N. Afrati et al. Optimizing Multiway Joins in a Map-Reduce Environment IEEE
Transactions On Knowledge And Data Engineering, pp. 1282- 1298, 2011
[6] Alper Okcan et al. Processing Theta-Joins using MapReduce SIGMOD11, June 1216,
2011 pp. 949-951
[7] Xiaofei Zhang et al. Efficient Multiway Theta Join Processing Using MapReduce
Proceedings of the VLDB Endowment, Vol. 5, No. 11, pp.1184-1196
[8] Anwar Shaikh et al. Join Query Processing in MapReduce Environment CNC 2012,
LNICST , pp.275-281
Join References
