Scaling MapReduce
Wednesday, February 24, 2010
9:58 AM
Scaling MapReduce
Mapper execution time is a function of local data size.
Split a mapper into two or more if its data becomes too large.
Join mappers if their data becomes small.
Reduce execution time is a function of the number of levels of reduce. 
Maintain at least log M reducers for M mappers. 
Add reducers as you add mappers; remove reducers as you remove mappers. 
The programmer sees none of this!   