Optimization and data locality in MapReduce

Data locality

Data locality means moving the computation to the data instead of moving the data to the computation. In a big data cluster the network bandwidth is scarce, while moving a small program is cheap. So the MapReduce scheduler tries to run each map task on the node that stores the input block.

Levels of locality, in order of preference:

1. Data-local: the map task runs on the same node as its block.
2. Rack-local: the same rack, but a different node (data passes via the rack switch).
3. Off-rack: a different rack (most expensive).

HDFS replicates each block on 3 nodes, which gives the scheduler more choices for local execution.

Optimization in MapReduce

Techniques used to make jobs faster and reduce network and disk use:

• Combiner: local aggregation of map output to reduce shuffle data.
• Data locality scheduling: as above.
• Speculative execution: a slow-running task (straggler) is run again on another node and the first to finish is used.
• Compression of map output and final output to cut I/O and network traffic.
• Proper number of mappers/reducers and block size for balance; avoid many small files (use CombineFileInputFormat).
• Custom partitioner to avoid skew so that reducers get equal loads.
• In-memory buffer tuning (e.g. io.sort.mb) to reduce spills to disk.
• JVM reuse to reduce task start-up cost.