MapReduce with GFS and HDFS

Why MapReduce is the de facto standard for scalable processing of big data

MapReduce is strictly a processing model, not a file system; it became the standard for scalable big data processing on top of distributed file systems for these reasons:

• Simple model: the programmer writes only map and reduce; parallelism, distribution and failure handling are automatic.
• Scalability: runs on hundreds or thousands of cheap commodity machines; adding nodes increases speed almost linearly.
• Data locality: moves computation to the node holding the data, saving network traffic.
• Fault tolerance: failed tasks are re-executed; data is replicated.
• Wide applicability: indexing, log analysis, sorting, graph and machine learning jobs.
• Open-source implementation: Hadoop made it available to everyone, and many tools (Hive, Pig) are built on it.
• Proven by Google on web-scale data, at low cost.

Association with GFS and HDFS

• GFS (Google File System) is Google's distributed file system. It stores huge files as 64 MB chunks, replicated (3 copies) on chunk servers, with a single master holding metadata. MapReduce at Google reads its input from and writes its output to GFS.
• HDFS (Hadoop Distributed File System) is the open-source equivalent. It uses a NameNode (metadata) and DataNodes (blocks of 128 MB, replicated 3 times). Hadoop MapReduce uses HDFS in the same way.
• Close link: the file system provides large, replicated, sequential-access storage and tells MapReduce where blocks are, so that the scheduler places map tasks near the data. MapReduce provides the processing layer on top.

 MapReduce (processing)
        |
 GFS / HDFS (storage, replication, locality)
        |
 Commodity machines