--- title: "MapReduce worker failure tolerance" subject: "Big Data Technologies" chapter: 3 key: "semester_8/big_data_technologies:mapreduce-fault-tolerance" source: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-fault-tolerance site: BCT Engineering --- # MapReduce worker failure tolerance Big Data Technologies · Chapter 3: Map-Reduce Framework - Asked in: 2073 Magh, 10 marks - Video: [MapReduce - Google's secret Sauce](https://www.youtube.com/watch?v=2joXRJNZbEg) — Eksplain - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-fault-tolerance - Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-3 ## Question How is MapReduce library designed to tolerate different machines (map/reduce nodes) failure while executing MapReduce job? ## Short answer The master detects worker failures via heartbeat misses and re-executes lost map and in-progress reduce tasks. Completed reducers are safe in HDFS. Speculative execution handles stragglers, bad records are skipped, and atomic renaming of output ensures determinism. ## Answer MapReduce is designed to run on large clusters of cheap commodity machines, where failures are normal. The library handles failures automatically by **re-execution**: the master detects the failure and re-schedules the lost work on another healthy worker. The user program does not have to deal with it. #### Master (JobTracker / Application Master) keeps the state The master stores, for every map and reduce task, its state (idle, in-progress, completed) and the identity of the worker machine running it. For each completed map task it also stores the location and size of the R intermediate files. #### Worker (map/reduce node) failure The master pings every worker periodically (heartbeat). If a worker does not respond within a certain time, the master marks it as **failed**. - **Completed map tasks on the failed worker are re-executed.** Their output is stored on the local disk of that machine, so it is no longer reachable after the failure. The tasks are reset to *idle* and given to other workers. - **In-progress map or reduce tasks on the failed worker** are also reset to idle and rescheduled. - **Completed reduce tasks are not re-executed**, because their output is already stored in the global file system (GFS/HDFS), which is replicated. - When a map task is re-executed on another worker, all reducers are notified, so that a reducer which has not yet read the data from the old worker will read it from the new one. ```text Master --ping--> Worker A (no reply) | +-- mark A failed +-- map tasks of A (done/in-progress) -> idle +-- reduce tasks of A (in-progress) -> idle +-- reschedule on Worker B, C +-- inform reducers of new map locations ``` #### Master failure There is only one master, so its failure is unlikely. The master writes periodic **checkpoints** of its data structures. If it dies, a new copy can be started from the last checkpoint. Otherwise the job is aborted and the client may retry it. (In Hadoop 1, JobTracker failure killed running jobs; YARN and HA ResourceManager improve this.) #### Semantics in the presence of failures If the user's map and reduce functions are **deterministic**, the output is the same as that of a fault-free sequential execution. This is guaranteed by **atomic commits** of task output: - Each map task writes R temporary files and sends their names to the master when finished. The master ignores a duplicate completion message for a map task that is already completed. - Each reduce task writes to a temporary output file and atomically renames it to the final file name. If the same reduce task runs on two machines, the atomic rename ensures only one final file exists. #### Slow workers (stragglers) A machine that runs unusually slowly (bad disk, CPU contention) lengthens the total job time. Near the end of the job the master starts **backup executions** of the remaining in-progress tasks on other machines. The task is marked complete when either the primary or the backup finishes. This is called **speculative execution**. #### Bad records If a particular input record always crashes the user code, the worker reports it to the master. After the master has seen more than one failure for the same record, it tells the next re-execution to skip that record. #### Summary | Failure | Action | |---|---| | Worker fails (map done) | Re-execute map task | | Worker fails (task in progress) | Reschedule the task | | Worker fails (reduce done) | No action, output is in GFS/HDFS | | Master fails | Restart from checkpoint / abort job | | Slow worker | Backup (speculative) execution | | Bad record | Skip on re-execution | ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. How does the master detect a failed worker? A. Periodic pings (heartbeats) with no response within a timeout B. Workers send an apology message C. The user reports it D. HDFS reports it Answer: A. The master pings workers periodically and marks one failed if it does not respond in time. (Key point: Failure detection) 2. Why are completed map tasks on a failed worker re-executed? A. Their results were wrong B. Their output is on the failed machine's local disk and is unreachable C. Maps always run twice D. The reducers requested it Answer: B. Map output is on local disk of the failed machine, so it is lost and the tasks are reset to idle. (Key point: Completed map tasks re-executed) 3. Why are completed reduce tasks on a failed worker not re-executed? A. Reducers never fail B. Their output is stored in the replicated global file system (GFS/HDFS) C. They are cached in master memory D. They produce no output Answer: B. Reduce output is written to GFS/HDFS, which is replicated, so it survives the worker failure. (Key point: Completed reduce tasks) 4. How is only one final output file ensured if a reduce task runs on two machines? A. The master kills one at random B. Both write the same file at once C. Each writes a temporary file and atomically renames it to the final name D. Output is merged by hand Answer: C. The atomic rename operation guarantees a single final file even with duplicate execution. (Key point: Atomic rename) ## Related in Big Data Technologies - [MapReduce workflow with example](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-workflow-example) — Big Data Technologies, chapter 3, asked 2× - [MapReduce data flow technique](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-data-flow) — Big Data Technologies, chapter 3 - [Optimization and data locality in MapReduce](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-optimization-data-locality) — Big Data Technologies, chapter 3 - [MapReduce max marks per college](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-max-marks-college) — Big Data Technologies, chapter 3 ## Related in other subjects - [Fault tolerance; Byzantine failure](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Abyzantine-failure-agreement) — Distributed System, chapter 9 - [MapReduce vs Hadoop; IR process](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Amapreduce-vs-hadoop-ir) — Information System, chapter 8 - [MapReduce working with example](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Amapreduce-working-big-data) — Information System, chapter 8 - [MapReduce with GFS and HDFS](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Amapreduce-gfs-hdfs) — Information System, chapter 8