--- title: "MapReduce workflow with example" subject: "Big Data Technologies" chapter: 3 key: "semester_8/big_data_technologies:mapreduce-workflow-example" source: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-workflow-example site: BCT Engineering --- # MapReduce workflow with example Big Data Technologies · Chapter 3: Map-Reduce Framework - Asked in: 2075 Bhadra, 10 marks; 2073 Bhadra, 5+5 marks (asked 2 times) - Video: [Map Reduce explained with example | System Design](https://www.youtube.com/watch?v=cHGaQz0E7AU) — ByteMonk - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-workflow-example - Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-3 ## Question Map Reduce is the heart of Hadoop eco-system? Define work flow of Map reduce with suitable examples. ## Short answer MapReduce is the core processing engine of Hadoop. Input splits are mapped to key-value pairs, shuffled and sorted by key, then reduced to aggregates. Example: word count maps each word to (word,1) and reducers sum counts, outputting (word, total). ## Answer **MapReduce** is a programming model and framework for processing large datasets in parallel on a cluster. It is the core processing engine of Hadoop: HDFS stores the data and MapReduce processes it, moving computation to where the data is. The programmer writes only two functions, `map` and `reduce`, and the framework handles splitting, scheduling, shuffling, and fault tolerance. #### Workflow ```text Input -> Split -> Map -> Shuffle & Sort -> Reduce -> Output (HDFS) (blocks) (k,v) (group by key) (k,list) (HDFS) ``` 1. **Input and splitting:** input files in HDFS are divided into input splits (usually one block of 128 MB). A RecordReader turns each split into (key, value) pairs. 2. **Map phase:** one map task per split runs the user's `map()` on every record and emits intermediate (key, value) pairs: `map(k1, v1) -> list(k2, v2)`. Map output is written to local disk, not HDFS. 3. **Combiner (optional):** a local mini-reducer that merges map output of the same node to cut network traffic. 4. **Partition, shuffle and sort:** a partitioner (default hash(key) mod number of reducers) decides the reducer for each key. Map output is copied over the network to reducers (shuffle), and sorted and grouped by key. 5. **Reduce phase:** `reduce(k2, list(v2)) -> list(k3, v3)` is called once per key to aggregate the values. 6. **Output:** reducers write results to HDFS, one file per reducer (part-r-00000 ...). The master (JobTracker, or ResourceManager and ApplicationMaster in YARN) schedules tasks and re-executes failed tasks. #### Example 1: Word count Input: ```text line1: "big data is big" line2: "data is useful" ``` - Map output: (big,1) (data,1) (is,1) (big,1) | (data,1) (is,1) (useful,1) - After shuffle and sort: big -> [1,1]; data -> [1,1]; is -> [1,1]; useful -> [1] - Reduce output: (big,2) (data,2) (is,2) (useful,1) ```java // Mapper public void map(LongWritable key, Text value, Context ctx) { for (String w : value.toString().split("\\s+")) ctx.write(new Text(w), new IntWritable(1)); } // Reducer public void reduce(Text key, Iterable vals, Context ctx) { int sum = 0; for (IntWritable v : vals) sum += v.get(); ctx.write(key, new IntWritable(sum)); } ``` #### Example 2: Maximum temperature per year Input records: (year, temperature). ```text map: emit (year, temp) reduce: for each year, return max of the temps list ``` Map: (1950,22) (1950,34) (1951,30) ... Reduce: (1950,34) (1951,30). Here the reducer can also be used as the combiner, since max is associative. #### Why "heart" of Hadoop It provides scalable, fault-tolerant batch processing, and higher-level tools such as Hive and Pig compile their queries into MapReduce jobs (or, in newer versions, Tez or Spark jobs on YARN). ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. Where is intermediate map output written? A. HDFS with 3 replicas B. Local disk of the mapper node C. The reducer's memory only D. The client machine Answer: B. Map output goes to local disk, not HDFS, avoiding replication cost. (Key point: Map output location) 2. How does the default partitioner decide which reducer gets a key? A. Randomly for each value B. By the order of arrival C. By the length of the key D. hash(key) mod number of reducers Answer: D. hash(key) mod R sends all values of one key to the same reducer. (Key point: Partitioner) 3. How often is reduce(k2, list(v2)) called? A. Once per key, with the list of its values B. Once per input record C. Once per mapper D. Once per HDFS block Answer: A. After shuffle and sort, reduce is called once per distinct key with all its grouped values. (Key point: Reduce call) 4. For lines 'big data is big' and 'data is useful', what is the reducer output? A. (big,1) (data,1) (is,1) (useful,1) B. (big,2) (data,1) (is,2) (useful,1) C. (big,2) (data,2) (is,2) (useful,1) D. (big,2) (data,2) (is,1) (useful,1) Answer: C. big appears 2 times, data 2, is 2, useful 1. (Key point: Word count result) ## Related in Big Data Technologies - [MapReduce data flow technique](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-data-flow) — Big Data Technologies, chapter 3 - [MapReduce worker failure tolerance](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-fault-tolerance) — Big Data Technologies, chapter 3 - [Role of HDFS in Hadoop](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Ahdfs-role-in-hadoop) — Big Data Technologies, chapter 6 - [Optimization and data locality in MapReduce](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-optimization-data-locality) — Big Data Technologies, chapter 3 - [MapReduce max marks per college](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Amapreduce-max-marks-college) — Big Data Technologies, chapter 3 ## Related in other subjects - [MapReduce vs Hadoop; IR process](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Amapreduce-vs-hadoop-ir) — Information System, chapter 8 - [MapReduce vs Hadoop](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Amapreduce-vs-hadoop) — Information System, chapter 8 - [Processor is heart of embedded system](https://www.bctengineer.com/question.md?key=semester_6%2Fembedded_system%3Aprocessor-heart-structural-units) — Embedded System, chapter 2 - [Distributed system and Hadoop](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Adistributed-system-hadoop-voluminous-data) — Information System, chapter 8