---
title: "Combiner function in MapReduce"
subject: "Big Data Technologies"
chapter: 3
key: "semester_8/big_data_technologies:mapreduce-combiner-function"
source: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-combiner-function
site: BCT Engineering
---

# Combiner function in MapReduce

Big Data Technologies · Chapter 3: Map-Reduce Framework

- Asked in: 2073 Magh, 10 marks
- Video: [Combiner in Hadoop](https://www.youtube.com/watch?v=Dd1JbnwNotI) — ARUN KUMAR
- Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14
- Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/mapreduce-combiner-function
- Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-3

## Question

What is the combiner function in mapreduce? Explain its purpose with suitable example.

## Short answer

A combiner is an optional function that runs on map output locally, performing partial aggregation before shuffle to reduce network traffic. It must be commutative/associative (e.g., summing counts), and can reuse the reducer class; in word count, local sums like (big,2) reduce shuffled records from 6 to 4.

## Answer

A **combiner** is an optional "mini-reducer" that runs on the output of each map task, on the same node, before the data is sent over the network to the reducers. It performs a local aggregation of the map output with the same key, so less data is shuffled.

#### Purpose

- **Reduces network traffic.** Map output is written to local disk and transferred to reducers. Combining it first sends fewer records across the cluster, where bandwidth is the scarce resource.
- **Reduces disk I/O** for spilled map output and the sort/merge work in the reducer.
- **Speeds up the job**, especially when keys repeat many times in a map's output (e.g. word count).

#### Properties

- It has the same signature as a reducer: input `(key, list of values)`, output `(key, value)` with the same types as map output.
- Hadoop may run it zero, one or many times, so the result of the job must **not depend** on how many times it runs.
- It is valid only for operations that are **commutative and associative** (sum, count, max, min). Average cannot be used directly (the average of averages is wrong); instead emit `(sum, count)` pairs.
- It is set in the driver: `job.setCombinerClass(MyReducer.class);`

#### Example: word count

Input split across two mappers:

- Mapper 1 text: "big data big"
- Mapper 2 text: "data big data"

Without combiner, mapper 1 sends `(big,1) (data,1) (big,1)` and mapper 2 sends `(data,1) (big,1) (data,1)`: 6 records.

With combiner, each mapper first sums locally:

- Mapper 1 sends `(big,2) (data,1)`
- Mapper 2 sends `(big,1) (data,2)`

Only 4 records cross the network. The reducer then gets `big -> [2,1]`, `data -> [1,2]` and outputs `(big,3)`, `(data,3)`.

```text
Map1: (big,1)(data,1)(big,1) -> Combiner -> (big,2)(data,1)
Map2: (data,1)(big,1)(data,1)-> Combiner -> (big,1)(data,2)
                    \  shuffle  /
              Reducer: (big,3) (data,3)
```

#### Code sketch

```java
public class WordCount {
  public static class TokenMapper
      extends Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    public void map(LongWritable k, Text v, Context c)
        throws IOException, InterruptedException {
      for (String w : v.toString().split("\\s+")) {
        word.set(w);
        c.write(word, one);
      }
    }
  }
  public static class SumReducer
      extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text k, Iterable<IntWritable> vals, Context c)
        throws IOException, InterruptedException {
      int sum = 0;
      for (IntWritable v : vals) sum += v.get();
      c.write(k, new IntWritable(sum));
    }
  }
  // in driver:
  // job.setCombinerClass(SumReducer.class);
  // job.setReducerClass(SumReducer.class);
}
```

Here the reducer class is reused as the combiner because summing is associative and commutative.

## Check yourself

The answer's key points as multiple-choice questions, with the correct option and why.

1. What is the main purpose of a combiner?
   A. Replace the reducer entirely
   B. Store output in HDFS
   C. Locally aggregate map output to reduce data shuffled over the network
   D. Split the input
   Answer: C. It acts as a mini-reducer on each mapper node so less data crosses the network. (Key point: Combiner purpose)

2. For which operation is a combiner NOT valid directly?
   A. Sum
   B. Max
   C. Count
   D. Average
   Answer: D. The average of averages is wrong; instead emit (sum, count) pairs. (Key point: Validity condition)

3. Why must the job result not depend on how many times the combiner runs?
   A. It always runs exactly once
   B. Hadoop may run it zero, one or many times
   C. It runs only on reducers
   D. It is deleted after use
   Answer: B. The framework is free to apply the combiner any number of times, including none. (Key point: Run count)

4. In the example, mapper 1 has 'big data big'. What does it send after combining?
   A. (big,2) (data,1)
   B. (big,1) (data,1) (big,1)
   C. (big,3)
   D. (big,2) (data,2)
   Answer: A. Locally big occurs twice and data once, so 2 records are sent instead of 3. (Key point: Word count example)

## Related in other subjects

- [Fibonacci single-purpose processor](https://www.bctengineer.com/question.md?key=semester_6%2Fembedded_system%3Aspp-fibonacci) — Embedded System, chapter 2, asked 2×
- [Single-purpose processor for power](https://www.bctengineer.com/question.md?key=semester_6%2Fembedded_system%3Aspp-power) — Embedded System, chapter 2, asked 2×
- [Factorial single-purpose processor](https://www.bctengineer.com/question.md?key=semester_6%2Fembedded_system%3Aspp-factorial) — Embedded System, chapter 2
- [Purpose of valves; cutoff valve](https://www.bctengineer.com/question.md?key=semester_5%2Fwater_supply_engineering%3Avalves-purpose-cutoff) — Water Supply Engineering, chapter 9