Hadoop daemons explained

Daemons are background Java processes (JVMs) that run on the machines of a Hadoop cluster. Hadoop 1 has five main daemons: three for HDFS (storage) and two for MapReduce (processing). They follow a master/slave design.

        MASTER                         SLAVES
 NameNode          <------>   DataNode (many)
 Secondary NameNode
 JobTracker        <------>   TaskTracker (many)

1. NameNode (HDFS master)

• Stores the metadata of HDFS: the directory tree, file names, permissions, the blocks that make up each file, and the DataNodes that hold each block.
• Keeps the metadata in RAM and persists it as fsimage (a snapshot) plus edits (a log of changes).
• Receives heartbeats and block reports from the DataNodes, and directs re-replication when a node fails.
• It is a single point of failure in Hadoop 1 (fixed by the HA NameNode in Hadoop 2).

2. DataNode (HDFS slave)

• Stores the actual data blocks (128 MB by default in Hadoop 2; 64 MB in Hadoop 1) on its local disks.
• Serves read and write requests from clients.
• Sends a heartbeat every 3 seconds and periodic block reports to the NameNode; creates, deletes and replicates blocks as instructed.

3. Secondary NameNode

• It is not a hot standby. It periodically downloads fsimage and edits from the NameNode, merges them into a new fsimage (checkpointing) and sends it back.
• This keeps the edits log small and makes NameNode restarts fast.

4. JobTracker (MapReduce master)

• Accepts jobs from clients, splits them into map and reduce tasks, and schedules the tasks on TaskTrackers close to the data (data locality).
• Monitors task progress and re-schedules failed tasks. There is one per cluster.

5. TaskTracker (MapReduce slave)

• Runs on each slave node and executes map and reduce tasks in its task slots, as directed by the JobTracker.
• Sends heartbeats and progress reports to the JobTracker.

Hadoop 2 (YARN)

JobTracker and TaskTracker are replaced by the ResourceManager (cluster-wide resource scheduling), the NodeManager (per-node container management) and the ApplicationMaster (one per job).

Summary