Role of HDFS in Hadoop

HDFS (Hadoop Distributed File System) is the storage layer of Hadoop. It stores very large files across many commodity machines and provides the data to MapReduce and other processing engines. It is modelled on the Google File System.

Roles of HDFS in Hadoop

1. Distributed storage of big data: a large file is split into blocks (128 MB default) that are spread across many DataNodes, so storage can grow to petabytes by adding nodes.
2. Fault tolerance: each block is replicated (default 3 copies) on different nodes and racks. If a node fails, the NameNode re-replicates the lost blocks from the surviving copies, so no data is lost.
3. Data locality for processing: HDFS tells the scheduler where blocks are stored, so MapReduce tasks run on the node (or rack) that holds the data. This moves computation to data instead of moving data, and saves network bandwidth.
4. High-throughput streaming access: it is optimised for write-once, read-many access to large files, rather than low-latency random access.
5. Stores job input and output: MapReduce reads input from HDFS and writes final output back to it.
6. Runs on commodity hardware: cheap machines are enough, since reliability comes from software replication.

Architecture

 Client --> NameNode (metadata)
   |            |
   +--data----> DataNode1 DataNode2 DataNode3
               (blocks replicated across nodes)

The NameNode (master) stores the namespace and block locations, and DataNodes (slaves) store the blocks.