DFS definition and HDFS client write

Distributed File System (DFS)

A DFS is a file system whose files are stored across several networked machines but accessed by users and programs as though they were on one local file system. It gives location transparency, replication, scalability and fault tolerance. Examples: NFS, AFS, GFS and HDFS.

HDFS client write process

HDFS has a NameNode (master, holds metadata) and many DataNodes (store blocks, default block size 128 MB, replication factor 3). The client writes as follows.

1. The client calls create() on the DistributedFileSystem object. It contacts the NameNode, which checks permissions and that the file does not already exist, and records the new file.
2. The client receives an FSDataOutputStream. As the client writes, data is split into packets and put in an internal data queue.
3. For each block, the NameNode selects a list of DataNodes (a pipeline) to hold the three replicas, using rack awareness (first replica on the client's node/rack, second on a different rack, third on another node of that second rack).
4. The client sends packets to the first DataNode, which stores them and forwards them to the second, which forwards them to the third.
5. Each DataNode sends an acknowledgement back along the pipeline. Packets move from the data queue to an ack queue until acknowledged by all.
6. When a block is full, the client asks the NameNode for the next set of DataNodes and repeats.
7. When writing finishes, the client calls close(), flushes the remaining packets, waits for acknowledgements, and tells the NameNode the file is complete.

 +--------+ 1 create  +----------+
 | Client |---------->| NameNode |
 | (JVM)  |<----------| metadata |
 +---+----+ 2 DN list +----------+
     | 3 write packets
     v
 +--------+  4 forward  +--------+  forward  +--------+
 |DataNode|------------>|DataNode|---------->|DataNode|
 |   1    |<------------|   2    |<----------|   3    |
 +--------+  5 ack      +--------+    ack    +--------+
     6 close / complete --> NameNode

If a DataNode fails during writing, the pipeline is closed, the failed node is removed, the remaining packets are put back on the data queue, and the NameNode later re-replicates the block to maintain the replication factor.

Features of a DFS

• Transparency: location, access and replication are hidden from the user.
• Scalability: more storage by adding nodes.
• Fault tolerance: replicas keep data available if a node fails.
• Concurrent access by many clients.

Rack awareness in write

The default placement is: replica 1 on the writer's node (or a random node if the client is outside the cluster), replica 2 on a node in a different rack, replica 3 on another node in the same rack as replica 2. This balances reliability against cross-rack bandwidth.

Key points

• Only the metadata passes through the NameNode; data goes straight from the client to DataNodes.
• A block is considered written when the minimum number of replicas (dfs.namenode.replication.min, default 1) have acknowledged it; remaining replicas are created asynchronously if needed.
• HDFS files are write-once: only append is allowed, no random update.