Availability and fault tolerance in GFS

Because GFS runs on thousands of commodity machines, component failure is the normal case. It therefore builds high availability (the system keeps serving requests) and fault tolerance (it keeps working correctly despite failures) into its design.

Availability

The system should remain usable even while parts of it fail or are being upgraded.

• Fast recovery: master and chunkservers are designed to restore their state and restart in seconds, whatever the cause of shutdown. Clients retry.
• Chunk replication: every chunk is stored on several chunkservers (three by default, configurable per file) on different racks. If one fails, reads go to another replica.
• Master replication: the operation log and checkpoints are replicated on multiple machines. If the master fails, monitoring infrastructure starts a new master elsewhere. Shadow masters provide read-only access during master downtime.

Fault tolerance

• Re-replication: the master tracks the replica count through HeartBeat messages. When a chunkserver dies or a replica is corrupted, it copies the chunk to other servers to restore the replication level.
• Chunk version numbers detect stale replicas that missed mutations while a server was down; stale replicas are garbage collected.
• Data integrity by checksums: each 64 KB block of a chunk has a 32-bit checksum, verified on reads. A corrupted replica is replaced from a good one.
• Operation log + checkpoints: metadata changes are logged durably before being visible to clients, so the master state can be recovered.
• Diagnostic logging helps in problem isolation and debugging.

Example

If a chunkserver holding chunk C crashes, the master notices missing heartbeats, sees C has only 2 replicas, and instructs another chunkserver to copy it from a surviving replica. Clients meanwhile read from the remaining replicas without interruption.