Fault tolerance techniques

Fault tolerance is the ability of a system to continue to provide correct service even when faults occur: faults are detected, masked, or recovered from, so failures are not visible to the user.

How it is ensured: by redundancy, together with detecting faults and recovering from them. Redundancy comes in three kinds:

• Information redundancy: extra bits, e.g. parity bits, Hamming codes and checksums, to detect and correct errors.
• Time redundancy: an action is repeated if needed, e.g. retransmission and transaction retry; used for transient and intermittent faults.
• Physical redundancy: extra hardware or software copies, e.g. triple modular redundancy, replicated servers and process groups.

Techniques

1. Failure detection: heartbeats, pinging and timeouts.
2. Failure masking by redundancy: process groups with replication; voting (TMR).
3. Recovery: backward recovery with checkpointing and message logging; forward recovery with error-correcting codes.
4. Reliable communication: retransmission, sequence numbers, reliable multicast.
5. Distributed commit: 2PC/3PC to keep all-or-nothing semantics in the presence of failures.
6. Agreement protocols: Byzantine agreement for arbitrary failures.
7. Avoiding faults: careful design, testing, using reliable components, and fault prevention.