--- title: "Faults, failures, errors; resilience" subject: "Distributed System" chapter: 9 key: "semester_7/distributed_system:faults-failures-resilience" source: https://www.bctengineer.com/q/semester_7/distributed_system/faults-failures-resilience site: BCT Engineering --- # Faults, failures, errors; resilience Distributed System · Chapter 9: Fault Tolerance - Asked in: 2069 Chaitra, 2+2+4 marks - Video: [Distributed Systems 2.4: Fault tolerance](https://www.youtube.com/watch?v=43TDfUNsM3E) — Martin Kleppmann - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_7/distributed_system/faults-failures-resilience - Whole chapter: https://www.bctengineer.com/md/semester_7/distributed_system/chapter-9 ## Question What do you mean by faults, failures and errors? How do you handle faults in Distributed System? Explain process resilience approach in brief. ## Short answer A fault is a defect, an error is an incorrect state caused by a fault, and a failure is a service deviation due to an error. Handling faults uses avoidance, detection, masking through redundancy, and recovery. Process resilience uses replicated groups (flat or hierarchical) to mask failures; K+1 members survive K crashes, 2K+1 survive K Byzantine failures. ## Answer #### Fault, error, failure - **Fault**: the cause of a problem, a defect in hardware, software or a link (e.g. a bug, a broken wire). - **Error**: the part of the system state that is wrong because of a fault (e.g. a wrong value in a variable). - **Failure**: the system does not deliver its specified service because of an error; this is visible to the user. A fault causes an error, which may lead to a failure. Example: a software bug (fault) produces a wrong total (error), so the printed report is wrong (failure). #### Handling faults - **Fault avoidance/prevention**: good design and testing. - **Fault detection**: heartbeats, timeouts, checksums. - **Fault masking** by redundancy: replication, voting such as TMR, retransmission. - **Recovery**: backward recovery (checkpoints, logs) and forward recovery (error-correcting codes). - **Reliable communication** and atomic commit protocols. #### Process resilience **Process resilience** means protecting a system against process failures by replicating a process into a **process group** that works as a single logical process: if a member fails, another takes over, so the service continues. **Group organization** - **Flat group**: all processes are equal and decide collectively (e.g. by voting). There is no single point of failure, but decision making is more complex and has more overhead. - **Hierarchical group**: one coordinator decides and the workers carry out its instructions. It is simple and fast, but if the coordinator fails, the group must elect a new one. ```text Flat group Hierarchical group P1--P2 Coordinator \ / / | \ P3--P4 W1 W2 W3 ``` **Failure masking and group size**: a group that can survive K failures is K-fault tolerant. With crash failures, K+1 members are enough; with arbitrary (Byzantine) failures, 2K+1 members are needed (with voting), and 3K+1 to reach agreement. **Agreement in faulty systems**: members must agree on decisions even when some members are faulty (consensus, Byzantine agreement). Membership changes need a **group membership** service that adds and removes members and detects failed ones (by pinging and timeouts). Example: a replicated database group of 3 servers answers client requests; if one crashes, the other two keep serving. ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. In the software bug example, the wrong total is the: A. Fault B. Error C. Failure D. Recovery Answer: B. The bug is the fault, the wrong value is the error, and the wrong printed report is the failure. (Key point: Fault chain) 2. Which is a fault masking technique? A. Heartbeats B. Replication with voting such as TMR C. Good design and testing D. Checksums Answer: B. Masking hides faults by redundancy; heartbeats and checksums are detection. (Key point: Fault masking) 3. An advantage of a flat process group is: A. Simple fast decisions B. No single point of failure C. Needs few messages D. Has a coordinator Answer: B. All members are equal, though decision making is more complex. (Key point: Flat group) ## Related in Distributed System - [Object replication; process resilience](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Aobject-replication-resilience) — Distributed System, chapter 7 - [Short note: Process resilience](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Aprocess-resilience-note) — Distributed System, chapter 9, asked 4× - [Avoiding faults; checkpointing compared](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Aavoid-faults-checkpointing) — Distributed System, chapter 9 - [Autonomous vs distributed system](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Aautonomous-vs-distributed) — Distributed System, chapter 1 - [Models of distributed system](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Amodels-of-ds) — Distributed System, chapter 1 - [True distributed system; middleware organisation](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Atrue-ds-middleware) — Distributed System, chapter 3 ## Related in other subjects - [Define resilience in brief](https://www.bctengineer.com/question.md?key=semester_3%2Fcivil_engineering_materials%3Aproperty-resilience) — Civil Engineering Materials, chapter 1, asked 3× - [Data analytics process and distributed computing](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adata-analytics-distributed-role) — Big Data Technologies, chapter 1 - [Distributed system in big data example](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adistributed-system-example) — Big Data Technologies, chapter 1 - [Short note: Distributed system implementation](https://www.bctengineer.com/question.md?key=semester_6%2Fobject_oriented_analysis_and_design%3Ashort-note-distributed-implementation-issues) — Object Oriented Analysis and Design, chapter 4