--- title: "Distributed system in big data example" subject: "Big Data Technologies" chapter: 1 key: "semester_8/big_data_technologies:distributed-system-example" source: https://www.bctengineer.com/q/semester_8/big_data_technologies/distributed-system-example site: BCT Engineering --- # Distributed system in big data example Big Data Technologies · Chapter 1: Introduction to Big Data - Asked in: 2074 Bhadra, 8 marks - Video: [Explaining Distributed Systems Like I'm 5](https://www.youtube.com/watch?v=CESKgdNiKJw) — HashiCorp, an IBM Company - Answer checked by Claude Opus 5.5 (high effort) on 2026-10-14 - Page: https://www.bctengineer.com/q/semester_8/big_data_technologies/distributed-system-example - Whole chapter: https://www.bctengineer.com/md/semester_8/big_data_technologies/chapter-1 ## Question Explain with example about the distributed system in Big Data. ## Short answer A Hadoop cluster stores a 1 TB weblog in 28 MB HDFS blocks replicate acros DataNodes. MapReduce map tasks run locally on each block; reducers combine counts. Node failures trigger re-eecution from replicas, demonstrating scalable, fault-tolerant distributed storage and parallel processing for big data. ## Answer A **distributed system** is a collection of independent computers (nodes) connected by a network that appears to the user as a single system. In big data, data and processing are spread over many nodes so that very large datasets can be stored and analysed in parallel. #### Key characteristics - Many nodes, no shared memory, communication by messages - Scalability by adding nodes - Fault tolerance through replication - Transparency (location, replication, failure hidden from the user) - Concurrency of many tasks #### Common architecture in big data ```text +----------------+ | Master node | (NameNode / JobTracker) +--------+-------+ +--------+--------+ | | | +----v--+ +---v---+ +--v----+ |Worker1| |Worker2| |Worker3| | data | | data | | data | | + CPU | | + CPU | | + CPU | +-------+ +-------+ +-------+ ``` The master keeps metadata and schedules work. Workers store data blocks and compute locally. #### Example 1: Hadoop cluster (word count over logs) A 1 TB web log is uploaded to HDFS. It is split into 128 MB blocks and each block is stored on three different DataNodes. A MapReduce job sends the map code to the nodes that already hold the blocks. Each mapper counts words in its block, and reducers combine the counts. If one node fails, the block is read from another replica and the task is re-run. #### Example 2: Google search Web pages are stored in GFS over thousands of machines and indexed by MapReduce. A search query is sent to many index servers in parallel, and the results are merged and returned within a fraction of a second. #### Other examples Facebook using HDFS for its data warehouse and HBase for messaging, Amazon S3 and DynamoDB, and Cassandra clusters for NoSQL. #### Advantages Handles petabytes, low-cost hardware, high availability, and fast parallel processing. #### Challenges Network latency, node failures, consistency of replicas, clock synchronisation and security. #### Components in an HDFS-based system - **NameNode (master):** keeps the file tree, block locations and replication information. - **DataNodes (workers):** store block replicas and send heartbeats and block reports. - **Client:** reads and writes by asking the NameNode for locations and then talking to DataNodes. - **Processing layer:** YARN/MapReduce schedules tasks near the blocks. #### Why distribute big data | Need | How a distributed system helps | |---|---| | Storage of PB data | Many disks combined into one namespace | | Speed | Parallel reading and processing | | Failures | Replication and task re-execution | | Cost | Commodity hardware | #### Design issues Transparency (users should not see node locations), consistency of replicas, scheduling and load balancing, security, and handling of partial failure. ## Check yourself The answer's key points as multiple-choice questions, with the correct option and why. 1. What best describes a distributed system? A. A single powerful computer with shared memory B. Independent computers connected by a network that appear to the user as one system C. A backup tape library D. A database with one table Answer: B. A distributed system is a set of independent nodes communicating by messages and presented to the user as a single system. (Key point: Definition of distributed system) 2. In the Hadoop word count example, how is a 1 TB log stored? A. On one DataNode as a single file B. Only in the NameNode C. Split into 128 MB blocks, each stored on three different DataNodes D. Split into 3 equal parts with no replicas Answer: C. HDFS splits the file into 128 MB blocks and keeps three replicas of each on different DataNodes. (Key point: Hadoop write and replication) 3. How does the MapReduce job process the blocks in the example? A. Map code is sent to the nodes already holding the blocks, and reducers combine the counts B. All blocks are first copied to the master C. Reducers read raw blocks and mappers combine counts D. A single node counts all words Answer: A. Mappers run where the blocks are and count words locally, and reducers combine the partial counts. (Key point: Code moves to data) 4. In a typical master-worker big data architecture, what does the master do? A. Stores all the data blocks B. Computes every map task itself C. Keeps metadata and schedules work D. Only sends heartbeats Answer: C. The master holds metadata and schedules work, while workers store blocks and compute locally. (Key point: Master and worker roles) ## Related in Big Data Technologies - [Why distributed computing for big data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adistributed-computing-necessary) — Big Data Technologies, chapter 1 - [Data analytics process and distributed computing](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Adata-analytics-distributed-role) — Big Data Technologies, chapter 1 - [Role of data scientist in big data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Arole-of-data-scientist) — Big Data Technologies, chapter 1, asked 2× - [Big data trends, challenges, characteristics](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Abig-data-trends-challenges) — Big Data Technologies, chapter 1 - [Hybrid data in big data warehouse](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Ahybrid-data-warehouse) — Big Data Technologies, chapter 1 - [Attributes of Big Data](https://www.bctengineer.com/question.md?key=semester_8%2Fbig_data_technologies%3Abig-data-attributes) — Big Data Technologies, chapter 1 ## Related in other subjects - [Big data and its processing](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Abig-data-processing) — Information System, chapter 8 - [Distributed system and Hadoop](https://www.bctengineer.com/question.md?key=semester_8%2Finformation_system%3Adistributed-system-hadoop-voluminous-data) — Information System, chapter 8 - [Autonomous vs distributed system](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Aautonomous-vs-distributed) — Distributed System, chapter 1 - [Models of distributed system](https://www.bctengineer.com/question.md?key=semester_7%2Fdistributed_system%3Amodels-of-ds) — Distributed System, chapter 1