Distributed system and Hadoop

Distributed system

A distributed system is a collection of independent computers connected by a network that work together and appear to the user as a single coherent system. They share resources and coordinate by passing messages. Features: resource sharing, scalability, concurrency, fault tolerance and transparency. Examples: the web, cloud platforms, Hadoop clusters.

Hadoop

Apache Hadoop is an open-source framework for storing and processing very large data on clusters of commodity hardware.

• HDFS: distributed storage; data split into blocks, replicated across nodes.
• YARN: resource management and job scheduling.
• MapReduce: parallel processing model.
• Hadoop Common: shared libraries.
• Ecosystem: Hive (SQL), Pig (scripts), HBase (NoSQL), Sqoop, Flume, Spark.
• Features: scalable, fault tolerant, low cost, flexible with structured and unstructured data.

How voluminous data are handled

1. The data is divided into large blocks and stored across many nodes in HDFS, each block replicated (usually 3 copies) for reliability.
2. Computation is sent to the data (data locality), instead of moving data to the computation.
3. MapReduce jobs run map tasks in parallel on all blocks, then reduce tasks aggregate the results.
4. Failed tasks or nodes are detected through heartbeats and work is re-run on other nodes.
5. Capacity grows by simply adding more nodes (horizontal scaling).
6. Higher-level tools (Hive, Pig, Spark) simplify queries and speed up analysis.