Components of Hadoop

Apache Hadoop is an open-source framework for distributed storage and processing of very large data sets on clusters of commodity hardware. It has core components and an ecosystem of related tools.

Core components

1. HDFS (Hadoop Distributed File System): the storage layer, based on the Google File System. Files are split into large blocks (128 MB) replicated (default 3 times) across DataNodes. The NameNode keeps metadata. It provides fault tolerance and high-throughput access.
2. MapReduce: the processing layer. A job is split into map tasks (process input splits in parallel, emit key-value pairs) and reduce tasks (aggregate by key). Computation moves to the data.
3. YARN (Yet Another Resource Negotiator): the resource-management layer (Hadoop 2+). The ResourceManager and NodeManagers allocate CPU and memory containers to applications, so that MapReduce, Spark and others can share the cluster.
4. Hadoop Common: shared Java libraries and utilities (I/O, RPC, serialization, security) used by all modules.

 +--------------------------------------+
 | Ecosystem: Hive Pig HBase Sqoop ...  |
 +--------------------------------------+
 | MapReduce (processing)               |
 +--------------------------------------+
 | YARN (resource management)           |
 +--------------------------------------+
 | HDFS (storage)                       |
 +--------------------------------------+

Ecosystem components

Key features

Scalable (add nodes), fault tolerant (replication, task re-execution), cost-effective (commodity hardware), and works with structured and unstructured data.