Distributed system in big data example

A distributed system is a collection of independent computers (nodes) connected by a network that appears to the user as a single system. In big data, data and processing are spread over many nodes so that very large datasets can be stored and analysed in parallel.

Key characteristics

• Many nodes, no shared memory, communication by messages
• Scalability by adding nodes
• Fault tolerance through replication
• Transparency (location, replication, failure hidden from the user)
• Concurrency of many tasks

Common architecture in big data

        +----------------+
        |  Master node   |  (NameNode / JobTracker)
        +--------+-------+
        +--------+--------+
        |        |        |
   +----v--+ +---v---+ +--v----+
   |Worker1| |Worker2| |Worker3|
   | data  | | data  | | data  |
   | + CPU | | + CPU | | + CPU |
   +-------+ +-------+ +-------+

The master keeps metadata and schedules work. Workers store data blocks and compute locally.

Example 1: Hadoop cluster (word count over logs)

A 1 TB web log is uploaded to HDFS. It is split into 128 MB blocks and each block is stored on three different DataNodes. A MapReduce job sends the map code to the nodes that already hold the blocks. Each mapper counts words in its block, and reducers combine the counts. If one node fails, the block is read from another replica and the task is re-run.

Example 2: Google search

Web pages are stored in GFS over thousands of machines and indexed by MapReduce. A search query is sent to many index servers in parallel, and the results are merged and returned within a fraction of a second.

Other examples

Facebook using HDFS for its data warehouse and HBase for messaging, Amazon S3 and DynamoDB, and Cassandra clusters for NoSQL.

Advantages

Handles petabytes, low-cost hardware, high availability, and fast parallel processing.

Challenges

Network latency, node failures, consistency of replicas, clock synchronisation and security.

Components in an HDFS-based system

• NameNode (master): keeps the file tree, block locations and replication information.
• DataNodes (workers): store block replicas and send heartbeats and block reports.
• Client: reads and writes by asking the NameNode for locations and then talking to DataNodes.
• Processing layer: YARN/MapReduce schedules tasks near the blocks.

Why distribute big data

Design issues

Transparency (users should not see node locations), consistency of replicas, scheduling and load balancing, security, and handling of partial failure.