Short note: Zookeeper

Apache ZooKeeper is a centralized, open-source coordination service for distributed applications. It maintains configuration information and provides naming, distributed synchronization and group services, so that applications in a cluster (Hadoop, HBase, Kafka) do not need to implement these tricky tasks themselves.

Data model

ZooKeeper stores data in a hierarchical namespace, like a file system, made of znodes. Each znode holds a small amount of data (under 1 MB) and can have children. The types are persistent, ephemeral (deleted when the client session ends) and sequential. Clients can set watches to be notified when a znode changes.

Architecture

          Clients
      /      |       \
  Server   Server   Server     <- ZooKeeper ensemble
(Follower) (Leader) (Follower)

An ensemble of an odd number of servers (3, 5, ...) replicates the data. One server is elected leader and handles all writes; the others are followers. Any server can serve reads, and followers forward writes to the leader. The service keeps working as long as a majority of servers is alive.

Services provided

• Configuration management
• Leader election
• Distributed locks and barriers
• Naming service and group membership (service discovery)
• Failure detection through ephemeral znodes

Role in Hadoop

HBase uses it to elect the active HMaster, track region servers and store the location of the META table. HDFS and YARN use it for automatic failover of the NameNode/ResourceManager in high-availability setups.

Properties

Sequential consistency, atomicity of updates, single system image, reliability and timeliness. It is very fast for read-heavy workloads.