What is Apache ZooKeeper?
Last Updated :
26 Apr, 2025
Zookeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives to implement higher-level services for synchronization, configuration maintenance, and group and naming.
In a distributed system, there are multiple nodes or machines that need to communicate with each other and coordinate their actions. ZooKeeper provides a way to ensure that these nodes are aware of each other and can coordinate their actions. It does this by maintaining a hierarchical tree of data nodes called “Znodes“, which can be used to store and retrieve data and maintain state information. ZooKeeper provides a set of primitives, such as locks, barriers, and queues, that can be used to coordinate the actions of nodes in a distributed system. It also provides features such as leader election, failover, and recovery, which can help ensure that the system is resilient to failures. ZooKeeper is widely used in distributed systems such as Hadoop, Kafka, and HBase, and it has become an essential component of many distributed applications.
Why do we need it?
- Coordination services: The integration/communication of services in a distributed environment.
- Coordination services are complex to get right. They are especially prone to errors such as race conditions and deadlock.
- Race condition-Two or more systems trying to perform some task.
- Deadlocks– Two or more operations are waiting for each other.
- To make the coordination between distributed environments easy, developers came up with an idea called zookeeper so that they don’t have to relieve distributed applications of the responsibility of implementing coordination services from scratch.
What is distributed system?
- Multiple computer systems working on a single problem.
- It is a network that consists of autonomous computers that are connected using distributed middleware.
- Key Features: Concurrent, resource sharing, independent, global, greater fault tolerance, and price/performance ratio is much better.
- Key Goals: Transparency, Reliability, Performance, Scalability.
- Challenges: Security, Fault, Coordination, and resource sharing.
Coordination Challenge
- Why is coordination in a distributed system the hard problem?
- Coordination or configuration management for a distributed application that has many systems.
- Master Node where the cluster data is stored.
- Worker nodes or slave nodes get the data from this master node.
- single point of failure.
- synchronization is not easy.
- Careful design and implementation are needed.
Apache Zookeeper
Apache Zookeeper is a distributed, open-source coordination service for distributed systems. It provides a central place for distributed applications to store data, communicate with one another, and coordinate activities. Zookeeper is used in distributed systems to coordinate distributed processes and services. It provides a simple, tree-structured data model, a simple API, and a distributed protocol to ensure data consistency and availability. Zookeeper is designed to be highly reliable and fault-tolerant, and it can handle high levels of read and write throughput.
Zookeeper is implemented in Java and is widely used in distributed systems, particularly in the Hadoop ecosystem. It is an Apache Software Foundation project and is released under the Apache License 2.0.
Architecture of Zookeeper
Zookeeper Services
The ZooKeeper architecture consists of a hierarchy of nodes called znodes, organized in a tree-like structure. Each znode can store data and has a set of permissions that control access to the znode. The znodes are organized in a hierarchical namespace, similar to a file system. At the root of the hierarchy is the root znode, and all other znodes are children of the root znode. The hierarchy is similar to a file system hierarchy, where each znode can have children and grandchildren, and so on.
Important Components in Zookeeper
ZooKeeper Services
- Leader & Follower
- Request Processor – Active in Leader Node and is responsible for processing write requests. After processing, it sends changes to the follower nodes
- Atomic Broadcast – Present in both Leader Node and Follower Nodes. It is responsible for sending the changes to other Nodes.
- In-memory Databases (Replicated Databases)-It is responsible for storing the data in the zookeeper. Every node contains its own databases. Data is also written to the file system providing recoverability in case of any problems with the cluster.
Other Components
- Client – One of the nodes in our distributed application cluster. Access information from the server. Every client sends a message to the server to let the server know that client is alive.
- Server– Provides all the services to the client. Gives acknowledgment to the client.
- Ensemble– Group of Zookeeper servers. The minimum number of nodes that are required to form an ensemble is 3.
Zookeeper Data Model
ZooKeeper data model
In Zookeeper, data is stored in a hierarchical namespace, similar to a file system. Each node in the namespace is called a Znode, and it can store data and have children. Znodes are similar to files and directories in a file system. Zookeeper provides a simple API for creating, reading, writing, and deleting Znodes. It also provides mechanisms for detecting changes to the data stored in Znodes, such as watches and triggers. Znodes maintain a stat structure that includes: Version number, ACL, Timestamp, Data Length
Types of Znodes:
- Persistence: Alive until they’re explicitly deleted.
- Ephemeral: Active until the client connection is alive.
- Sequential: Either persistent or ephemeral.
Why do we need ZooKeeper in the Hadoop?
Zookeeper is used to manage and coordinate the nodes in a Hadoop cluster, including the NameNode, DataNode, and ResourceManager. In a Hadoop cluster, Zookeeper helps to:
- Maintain configuration information: Zookeeper stores the configuration information for the Hadoop cluster, including the location of the NameNode, DataNode, and ResourceManager.
- Manage the state of the cluster: Zookeeper tracks the state of the nodes in the Hadoop cluster and can be used to detect when a node has failed or become unavailable.
- Coordinate distributed processes: Zookeeper can be used to coordinate distributed processes, such as job scheduling and resource allocation, across the nodes in a Hadoop cluster.
Zookeeper helps to ensure the availability and reliability of a Hadoop cluster by providing a central coordination service for the nodes in the cluster.
How ZooKeeper in Hadoop Works?
ZooKeeper operates as a distributed file system and exposes a simple set of APIs that enable clients to read and write data to the file system. It stores its data in a tree-like structure called a znode, which can be thought of as a file or a directory in a traditional file system. ZooKeeper uses a consensus algorithm to ensure that all of its servers have a consistent view of the data stored in the Znodes. This means that if a client writes data to a znode, that data will be replicated to all of the other servers in the ZooKeeper ensemble.
One important feature of ZooKeeper is its ability to support the notion of a “watch.” A watch allows a client to register for notifications when the data stored in a znode changes. This can be useful for monitoring changes to the data stored in ZooKeeper and reacting to those changes in a distributed system.
In Hadoop, ZooKeeper is used for a variety of purposes, including:
- Storing configuration information: ZooKeeper is used to store configuration information that is shared by multiple Hadoop components. For example, it might be used to store the locations of NameNodes in a Hadoop cluster or the addresses of JobTracker nodes.
- Providing distributed synchronization: ZooKeeper is used to coordinate the activities of various Hadoop components and ensure that they are working together in a consistent manner. For example, it might be used to ensure that only one NameNode is active at a time in a Hadoop cluster.
- Maintaining naming: ZooKeeper is used to maintain a centralized naming service for Hadoop components. This can be useful for identifying and locating resources in a distributed system.
ZooKeeper is an essential component of Hadoop and plays a crucial role in coordinating the activity of its various subcomponents.
Reading and Writing in Apache Zookeeper
ZooKeeper provides a simple and reliable interface for reading and writing data. The data is stored in a hierarchical namespace, similar to a file system, with nodes called znodes. Each znode can store data and have children znodes. ZooKeeper clients can read and write data to these znodes by using the getData() and setData() methods, respectively. Here is an example of reading and writing data using the ZooKeeper Java API:
Java
ZooKeeper zk = new ZooKeeper("localhost:2181", 3000, null);
String path = "/myZnode";
String data = "hello world";
zk.create(path, data.getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
byte[] bytes = zk.getData(path, false, null);
String readData = new String(bytes);
System.out.println(readData);
zk.close();
|
Python3
from kazoo.client import KazooClient
zk = KazooClient(hosts='localhost:2181')
zk.start()
zk.ensure_path('/gfg_node')
zk.set('/gfg_node', b'some_data')
data, stat = zk.get('/gfg_node')
print(data)
zk.stop()
|
Session and Watches
Session
- Requests in a session are executed in FIFO order.
- Once the session is established then the session id is assigned to the client.
- Client sends heartbeats to keep the session valid
- session timeout is usually represented in milliseconds
Watches
- Watches are mechanisms for clients to get notifications about the changes in the Zookeeper
- Client can watch while reading a particular znode.
- Znodes changes are modifications of data associated with the znodes or changes in the znode’s children.
- Watches are triggered only once.
- If the session is expired, watches are also removed.
Similar Reads
What is Apache Camel?
In today's technology-driven world, seamless integration between different applications and systems is essential for businesses to stay competitive and efficient. The Java Camel Framework, often referred to as Apache Camel, is a versatile open-source framework that facilitates the integration of div
6 min read
Apache HBase
Prerequisite - Introduction to Hadoop HBase is a data model that is similar to Google's big table. It is an open source, distributed database developed by Apache software foundation written in Java. HBase is an essential part of our Hadoop ecosystem. HBase runs on top of HDFS (Hadoop Distributed Fil
5 min read
What is Apache Kafka Streams?
Kafka Streams is a library for processing and analyzing data stored in Kafka. It expands on crucial stream processing ideas such as clearly separating event time from processing time, allowing for windows, and managing and querying application information simply but effectively in real time. Kafka S
4 min read
Apache JMeter - An Introduction
Apache JMeter is a pure Java-based open-source software application used for testing load testing functional behavior and measuring performance. It is used for testing a variety of services,i.e. web applications, databases, etc. Apache JMeter FeaturesFeatures of JMeter that make it so powerful are m
3 min read
How to install Kafka with Zookeeper on Ubuntu?
Apache Kafka is an open-source distributed event streaming platform used to handle real-time data feeds. It is designed to handle a high volume, high throughput, and low latency data streams, and can be used to process, store, and analyze data in real-time. Kafka can be used to build real-time strea
4 min read
Apache Kafka - Topics using CLI
In Apache Kafka, a topic is a category or stream of messages that the Kafka message broker (or cluster) stores. Producers write data to topics and consumers read from topics. A Topic in Kafka is similar to a table in a database or a stream in a stream processing system. Each topic is divided into a
3 min read
Apache Commons vs Java NIO
If you work with files or reports and need to automate a process, you always wondered what an efficient way is to automate the process. Being in finance, I have experienced this and wondered if there is a library available to avoid writing boilerplate code. File watcher to the rescue, but wait what
6 min read
Components of Apache Spark
Spark is a cluster computing system. It is faster as compared to other cluster computing systems (such as Hadoop). It provides high-level APIs in Python, Scala, and Java. Parallel jobs are easy to write in Spark. In this article, we will discuss the different components of Apache Spark. Spark proces
5 min read
Apache Camel - Content Based Routing
In the world of integration and message routing, one common challenge is determining how to route messages or files to different destinations based on their content. Apache Camel, a popular integration framework, provides a powerful feature known as content-based routing that allows you to make rout
4 min read
Apache Kafka - Create Producer using Java
Apache Kafka is a publish-subscribe messaging system. A messaging system lets you send messages between processes, applications, and servers. Apache Kafka is software where topics (A topic might be a category) can be defined and further processed. Read more on Kafka here: What is Apache Kafka and Ho
4 min read