Scribd
Upload
132 views

The Origin of The Name "Zookeeper"

Apache Zookeeper is a coordination service that enables synchronization across distributed applications like Hadoop. It helps manage configuration, implement reliable messaging, run redundant services, and synchronize process execution across nodes in a cluster. It was developed at Yahoo and named Zookeeper because distributed systems are difficult to manage like a "zoo".

Uploaded by

akurathikotaiah
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
132 views

The Origin of The Name "Zookeeper"

Apache Zookeeper is a coordination service that enables synchronization across distributed applications like Hadoop. It helps manage configuration, implement reliable messaging, run redundant services, and synchronize process execution across nodes in a cluster. It was developed at Yahoo and named Zookeeper because distributed systems are difficult to manage like a "zoo".

Uploaded by

akurathikotaiah
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 4

Apache Zookeeper is a coordination service for distributed application that enables synchronization

across a cluster.
So, in case of Hadoop, ZooKeeper will help you with coordination between Hadoop nodes. 

For example, it makes it easier to:

 Manage configuration across nodes. If you have dozens or hundreds of nodes, it


becomes hard to keep configuration in sync across nodes and quickly make changes.
ZooKeeper helps you quickly push configuration changes.
 Implement reliable messaging. With ZooKeeper, you can easily implement a
producer/consumer queue that guarantees delivery, even if some consumers or even one of
the ZooKeeper servers fails.
 Implement redundant services. With ZooKeeper, a group of identical nodes (e.g. database
servers) can elect a leader/master and let ZooKeeper refer all clients to that master server. If
the master fails, ZooKeeper will assign a new leader and notify all clients.
 Synchronize process execution. With ZooKeeper, multiple nodes can coordinate the start
and end of a process or calculation. This ensures that any follow-up processing is done only
after all nodes have finished their calculations.

The Origin of the Name “ZooKeeper”


ZooKeeper was developed at Yahoo! Research. Yahoo had been working on ZooKeeper for
a while and pitching it to other groups. At the time the ZooKeeper group had been working
with the Hadoop team and had started a variety of projects with the names of animals,
Apache Pig being the most well known. As the group started talking about different possible
names, one of the group members mentioned that they should avoid another animal name
because it started to sound like a zoo. That is when it clicked: distributed systems are a zoo.
They are chaotic and hard to manage, and ZooKeeper is meant to keep them under control.

Projects which uses ZooKeeper


 Apache BookKeeper- BookKeeper(ZooKeeper subproject) is a replicated service to
reliably log streams of records.
 Apache Hadoop MapReduce- The next generation of Hadoop MapReduce (colled
"Yarn") uses ZooKeeper.
 Apache HBase- HBase is the Hadoop database. Its an open-source, distributed,
column-oriented store modeled after the Google paper, Bigtable. HBase uses
ZooKeeper for master election, server lease management, bootstrapping, and
coordination between servers.
 Apache Kafka- Kafka is a distributed publish/subscribe messaging system. Kafka
queue consumers uses Zookeeper to store information on what has been consumed
from the queue.
 Apache Storm- Storm uses Zookeeper to store all state so that it can recover from
an outage in any of its (distributed) component services.

World without ZooKeeper


When designing a distributed system, there is typically a need for designing and developing
some coordination services:

 Name service— A naming service is a service that maps a name to some


information associated with that name. A telephone directory is a name service that
maps the name of a person to his/her telephone number. In the same way, a DNS
service is a name service that maps a domain name to an IP address. In your
distributed system, you may want to keep a track of which servers or services are up
and running and look up their status by name.
 Locking— To allow for serialized access to a shared resource in your distributed
system, you may need to implement distributed mutexes.
 Synchronization— Hand in hand with distributed mutexes is the need for
synchronizing access to shared resources. Whether implementing a producer-
consumer queue or a barrier.
 Configuration management— The configuration of your distributed system must
centrally stored and managed.This means that any new nodes joining should pick up
the up-to-date centralized configuration as soon as they join the system.
 Leader election— Your distributed system may have to deal with the problem of
nodes going down, and you may want to implement an automatic fail-over strategy.
You can do this by leader election.

Previous systems in a distributed sytems have implemented components like distributed


lock managers or have used distributed databases for coordination. While it's possible to
design and implement all of these services from scratch, it's extra work and difficult to
debug any problems, race conditions, or deadlocks. Just like you don't go around writing
your own hashing function in your code, there was a need that people shouldn't go around
writing their own name services or leader election services from scratch every time they
need it. Moreover, you could hack together a very simple group membership service
relatively easily, but it would require much more work to write it to provide reliability,
replication, and scalability. This led to the development and open sourcing of Apache
ZooKeeper, an out-of-the box reliable, scalable, and high-performance coordination service
for distributed systems.

ZooKeeper, in fact, borrows a number of concepts from these prior systems. It does not
expose a lock interface or a general purpose interface for storing data, however. The design
of ZooKeeper is specialized and very focused on coordination tasks. It is certainly possible
to build distributed systems without using ZooKeeper. ZooKeeper, however, offers
developers the possibility of focusing more on application logic rather than on arcane
distributed systems concepts. Programming distributed systems without ZooKeeper is
possible, but more difficult.

Why is Distributed Systems Coordination


Hard?
When an application starts up, all of the different processes needs to find the application
configuration. Over time this configuration may change. We could shut everything down,
redistribute configuration files, and restart, but that may incur extended periods of
application downtime during reconfiguration. Also as the load changes, we want to be able
to add or remove new machines and processes.

The problems described above are functional problems that you can design solutions for
and you can test your solutions before deployment. But the truly difficult problems
encounter, when the distributed applications have to do with faults specifically, crashes and
communication faults. These failures can crop up at any point, and it may be impossible to
enumerate all the different cases that need to be handled.

One of the diferences between single machine and distributed applications is: When a
single machine crashes, all the processes running on that machine fail. If there are multiple
processes running on the machine and a process fails, the other processes can find out
about the failure from the operating system. The operating system can also provide strong
messaging guarantees between processes. All of this changes in a distributed environment:
if a machine or process fails, other machines will keep running and may need to take over
for the faulty processes. To handle faulty processes, the processes that are still running
must be able to detect the failure; messages may be lost, and there may even be clock drift.

Okay, so we cannot have an ideal fault-tolerant, distributed, real-world system that


transparently takes care of all problems that might ever occur. We can strive for a slightly
less ambitious goal, though.

Having pointed out that the perfect solution is impossible, we can repeat that ZooKeeper is
not going to solve all the problems that the distributed application developer has to face. It
does give the developer a nice framework to deal with these problems, though.

What does a ZooKeeper do?


ZooKeeper is itself a distributed application providing services for developing a distributed
application. It coordinates a group of nodes within the cluster and maintains shared data
with effective synchronization techniques. Some of the services provided by zookeeper are:

 ZooKeeper exposes a simple interface for Naming service which identifies the nodes
in a cluster by name simialr to DNS.
 ZooKeeper provides for an easy way for you to implement distributed mutexes to
allow for serialized access to a shared resource in your distributed system.
 You can use ZooKeeper to centrally store and manage the configuration of your
distributed system. This means that any new nodes joining will pick up the up-to-date
centralized configuration from ZooKeeper as soon as they join the system. This also
allows you to centrally change the state of your distributed system by changing the
centralized configuration through one of the ZooKeeper clients.
 ZooKeeper provides off-the-shelf support for leader election which will deal with the
problem of nodes going down.

You might also like