Introducing Exhibitor - A Supervisor System for Apache ZooKeeper

by Jordan Zimmerman

ZooKeeper
ZooKeeper is a high-performance coordination service for distributed applications. We've already open sourced a client library for ZooKeeper, Curator. In addition to Curator for clients, we found a need for a supervisor service that runs alongside ZooKeeper server instances. Thus, we are introducing Exhibitor and open sourcing it.

ZooKeeper Administrative Issues
Managing a ZooKeeper cluster requires a lot of manual effort — see the ZooKeeper Administrator's Guide for details. In particular, ZooKeeper is statically configured. The instances that comprise a ZooKeeper ensemble must be hard coded into a configuration file that must be identical on each ZooKeeper instance. Once the ZooKeeper instances are started it's not possible to reconfigure the ensemble without updating the configuration file and restarting the instances. If not properly done, ZooKeeper can lose quorum and client's can perceive the ensemble as being unavailable.

In addition to static configuration issues, ZooKeeper requires regular maintenance. When using ZooKeeper versions prior to 3.4.x you are advised to periodically clean up the ZooKeeper log files. Also, you are advised to have a monitor of some kind to assure that each ZooKeeper instance is up and serving requests.

Exhibitor Features
Exhibitor provides a number of features that make managing a ZooKeeper ensemble much easier:

• Instance Monitoring: Each Exhibitor instance monitors the ZooKeeper server running on the same server. If ZooKeeper is not running (due to crash, etc.), Exhibitor will rewrite the zoo.cfg file and restart it.
• Log Cleanup: In versions prior to ZooKeeper 3.4.x log file maintenance is necessary. Exhibitor will periodically do this maintenance.
• Backup/Restore: Exhibitor can periodically backup the ZooKeeper transaction files. Once backed up, you can index any of these transaction files. Once indexed, you can search for individual transactions and “replay” them to restore a given ZNode to ZooKeeper.
• Cluster-wide Configuration: Exhibitor attempts to present a single console for your entire ZooKeeper ensemble. Configuration changes made in Exhibitor will be applied to the entire ensemble.
• Rolling Ensemble Changes: Exhibitor can update the servers in the ensemble in a rolling fashion so that the ZooKeeper ensemble can stay up and in quorum while the changes are being made.
• Visualizer: Exhibitor provides a graphical tree view of the ZooKeeper ZNode hierarchy.
• Curator Integration: Exhibitor and Curator (Cur/Ex!) can be configured to work together so that Curator instances are updated for changes in the ensemble.
• Rich REST API: Exhibitor exposes a REST API for programmatic integration.

Easy To Use GUI
Exhibitor is an easy to use web application with a modern UI:

Easy To Integrate
There are two versions of Exhibitor:

• Standalone: The standalone version comes pre-configured as a Jetty-based self-contained application.
• Core: The core version can be integrated into an existing application or you can build an extended application around it.

For More Information...

• Exhibitor binaries are posted to Maven Central.
• The source code for Exhibitor is hosted at Github: https://github.com/Netflix/exhibitor
• Extensive documentation is currently at https://github.com/Netflix/exhibitor/wiki

We're Hiring!
Like what you see? Netflix is a great place for programmers. Check out our Jobs Board today.