Introducing Curator - The Netflix ZooKeeper Library

By Jordan Zimmerman

Open Source at Netflix
We are committed to Open Source Software at Netflix. We've blogged about it in the past. Today we are announcing a portal for open source projects from Netflix. The portal is currently hosted on Github. There are several projects in the pipeline (including Curator which we're announcing today):

Curator - The Netflix ZooKeeper Library
Astyanax - The Netflix Cassandra Client
Priam - Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra
CassJMeter - JMeter plugin to run cassandra tests

ZooKeeper
ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface. For full details on ZooKeeper, refer to these pages:

Difficult to Use Correctly
While ZooKeeper comes bundled with a Java client, using the client is non-trivial and error prone. Users of the client are expected to do a great deal of manual housekeeping.

Connection Issues:

Initial connection: the ZooKeeper client does a handshake with the server that takes some time. Any methods executed synchronously against the server (e.g. create(), getData(), etc.) will throw an exception if this handshake hasn't completed.
Failover: if the ZooKeeper client loses its connection to the server, it will failover to another server in the cluster. However, this process puts the client back into "initial connection" mode.
Session expiration: there are edge cases that can cause the ZooKeeper session to expire. Clients are expected to watch for this state and close and re-create the ZooKeeper instance.

Recoverable Errors:

When creating a sequential ZNode on the server, there is the possibility that the server will successfully create the ZNode but crash prior to returning the node name to the client.
There are several recoverable exceptions thrown by the ZooKeeper client. Users are expected to catch these exceptions and retry the operation.

Recipes:

The standard ZooKeeper "recipes" (locks, leaders, etc.) are only minimally described and subtly difficult to write correctly.
Some important edge cases are not mentioned in the recipes. For example, the lock recipe does not describe how to deal with a server that successfully creates the Sequential/Ephemeral node but crashes before returning the node name to the client. If not dealt with properly, dead locks can result.
Certain use cases must be conscious of connection issues. For example, Leader Election must watch for connection instability. If the connected server crashes, the leader cannot assume it is safe to continue as the leader until failover to another server is successful.

The above issues (and others like it) must be addressed by every user of ZooKeeper. Solutions are neither easy to write nor obvious and can take considerable time. Curator deals with all of them.

What is Curator?
Curator n kyoor͝ˌātər: a keeper or custodian of a museum or other collection - A ZooKeeper Keeper. It consists of three related projects:

curator-client - A replacement for the bundled ZooKeeper class that takes care of some low-level housekeeping and provides some useful utilities
curator-framework - The Curator Framework is a high-level API that greatly simplifies using ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing connections to the ZooKeeper cluster and retrying operations.
curator-recipes - Implementations of some of the common ZooKeeper "recipes". The implementations are built on top of the Curator Framework.

Curator is focused on the recipes: locks, leaders, etc. Most people interested in ZooKeeper don't need to be concerned with the details of connection management, etc. What they want is a simple way to use the recipes. Curator is directed at this goal.

Curator deals with ZooKeeper complexity in the following ways:

Retry Mechanism: Curator supports a pluggable retry mechanism. All ZooKeeper operations that generate a recoverable error get retried per the configured retry policy. Curator comes bundled with several standard retry policies (e.g. exponential backoff).
Connection State Monitoring: Curator constantly monitors the ZooKeeper connection. Curator users can listen for state changes in the connection and respond accordingly.
ZooKeeper Instance Management: Curator manages the actual connection to the ZooKeeper cluster using the standard ZooKeeper class. However, the instance is managed internally (though you can access it if needed) and recreated as needed. Thus, Curator provides a reliable handle to the ZooKeeper cluster (unlike the built-in implementation).
Correct, Reliable Recipes: Curator comes bundled with implementations of most of the important ZooKeeper recipes (and some additional recipes as well). The implementations are written using ZooKeeper best practices and take account of all known edge cases (as mentioned above).
Curator's focus on recipes makes your code more resilient as you can focus strictly on the ZooKeeper feature you're interested in without worrying about correctly implementing ZooKeeper housekeeping requirements.

ZooKeeper at Netflix
ZooKeeper/Curator is being used extensively at Netflix. Some of the uses are:

InterProcessMutex used for ensuring unique values in various sequence ID generators
Cassandra Backups
TrackID Service
Our Chukwa collector uses the LeaderSelector for various housekeeping tasks
We make use of some third party services that allow only a limited number of concurrent users. The InterprocessSemaphore is used to manage this.
Various Caches

Accessing Curator

The Curator binaries are posted to Maven Central which makes accessing them very easy
The source code for Curator is hosted at Github: https://github.com/Netflix/curator
Extensive documentation is currently at https://github.com/Netflix/curator/wiki

Like what you see? We're hiring!