Redis Cluster, the distributed version of Redis, is set to be another major feature following Lua scripting support. The official announcement states that the beta version will be released in Q3 of this year, with the first stable release by year-end. Currently, while the stable version of Redis does not yet integrate distributed functionality, development on Redis Cluster in the dev branch has made significant progress. We can already set up a Redis cluster and use some of its features. Today, based on the latest dev branch code, I attempted to set up a three-node Redis cluster. Let me briefly summarize the process here, hoping it helps fellow readers!(Note: The setup method in this blog post is a completely unofficial, makeshift approach. Redis has officially released the proper Redis Cluster setup guide, linked here: http://redis.io/topics/cluster-tutorial.)
First, download the latest Redis dev branch source package. This is straightforward: you can download it from the Redis GitHub homepage, or directly run “git clone git://github.com/antirez/redis.git” to clone the entire Redis code repository, provided you have git installed.
Second, install Redis. Since I’m setting up a three-node Redis cluster, Redis must be installed on all three nodes. Let’s name these nodes Redis Cluster Node1, Node2, and Node3 for now, with the installation directory being /usr/local/redis/.
Third, modify the Redis configuration file. Even in the dev branch that supports distributed functionality, Redis Cluster is not enabled by default in the configuration file. So before starting the Redis server, we need to modify the config file to enable Redis Cluster. For simplicity, we’ll only modify settings related to Redis Cluster, leaving other unrelated settings at their defaults. The specific changes are as follows:
################################ REDIS CLUSTER ###############################
#
# Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
cluster-enabled yes
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system does not have
# overlapping cluster configuration file names.
#
cluster-config-file nodes-6379.conf
# Cluster node timeout is the amount of seconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiplicators of the node timeout.
#
cluster-node-timeout 15
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
Fourth, start the Redis servers on all three nodes. At this point, all three Redis server nodes will start running in Redis Cluster mode, but they do not automatically form a cluster because they are still in a state of “I don’t know you, you don’t belong to me.” Each is a solitary Redis node, or a cluster containing only one node. We can connect to the servers via the Redis client to check their status. Figure 1 shows the status query method and results, where the cluster nodes command is used to view all nodes in the current Redis node’s cluster, and cluster info is used to view the overall status of the cluster. From the figure, we can see the Redis cluster contains only one Redis node, which is the current node itself, and the overall cluster status is fail.
Figure 1: Redis Node Status Before Cluster Formation
Fifth, set up the Redis cluster. Setting up the cluster here simply means making the three previously started Redis nodes aware of each other. So how is this achieved? The answer is the cluster meet command. This command’s purpose is to proactively introduce the current node to another node. Figure 2 shows the execution method and effect of the cluster meet command. From the figure, we see that we used the cluster meet command to introduce Redis Cluster Node1 to Redis Cluster Node2 (node IP address 192.168.32.3, running port 6379) and Redis Cluster Node3 (node IP address 192.168.32.4, running port 6379). Afterward, checking the cluster nodes and cluster status again shows that the three nodes have been successfully merged into the same cluster.
Figure 2: Setting Up the Redis Cluster
Sixth, assign hash slots to the Redis nodes in the cluster. Through the above operations, we have brought three independent Redis nodes into the same cluster. Have we now completed all the work of setting up the cluster?Not at all! As we can see from checking the cluster status in Figure 2, the current cluster state is still fail. At this point, the Redis cluster is non-functional and unable to process any Redis commands. So why is the cluster state still fail? I found the reason by consulting the official documentation, and the original text is quoted below:
The FAIL state for the cluster happens in two cases.
1) If at least one hash slot is not served as the node serving it currently is in FAIL state.
2) If we are not able to reach the majority of masters (that is, if the majorify of masters are simply in PFAIL state, it is enough for the node to enter FAIL mode).
Clearly, the reason our cluster is in a fail state is not the second case. That means at least one hash slot is not being served! Think about it for a moment, and it’s obvious! It’s not just one hash slot that isn’t served—there is no Redis node serving any hash slots at all! As we all know, Redis Cluster partitions data based on primary keys using hash slots. So, a key-value data entry is automatically mapped to a hash slot via an algorithm. However, which Redis node a hash slot is stored on is not automatically mapped; it must be manually assigned by the cluster administrator. So, how many hash slots do we need to assign to Redis nodes? According to the source code, the number is 16384. This means we need to distribute these 16384 hash slots across the three nodes in the cluster. There are several commands for assigning hash slots in Redis, including cluster addslots, cluster delslots, and cluster setslot. Since we are currently in the initialization phase of the cluster, we can choose cluster addslots to assign hash slots. The syntax for this command is cluster addslots slot1 [slot2] … [slotN].
Figure 3: Contents of nodes-6379.conf on Redis Cluster Node1
But another problem arises. For 16384 hash slots, we certainly can’t assign them one by one using cluster addslots, can we? Fortunately, there is another method for assigning hash slots: using the cluster configuration file, which is the file specified by the cluster-config-file parameter we set in Step 3, to complete the segmented configuration of hash slots. At this point, we find the configuration file named nodes-6379.conf in the directory where we started the Redis server. Figure 3 shows the contents of nodes-6379.conf on Redis Cluster Node1. From the figure, we can see that the content of this file is the same as the result of executing the cluster nodes command—it’s a description of all Redis nodes in the cluster. What we need to modify is adding the hash slot assignment information for each node. For the contents of nodes-6379.conf on Redis Cluster Node1, the modification is as follows:cda76a0a094d2ce624e33bed7f3c75689a4128fd :0 myself,master – 0 0 connected 0-5000 (Note: append the hash slot range after the line describing the node itself, i.e., the line containing ‘myself’).Similarly, forRedis Cluster Node2’s nodes-6379.conf file, append 5001-10000, and forRedis Cluster Node3’s nodes-6379.conf file, append 10001-16383. After this configuration,Redis Cluster Node1 is responsible for storing all hash slots from 0 to 5000, Redis Cluster Node2 is responsible for all hash slots from 5001 to 10000, and Redis Cluster Node3 is responsible for all hash slots from 10001 to 16383.
Save the above modifications, restart the three Redis servers via the Redis client, and execute the commands cluster nodes and cluster info again. We can then see that the cluster status is now ok (as shown in Figure 4). This indicates that the previous modifications to the cluster configuration files did indeed take effect. If the cluster status is still fail after these steps, you can try re-modifying the nodes-6379.conf files on the three nodes by deleting all configuration lines except the one containing ‘myself’, then save and restart the servers. Of course, one point must be made here: I have not seen any official documentation on the Redis Cluster configuration file, so while this configuration method is effective, it may not necessarily be the officially recommended approach!
Figure 4: Cluster Status After Modifying and Restarting with the Cluster Configuration File
Finally, let’s execute Redis commands. Once the cluster status shows as ok, we can execute Redis commands just as we would on a standalone Redis instance. Figure 5 shows the execution result of the “set foo bar” command when the Redis client connects to Redis Cluster Node1. From the figure, we can see that the hash slot for the primary key ‘foo’ is stored on Redis Cluster Node3. We then connect to Redis Cluster Node3, execute the same command, and it runs correctly!
Figure 5: Executing Redis Commands
