Redis single-point failure is probably an issue many people face, since the Redis cluster solution is still under development in Redis cluster Specification.
Of course, some netizens have also proposed the Redis High Availability Failover Transition Plan, and after reading this article, I combined its ideas with our existing situation to attempt a simple failover solution.
1: Two Redis write servers — bind the HAproxy VIP 192.168.1.100 on Redis_Master (192.168.1.10) and Redis_Slave (192.168.1.11). Four Redis read servers — bind the HAproxy VIP 192.168.1.101 on Slave1, Slave2, Slave3, and Slave4.
2: Redis_Master and Redis_Slave are started in Master-Slave mode (configured in the config file). Then send the slaveof command to Redis_Master (the master-slave relationship must be set in the config file first, and then use the command), designating it as the slave of Redis_Slave. This way, Redis_Master and Redis_Slave achieve bidirectional synchronization, which is the core of avoiding write single-point failures.
3: Start a monitoring script to scan both Redis services. If Redis_Master goes down, the monitoring script sends the slaveof NO ONE command to the online Redis_Slave, setting it as a temporary Master. At the same time, since the Redis_Master server is down, VIP 192.168.1.100 automatically fails over to the TempMaster, without affecting the application’s write operations to Redis. At this point, newly generated application data is saved to Redis_Slave, though the original read-only servers will experience brief data inconsistency. Then, start Redis_Master again using a command; in the config file it starts as a slave of Redis_Slave.
The local test monitoring script is as follows. This script can be used whether Redis_Master or Redis_Slave goes down. If Redis_Slave goes down, you don’t need to check the failoverProcess:
#!/bin/sh
# Monitor Redis services
masterProcess=`ps -ef|grep '/etc/redis.conf'|grep -v grep|wc -l` # Config file for master mode startup
failoverProcess=`ps -ef|grep '/etc/redis_failover.conf'|grep -v grep|wc -l` # Config file for slave mode startup
if [ $masterProcess -eq 0 -a $failoverProcess -eq 0 ];then
datetime=`date '+%Y%m%d%H:%M:%S'`
/usr/local/bin/redis-cli -h 127.0.0.1 -p 6380 slaveof NO ONE # Turn this Slave into TempMaster
mv /data/redisdb/dump.rdb /data/redisdb/$datetime”.dump.rdb” # Back up original data
/usr/local/bin/redis-server /etc/redis_failover.conf # Start with another config file in slave mode to auto-sync the latest data
sleep 2
/usr/local/bin/redis-cli -h 127.0.0.1 -p 6379 slaveof NO ONE # Revert to previous master
sleep 2
/usr/local/bin/redis-cli -h 127.0.0.1 -p 6380 slaveof 127.0.0.1 6379 # Restore previous master-slave relationship
sleep 2
/usr/local/bin/redis-cli -h 127.0.0.1 -p 6379 slaveof 127.0.0.1 6380 # Achieve bidirectional sync
else
echo “process is ok!”
fi
4: If any of the four slave servers under Master and Slave goes down, simply restart it and data will automatically sync from the slaveof machine.
The above is just an approach; I personally feel it should be tailored to the actual situation. Ultimately, it’s about solving the single-point and data consistency problems, which can also be achieved using dual-write or multi-write methods.
Redis High Availability Solution
Original article http://haili.me/archives/416.html