Spirit boy online carry the pot

Redis Records of online primary/secondary switchover

background

The instance host where the Redis master resides needs to be offline for maintenance. How to perform primary/secondary switchover has the least impact on online services. The current architecture is shown below

Technical solution

Plan 1

The introduction of redis sentinel.

Steps:

  1. The deployment of redis sentinel
  2. Modify service configurations and modify service codes.
  3. kill master.
  4. check
  5. Create a new master-slave logic

Advantages:

  1. Kill the master database. Sentinel automatically promotes slave -> master.

Disadvantages:

  1. Need to deploy Redis Sentinel, new middleware.
  2. You need to change the business code.

Scheme 2

Switch directly

Steps:

  1. Make writable from the library,redis-cli -h redis-slave -p 6379 CONFIG set slave-read-only no
  2. Modify the service configuration to point the redis address to the writable slave instance and restart the service application
  3. Ensure that the primary/secondary synchronization is complete and stop the synchronization.redis-cli -h redis-slave -p 6379 slaveof no one
  4. Create a new master-slave logic.

Advantages:

  1. In line with the intuition
  2. No code changes

Disadvantages:

  1. If any problem occurs during the switchover, the configuration and service project publication must be rolled back in time. It takes a long time and may cause data loss.

Plan 3

Introduce 4 layer proxy, haProxy

Steps:

  1. Deploy haProxy as the proxy for the current master.
  2. Modify the service configuration, and the redis address points to the HaProxy address to restart the service application.
  3. Make the slave library writable
  4. Modify the haProxy configuration to point to the writable slave library, and reload the configuration.
  5. Ensure that the primary/secondary synchronization is complete and stop the synchronization.
  6. Create a new master-slave logic.

advantages

  1. Haproxy is introduced in the middle layer, because haProxy supports dynamic reload configuration, and ensures that the connection before gracefully closes, and the business side can not perceive the primary/secondary switch of REDis.
  2. The primary/secondary switchover time is short

disadvantages

  1. New middleware has been introduced

Determine the solution

To be on the safe side, use plan 3.

The implementation of

  1. The post-HaProxy architecture was introduced

  2. Perform a primary/secondary switchover


export redis-slave=REDIS_SLAVE_IP
export redis-master=REDIS_MASTER_IP

Write from library
redis-cli -h redis-slave -p 6379 CONFIG set slave-read-only no

Change the haProxy configuration
sed -i 's/redis-master/redis-slave/g' haproxy.cfg

# reload haproxy
haproxy -f haproxy.cfg -p /tmp/haproxy.pid -sf `cat /tmp/haproxy.pid`

Make sure the master/slave synchronization is over
redis-cli -h redis-slave -p 6379 info | grep master_sync_in_progress
The value of # must be 0

# Stop master/slave synchronization
redis-cli -h redis-slave -p 6379 slaveof no one

# Create a new master and slave
redis-cli -h redis-master -p 6379 slaveof redis-slave 6379

Confirm that the new master/slave synchronization is over
redis-cli -h redis-master -p 6379 info | grep master_sync_in_progress
The value of # must be 0

# end
Copy the code
  1. Post-switch architecture

summary

It simply records the choice and operation of Redis master/slave switching.