preface

The method of master/slave switchover technology is:

  • When the primary server is down, the secondary server needs to be manually switched over (slaveof no one) to the master/slave server, which requires human intervention.
  • This is time-consuming and laborious, but it also causes the service to be unavailable for a period of time. This is not the recommended approach.
  • For more time, we prioritized sentinel mode, which is currently the dominant mode for enterprise applications. The Redis Sentinel is a highly available implementation of Redis.
  • Sentinel is a tool for managing multiple Instances of Redis. It can monitor, notify Redis, and failover automatically.

Basic concept of Redis Sentinel

A schematic of master-slave replication and Sentinel high availability architecture

Redis Sentinel architecture

Main functions of Redis Sentinel Main functions of Sentinel include:

Primary node survival detection, primary/secondary operation detection, automatic failover, and primary/secondary switchover.

  • The minimum markup configuration for Redis is one host and one slave;
  • Redis’ Sentinel system can be used to manage multiple Redis servers.
  • The system can perform the following four tasks:

1. Monitoring: Sentinel will constantly check whether the master and slave servers are running properly; 2. Notification: Sentinel will send notifications to administrators or other applications through API scripts when a monitored Redis server has problems; 3. Automatic failover: Sentinel initiates automatic failover when the primary node is not working properly. It upgrades one of the slave nodes that is in a master-slave relationship with the failed master node to the new master node and points other slave nodes to the new master node; 4. Configuration provider: In Redis Sentinel mode, when the client application is initialized, it will connect to the collection of Sentinel nodes to obtain information about the primary node;

How does the Redis Sentinel work

  • When the Sentinel node connects to a Redis instance, it creates two connections: CMD and pub/sub. Sentinel sends commands to Redis via a CMD connection and connects to other Sentinel instances on the Redis instance via pub/sub.
  • Sentinel commands to interact with the Redis master and slave nodes
  • Each Sentinel sends a PING every second to its known primary instance, dependent instance, and other Sentinel instances.

  • Sentinel marks an instance as subjectively offline if it takes longer than the time specified by down since the last valid reply (milliseconds) to the PING command.

  • If the primary server is marked as subjective deregistered, then all Sentinel nodes of the primary server are monitored to confirm that the primary server has indeed entered a subjective deregistered state once per second.

  • If the primary server is marked as subjective logged out and there are enough sentinels (at least as specified in the profile) to match this judgment within the specified time range, the primary server is marked as objective logged out.

  • Typically, each Sentinel sends an INFO command every 10 seconds to all of its known primary and secondary servers. When Sentinel flags the master server as offline, the frequency at which Sentinel sends the INFO command to all slave servers of the offline master server will change from once every 10 seconds to once every second.

  • Sentinel and other sentinels negotiate the status of the primary node. If the primary node is in the SDOWN state, the new primary node is automatically selected by voting. Point the remaining slave nodes to the new master node for data replication.

  • If there are not enough sentries to allow the primary server to log out, the objective logout status of the primary server is removed. When the primary server returns a valid response to Sentinel’s PING command, the subjective offline status of the primary server will be deleted.

  • note

A robust Redis Sentinel cluster should use at least three Sentinel instances and be sure to place those instances on different computers and even on different physical areas. Sentinel cannot guarantee strong consistency. Sentinel is supported in common client application libraries. Sentinel needs constant testing and observation to ensure high availability

test

  • Creating a Configuration file

The simple configuration is as follows:

port 16379  # Sentinel port numberDaemonize Yes Sentinel Monitor Master 127.0.0.1 6379 1# monitoring master
protected-mode no
logfile "/usr/local/bin/sentinel-1/sentinel-1.log"   # log file
Copy the code
  • First, start redis to set the cluster, start Redis -CLI, and set 6379 as master

  • Restart sentinel
sudo redis-sentinel sentinel-1/sentinel.conf
Copy the code

  • Close to 6379

  • View the roles of the other two Redis-CLI

  • Start again, 6379

  • Viewing sentinel Logs

Sentinel. Conf

# Example sentinel.conf

# *** IMPORTANT ***
Bind IP address
# bind 127.0.0.1 192.168.1.1
Protected mode (whether to disable external links except for bound IP addresses)
# protected-mode no

# port <sentinel-port>
# Port on which this Sentinel instance is running
port 26379

# By default, Redis Sentinel does not run as a daemon. You can set this to yes if desired.
daemonize no

# Redis will write a pid file in /var/run/redis-sentinel.pid after enabling the daemon to run
pidfile /var/run/redis-sentinel.pid

Specifies the name of the log file. If the value is null, Sentinel log standard output is forced. Daemon, if the log is logged using standard output, the log is sent to /dev/null
logfile ""

# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in an environment where NAT can access Sentinel externally via non-local addresses.
#
# When announce-IP is provided, Sentinel declares the specified IP address in the communication, rather than automatically detecting the local address as usual.
#
# Similarly, Sentinel announces the specified TCP port when the announce-port is available and non-zero.
#
# These two options do not need to be used together, if only announce-ip is provided, Sentinel will declare the specified IP and the server port specified by the "port" option.
# If only announce-port is provided, Sentinel will notify the automatically detected local IP and the specified port.
#
# Example:
#
IP # sentinel announce - 2

# dir <working-directory>
# Each long-running process should have a well-defined working directory. For the Redis Sentinel, / TMP is its working directory.
dir /tmp

# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tell the Sentinel listener to specify the primary node and only determine its O_DOWN status if at least 
      
        sentinels are in agreement.
      
#
#
# Duplicates are discovered automatically, so you do not need to specify duplicates.
# Sentinel itself will override this configuration file, adding copies with other configuration options. Also note that the configuration file is overwritten when the replica is upgraded to the master replica.
#
Note: The name of the master node cannot contain special characters or Spaces.
# Valid characters can be a-z 0-9 and the three characters ".-_".Sentinel Monitor MyMaster 127.0.0.1 6379 2If redis has a password configured, then authentication must be configured, otherwise the switch cannot be automatically switched
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd

# sentinel down-after-milliseconds <master-name> <milliseconds>
#
If the primary node or replica does not respond to PING within a specified period of time, the node is considered to be offline.
#
# Default is 30 seconds
sentinel down-after-milliseconds mymaster 30000

# sentinel parallel-syncs <master-name> <numreplicas>
#
# How many replica nodes to synchronize data during failover
sentinel parallel-syncs mymaster 1

# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in a number of ways:
#
# - The time required to restart a failover after a previous failover has been tried by a given Sentinel against the same primary server and is twice the failover timeout.
#
# - When a slave synchronizes data from an incorrect master, it starts calculating the time. Until the slave is corrected to synchronize data to the correct master.
#
# - The time required to cancel failover that has been in progress but has not generated any configuration changes
#
# - Configure the maximum time required for all Slaves to point to the new master when failover is performed.
# Even after this timeout, slaves will still be configured to point to the master.
#
3 minutes by default
sentinel failover-timeout mymaster 180000

# Script execution
#
# Sentinel Notification-script and Sentinel reconfig-script Scripts used to configure calls to notify the system administrator or reconfigure the client after a failover.
The script executes with the following rules for error handling:
#
If the script exits with "1", it will be executed again later (the maximum number of retries is currently set to 10).
#
If the script exits with a value of "2" (or higher), execution will not be retried.
#
If the script terminates as a result of receiving a signal, the behavior is the same as exit code 1.
#
The maximum running time of the script is 60 seconds. When this limit is reached, the script terminates with SIGKILL and execution is retried.

# Notification script
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script (e.g. -sdown, -odown, etc.) for any Sentinel event generated at the warning level.
This script should notify the system administrator by email, SMS or any other messaging system that there is a problem with the Redis system being monitored.
#
Call the script with two arguments: the first is the event type and the second is the event description.
#
# This script must exist and be executable in order to start Sentinel when this option is provided.
#
#, for instance:
#
# sentinel notification-script mymaster /var/redis/notify.sh

# Customer reconfigure script
#
# sentinel client-reconfig-script <master-name> <script-path>
#
When the primary server changes due to failover, scripts can be invoked to perform application-specific tasks to notify the client that the configuration has changed and that the primary server address has changed.
#
# The following arguments will be passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# 
      
        Is currently always failover" failover"
      
# 
      
        is "leader" or "observer"
      
#
# Parameter from-ip, from-port, to-ip, to-port is used to pass the old address of the primary server and the new address of the selected replica.
#
#, for instance:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

# safe
# Avoid script reset, default is yes
# SENTINEL SET will not be able to change the notification-script and client-reconfig-script at run time by default.
# This avoids a simple security issue where the client can set the script to anything and trigger failover in order to execute the program.
sentinel deny-scripts-reconfig yes

Rename the REDIS command
#
#
In this case, Sentinel can be told to use a different command name than the normal command name.
For example, if the master "mymaster" and the related replica "CONFIG" are all renamed to "GUESSME", I can use:
#
# SENTINEL rename-command mymaster CONFIG GUESSME
#
Every time Sentinel uses CONFIG, it will use GUESSME. Note that there is actually no need to respect the command case, so writing "config guessme" in the above example is the same.
#
The # SENTINEL SET can also be used to perform this configuration at runtime.
#
To set the command back to its original name (undo renaming), rename the command to itself:
#
# SENTINEL rename-command mymaster CONFIG CONFIG
Copy the code