Writing in the front

  • See JAVA- Part 11 -Redis for other content

Redis start

  • The server port is changedredis-server --port 6380
  • Client connectionredis-cli -p 6380
  • Specifies the IP address of the client-h

Viewing configuration Files

  • directory/usr/local/ext
  • The configuration fileredis.conf
  • Start from the configuration file
/ / a copy cat redis. Conf | grep -v "#" | grep -v "^ $" > redis - 6379. The conf / / configuration content, modify the configuration file port 6379 / / port daemonize no start / / the background, Logfile "" // logfile. Do not display in the console dir /usr/local/etc/data // stored folder // start redis-server redis-6379.conf // copy the modified configuration content cp redis-6379.conf redis-6380.confCopy the code

Basic Server Configuration

  • Set up the server to run as a daemondaemoniz yes|no
  • Binding host AddressThe bind 127.0.0.1
  • Setting the number of databasesdatabases 16
  • Set portport 6370

The log configuration

  • Set up the server to specify the logging levelLoglevel debug information (process) | verbose (default) | notice | warning
  • Log file namelogfile port.log

Client Configuration

  • Set the maximum number of client connectionsmaxclients 00 means no limit
  • The connection is closed when the maximum duration is reachedThe timeout (in seconds) 3000 Disable the function

Quick configuration of multiple servers

  • The specified configuration file is imported and loadedinclude /path/server-port.conf, similar to inheritance

persistence

  • The operation of storing data in a permanent storage medium, such as a hard disk, and restoring the stored data at a specified time
  • Data snapshot RDB/ process log AOF

RDB

  • The storage efficiency is high. The data stored at a certain point in time is fast and is usually used for disaster recovery
  • Will lose data, background execution, consumption of child processes; Redis version RDB format version is not unified
  • save
  • Generate the filedump.rdb
  • File saved in/usr/local/etc/data
  • configuration
RDB // Whether to compress rdbCompression yes // RDBchecksum yesCopy the code
  • Save blocks the current server and is not recommended for online use

The background to perform

  • bgsave, recommended use
  • callforkFunction generates a child process to save
  • The saved results are returned to the console/log file
Background saving terminated with success
Copy the code
  • If an error occurs in the background storage procedure, the function is enabled by default
stop-writes-on-bgsave-error yes
Copy the code

automated

  • Influence the data; Made a real impact; Without data comparison, an impact quantity is produced
  • The underlying callbgsave
  • Setting by configuration
//second Specifies the monitoring time range. Changes Monitor key changes save second changesCopy the code
  • Log information
2 changes in 10 seconds. Saving...
Background saving started by pid 20409
DB saved on disk
Background saving terminated with success
Copy the code

Special launch

  • Full replication, master/slave replication
  • The server is being restarted. Proceduredebug reload
  • Specifies to save data when shutting down the servershutdown save

AOF

  • Record the operation process to eliminate the risk of data loss
  • Append only file, which is recorded as a separate log
  • First put the operation command inAOP writes commands to flush the cacheSynchronizes commands to an AOP file after a certain stage
  • Three strategiesappendfsync
Always Every write operation is synchronized to the AOF file, data zero error, poor performance Everysec per second, data quasi-certainty is high, high performance, but sudden downtime loss of one second of data NO System control is controlled by the operating systemCopy the code
  • The principle of

use

  • configuration
// Enable AOF persistence, Default is not open appendonly yes | no / / AOF write data strategy appendfsync always | everysec | no / / change the file name appendfilename filename/dir/folderCopy the code

rewrite

  • Data collation improves disk utilization and reduces persistence time
  • Timeout data is not written, instructions that do not affect the result are ignored, and multiple instructions are merged into one instruction
  • Manual overridebgrewriteaofWhen rewritten, it will be stored in the same way as RDB
  • Automatic override, configuration
// Aof_current_size > // aof_rewrite-precentage // aof_current_size > auto-aof-rewrite-min-size aof_cuuent_size - aof_base_size / aof_base_size >= auto-aof-rewrite-precentageCopy the code

Difference between RDB and AOF

The transaction

  • Open the transactionmulti, and the subsequent instruction returnsQUEUED
  • Perform transactionsexecExecute all instructions
  • After multi, all subsequent instructions are added to the transaction, and when exec is encountered, the transaction is executed
  • Cancel the transactiondiscard
  • If the following commands have syntax errors, the other commands will not exist
  • As long as it is syntactically correct but cannot be executed correctly, a command that runs incorrectly will not be executed

The lock

  • Monitoring datawatch key1 key2...
  • Cancel the monitoringunwatch
  • Once the data monitored by watch is changed, if the following transaction is started and not executed, it cannot be executed

A distributed lock

  • Public locksetnx lock-key value, using the return value characteristics of the setnx command, if there is a value, the return setting fails, and if there is no value, the setting succeeds. Now that I’m done with the data,del lock-key
  • Through the protocol, before the operation of data, first operationlock-keyIf you can do it, do it. If you can’t, don’t do it
  • Time the lock in case you forget to unlock itexpire lock-key seconds

Deletion policy

  • Outdated data is retained in Redis
  • In the memory, create a hash space for expired data and manage expired data

Time to delete

  • When the time comes, the CPU handles the deletion, trading processor performance for storage (time for space)

Lazy to delete

  • The data is not processed at that time. If the data expires the next time you access it, the system clears it and returns that it does not exist
  • expireIfNeeded() 在get keyWhen performing
  • Save CPU performance, high memory pressure, exchange storage space for processor performance (space for time)

Periodically delete

  • When Redis starts, the configuration is readserver.hzThe default value is 10, executed per secondserver.hztimeserverCron()->databasesCron()->activeExpireCycle()
  • activeExpireCycle()For each libraryexpiresCheck one by one and execute each time250ms/server.hzTo a certainexpiresDuring the detection, W keys are randomly selected for detection. If the key times out, the key is deleted. If the number of keys deleted in a round >W*25%, prove that the expiration data is more, in the test; After passing, check the next one
  • W = valueACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOPProperty value, set in the configuration file
  • current_dbSpecial recordactiveExpireCycle()Into which libraryexpiresSpace to perform
  • Fixed CPU resources per second Maintenance memory, random check, key spot check

Out of algorithm

  • Redis uses memory to store data and is called before each instruction is executedfreeMemoryIfNeeded()Check whether the memory is sufficient. If the minimum storage requirements for newly added data are not met, delete some data temporarily according to the eviction algorithm
  • Maximum available memorymaxmemory, the proportion of the physical memory. The default value is 0 and there is no limit
  • Select the number of data to be deleted each timemaxmemory-samples, using the method of randomly selecting data
  • Deletion policymaxmemory-policy XXXXXX
  • lruFrom the present use of time, eliminate the longest

A master-slave replication

  • Several Redis are connected together to ensure data synchronization, realize high availability, and realize data redundancy backup
  • Master provides data; Slave Receives data

  • The core job is to copy master data to slave
  • A master is responsible for multiple slaves, and one slave corresponds to only one master
  • Master only writes, slave only reads
  • Master is down and some slaver acts as master
  • Master/slave replication implements read/write separation, load balancing, fault recovery, data redundancy, and high availability
  • The slave connects to the master, and the master synchronizes data repeatedly to the slave

Establish connection phase

  • Slave Sends the IP address and port number of the master
Slaveof IP port // The client directly sends the command redis-server-slaveof IP port // The server starts the command slaveof IP port //slave machine configuration //6380 connects to 6379 slaveof 127.0.0.1 6379 //slave Slaveof no oneCopy the code
  • Permission to verify
//slave auth password masterauth password // configuration file redis-cli -a password // Requirepass password // Master requirepass password Config set requirepass password // Run the config get requirepass command to set the passwordCopy the code
  • Finally, the slave stores the address and port of the master. The master saves the slave port. The socket to which the connection is set up

Data synchronization phase

  • Data synchronization is performed during peak traffic hours

  • Requesting data synchronization
  • Create an RDB to synchronize data
  • Restore RDB data synchronization
  • Request partial data synchronization. Partial data refers to the data generated during full replication, also known as incremental replication
  • Restore some synchronized data
  • In the final state, slave has all the data of the master, including the data received during RDB. The master saves the current synchronization position of the slave. The next synchronization starts from the end of the previous one
  • Master copy buffer to avoid data overflow 2The average timeTotal data
repl-backlog-size 1mb
Copy the code
  • Slave only supports reads and disables writes
slave-server-stable-date yes|no
Copy the code

Command propagation phase (constant synchronization)

  • Server run ID, the identification code for each server run, 40 characters, is a random hexadecimal character; Used for transmission during the service, identification, used to deal with whether a machine, whether data synchronization;info serverTo viewrunid; When the master connects to the slave for the first time, it sends its running ID to the slave. The slave saves the ID and carries it with each update. If the id is different, the previous master is not the same master
  • The replication buffer is a first-in, first-out queue used to store commands executed by the server. Each time a command is propagated, the master records the command propagated and stores it in the replication buffer. A command, stored as aof, stores byte values in character units and is given a number (offset) for identification; The offset is used to distinguish the difference between the current data transmission of different slaves
  • The copy offset, a number describing the instruction byte position in the skin buffer; Master, which records instruction byte offsets sent to all slaves once; Slave: records the byte offset of the instruction sent by the master. Function, synchronize data, compare differences

heartbeat

  • During command transmission, the master and slave exchange information and use the heartbeat mechanism for maintenance to keep them online

  • The master processes the policy and sends it through the slaveREPLCONF ACKCommand to determine
// If the number of slaves is less than two or the delay of all slaves is greater than or equal to 10 seconds, the master write function is forcibly disabled and data synchronization is stopped. Min-rabes-to-write 2 min-rabes-max-lag 8Copy the code

Total workflow

Some of the problems

  • Set a reasonable timeout period
repl-timeout
Copy the code
  • Ping command send frequency, determine the connection
repl-ping-slave-period
Copy the code
  • If the delay is too large, multiple slaves may obtain the same data asynchronously, and data access to one slave is temporarily blocked. After this function is enabled, it only responds to a few commands, such as info and slaveof
slave-serve-stale-data yes|no
Copy the code

The sentry

  • Resolve master downtime
  • Select one of the slaves as master, notify all slaves to connect to the new mater, and start the new master and slave
  • Sentinel, distributed system, for each server in the master-slave structuremonitoringWhen the master is down, send to the sentinels and clientsnotice, select a new master, connect all slaves to the new master, and tell the client the new server addressAutomatic failover
  • Sentinel is also a Redis server that does not provide data services and has an odd number of configurations
  • configurationsentinel.confThe configurations are the same, but the ports are different
Sed 's/26379/26381/g' sentinel-26379.conf > sentinel-26381.confCopy the code
  • Start the sentryRedis-sentinel sentinel- Port id. conf
  • The whole process takes place in the background

monitoring

  • Synchronization information
  • Synchronize the status information of each node to obtain (ping) Sentinel status, (info)master status (RUNID, role, and slave information), and (info)slave information

notice

  • Keep the unicom

failover

  • Master offline

  • Select a sentinel responsible for selecting the new master

  • Select the new master, select the online, responsive, short disconnect from the original master, priority (priority, offset, rUNId)
  • Send to the new masterslaveof no oneDisconnect from the original master
  • Send to other slavesSlaveof new masterIP and portTo establish a new connection

The cluster

  • Use the network to connect a number of computers, and provide a unified management mode, so that the external presentation of a single service effect
  • Distribute the access pressure of a single server, load balancing; Decentralized storage pressure of a single server, scalability; Reduce service disasters caused by the failure of a single server

Redis cluster

  • Slot for storing keys
  • Through algorithm design, calculate the location of the key should be saved; Increase or decrease the number of servers and processing slots
  • Internal communication design, each server interconnect, each recording slot position; If the client receives a hit, it returns directly. If the client does not receive a hit, it goes to find the location directly

Cluster cluster

  • configuration
// Whether to set cluster cluster-enabled yes // Configuration file name cluster-config-file nodes-6379.conf // Timeout duration ms cluster-node-timeout 10000 // Minimum number of slave connections cluster-migration-barrier countCopy the code
  • Establish, requireredis-trib.rb, the configurationRuby and gemThe environment
Flushall cluster reset is used for each server before the flushall cluster reset command is executed. 1 indicates that a master is connected to a slave. Rb create --replicas 1 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382./redis-trib.rb create --replicas 1 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:127.0.0.1 6383:6384Copy the code
  • use
Redis -cli -c // Check the node status on the client cluster nodesCopy the code

Primary/secondary offline and primary/secondary switchover

  • It is received from the offline or reconnected master node and notified to other master nodes
  • The master node goes offline, and the slave node becomes an MSTER after the timeout period expires
  • The master node reconnects to the master node as the slave node

Enterprise-level solutions

Cache warming

  • Before the system starts, load the cache data directly to the cache system in advance to avoid querying the database and then caching when users request the cache data. Users should directly query the cache data prepared in advance

Cache avalanche

  • The amount of instant expiration data is too large, which causes pressure on the database server. Therefore, you need to monitor server running data to avoid centralized expiration time
  • In a short period of time, more keys in the cache will expire and ask the database for them
  • The database received a large number of requests that could not be processed in time, and Redis began to have a timeout due to a large number of requests being backlogged
  • The traffic surges and resources are occupied

Cache breakdown

  • A single key is hot or expired, resulting in a large number of instant visits. Redis is not hit, resulting in a large number of visits to the database

The cache to penetrate

  • Access data that does not exist
  • This data does not exist in the database and is not persisted in REIDS; Redis large area of missed, abnormal URL access

Performance Indicator Monitoring

  • Performance indicators

  • Memory specifications Memory

  • Basic activity

  • Persistence indicator

  • Error indicator Error

Monitoring commands

  • redis-benchmark
/ / c the number of connections - n number of requests redis - benchmark [-h] [p] - [-c] [-n < requests >] [k] / / with no arguments, Benchmark -C 50-n 100 redis-benchmark -C 50-n 100 redis-benchmark -C 50-n 100Copy the code
  • Displays server debugging informationmonitor, can be executed by one client and see statements executed by another client
  • slowlogSlowly Query logs and record commands that time out
Slowlog gets slow query logs/Number of log entries/Reset slowlog get/len/reset // Related configuration // Set slower query time offline microsecond slowlog-log-slower than 1000 // Sets the log display length of slow query commands slowlog-max-len 100Copy the code