The installation

Single machine installation

  • docker install
docker run -d -p 6379:6379 --name redis-redisbloom redislabs/rebloom:latest
Copy the code
  • manual install
Yum -y install GCC automake autoconf libtool make vim wget http://download.redis.io/releases/redis-5.0.5.tar.gz tar ZXF Gz -c /soft/ Servers CD /soft/ Servers /redis-5.0.5 make make install./utils/install_server.shCopy the code

Persistent configuration

RDB trigger

  • Manual trigger:

Manual triggering is divided into blocking triggering of the save command and non-blocking triggering of the BGsave command. Save triggers foreground blocking. Mainly used in server shutdown maintenance. Bgsave triggers the fork() command to create a child process that asynchronously writes to disk.

  • Config file setting rules trigger:

Find the SNAPSHOTTING module in the CONF configuration file,

save 900 1
save 300 10
save 60 10000The bgsave command is triggered, respectively900The operands in seconds are reached1Time,300The operands in seconds are reached10Time,60The operands in seconds are reached10000Triggers bgSave.Copy the code

Aof configuration

Aof configuration in conf file:

Appendonly no: Aof is disabled by default. You can change it to appendonly yes. Appendfilename "appendone. aof" : default filename. Appendfsync Everysec: The IO level defaults to flush every second. At most one buffer is lost. Appendfsync always: The IO level is flush on every write. Appendfsync no: The IO level is Redis and does not actively flush until the kernel's buffer is full. At most, a single buffer. No-appendfsync-on-rewrite no: Do not actively call Flush () in an IO contention while rewriting or RDB occurs. Turn on yes if data sensitivity is high. Auto-aof-rewrite-percentage 100: indicates that the percentage of the volume size after the last rewrite is 100%, which changes from 64 to 128. Auto-aof -rewrite-min-size 64mb: the aof file automatically rewrites when it reaches 64mb.Copy the code

The cluster

Primary and secondary replication clusters

Replicaof can be used in slave to follow the master. The master provides full read and write services, and the slave provides read services. If the slave hangs, it can be restarted directly. If there are previous records of following the master, then full RDB will not occur. However, full RDB transmission must occur when AOF is enabled.

Help replicaof Replicaof 172.17.0.4 6379Copy the code

Redis-server /etc/redis/6379.conf –replicaof 172.17.0.4 6379 Conf –replicaof 172.17.0.4 6379 –appendonly yes 172.17.0.4 sends the full RDB to the local redis-server /etc/redis/6379.conf –replicaof 172.17.0.4 6379 –appendonly yes

Configure primary/secondary replication in the configuration file:

  • replicaof

Configure the IP address and port of the master

  • masterauth

Password for accessing master

  • replica-serve-stale-data yes

When the slave starts, if the amount of data in the master is large, the old data in the slave is exposed during data transmission. If the value is NO, the slave can provide services only after synchronization.

  • replica-read-only yes

Yes indicates that salve is read-only. No indicates that slave supports writing.

  • repl-diskless-sync no

If yes, the RDB is sent directly over the network.

  • repl-backlog-size 1mb

The master maintains a message queue cache of data temporarily written to it. If the salve is suspended and started again, the master may have a data increment. The slave can retrieve data from the master using an RDB file or give the master an offset. 1mb is set to set the size of the queue. If the master has a large traffic volume and the data corresponding to the offset given by the slave is crowded out, the slave cannot recover the data. In this case, a full RDB is triggered.

  • min-replicas-to-write 3

Three healthy slaves are required for the master to write successfully.

  • min-replicas-max-lag 10

A slave is considered healthy only when its latency is less than min-Replicas-max-lag seconds

The guard mechanism

  1. Look for three machine configurations
Port 26379 Sentinel Monitor MyMaster 172.17.0.4 6379 2# Specifies the quorum quorum. Two of three machines are requiredCopy the code
  1. Three machines start sentry

redis-server 26379.conf –sentinel

#The boot process
# cat 26379.log63: X 09 Jan 2021 15:58:54. 620 # Sentinel ID is c39a7cb614dc1f38394455c4ff21c69efa9081cd 63: X 09 Jan 2021 15:58:54. 620 # +monitor master mymaster 172.17.0.4 6379 Quorum 2 63:X 09 Jan 2021 15:58:54.622 * +slave slave 172.17.0.5:6379 172.17.0.5 6379 @myMaster 172.17.0.4 6379 63:X 09 Jan 2021 15:58:55.335 * + Sentinel sentinel 6379 63 26379 @ 7 c23231eea5562ae404c6bc1fbdfe92e2fcf16aa 172.17.0.3 mymaster 172.17.0.4: X 09 Jan 2021 15:58:55. 559 * 1 + sentinel sentinel fd38d8910b6a7a5ecff59422f5a11bfb4266913 172.17.0.4 26379 @ mymaster 172.17.0.4, 6379, 63:09 Jan 2021 X 16:00:35.141 * +slave slave 172.17.0.3:6379 172.17.0.3 6379 @myMaster 172.17.0.4 6379
# cat 26379.conf
port 26379
sentinel myid c39a7cb614dc1f38394455c4ff21c69efa9081cd
# Generated by CONFIG REWRITEDir "/soft/ Servers /redis" protected-mode no sentinel deny-scripts-reconfig yes sentinel monitor mymaster 172.17.0.4 6379 2 Sentinel config-epoch mymaster 0 Sentinel leader-epoch mymaster 0 Sentinel known- Replica myMaster 172.17.0.5 6379 Sentinel Known-replica myMaster 172.17.0.3 6379 Sentinel Known-sentinel mymaster 172.17.0.4 26379 1 fd38d8910b6a7a5ecff59422f5a11bfb4266913 sentinel known - sentinel mymaster 172.17.0.3 26379 7c23231eea5562ae404c6bc1fbdfe92e2fcf16aa sentinel current-epoch 0

#Master downtime process
# kill9 -$master_pid
# cat 26379.conf[root@b2c2f37cb93b redis]# 68:X 09 Jan 2021 16:10:01.360 # +sdown Master myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.414 # +odown Master MyMaster 172.17.0.46379 # Quorum 2/2 68:X 09 Jan 2021 16:10:01.414 # +new-epoch 1 68:X 09 Jan 2021 16:10:01.414 # +try-failover master myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.415 # +vote-for-leader C39a7cb614dc1f38394455c4ff21c69efa9081cd 1, 68: X 09 Jan 2021 16:10:01. 420 # 1 fd38d8910b6a7a5ecff59422f5a11bfb4266913 Voted for c39a7cb614dc1f38394455c4ff21c69efa9081cd 1:68 X 09 Jan 2021 16:10:01. 422 # 7c23231eea5562ae404c6bc1fbdfe92e2fcf16aa voted for c39a7cb614dc1f38394455c4ff21c69efa9081cd 1 68:X 09 Jan 2021 16:10:01.469 # + election-leader master myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.469 # + fail-state-select-slave master mymaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.537 # +selected-slave slave 172.17.0.5:6379 172.17.0.5 6379 @myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.537 * + fail-state-send-slaveof -noone slave 172.17.0.5:6379 172.17.0.5 6379 @mymaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.601 * + fail-state-wait-promotion slave 172.17.0.5:6379 172.17.0.5 6379 @myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.607 # + Promoted -slave slave 172.17.0.5:6379 172.17.0.5 6379 @myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.607 # + Fail-state-reconf-Slaves Master MyMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:01.665 * +slave-reconf-sent slave 172.17.0.3:6379 172.17.0.36379 @mymaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:02.579 # -odown master mymaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:02.636 * +slave-reconf-inprog slave 172.17.0.3:6379 172.17.0.36379 @myMaster 172.17.0.4 6379 68:X 09 Jan 2021 16:10:02.636 * +slave-reconf-done slave 172.17.0.3:6379 172.17.0.36379 @myMaster 172.17.0.46379 68:X 09 Jan 2021 16:10:02.712 # +failover-end master mymaster 172.17.0.46379 68:X 09 Jan 2021 16:10:02.712 # +switch-master myMaster 172.17.0.4 6379 172.17.0.5 6379 68:X 09 Jan 2021 16:10:02.713 * +slave slave 172.17.0.3:6379 172.17.0.3 6379 @myMaster 172.17.0.5 6379 68:X 09 Jan 2021 16:10:02.713 * +slave slave 172.17.0.4:6379 172.17.0.4 6379 @myMaster 172.17.0.5 6379 68:X 09 Jan 2021 16:10:32.737 # +sdown slave 172.17.0.4:6379 172.17.0.4 6379 @myMaster 172.17.0.5 6379
# cat 26379.conf
port 26379
sentinel myid c39a7cb614dc1f38394455c4ff21c69efa9081cd
# Generated by CONFIG REWRITEDir "/soft/ Servers /redis" protected-mode no sentinel deny-scripts-reconfig yes sentinel monitor mymaster 172.17.0.5 6379 2 Sentinel config-epoch mymaster 1 Sentinel leader-epoch myMaster 1 Sentinel Known - Replica myMaster 172.17.0.3 6379 Sentinel Known-replica myMaster 172.17.0.4 6379 Sentinel Known-sentinel mymaster 172.17.0.3 26379 7 c23231eea5562ae404c6bc1fbdfe92e2fcf16aa sentinel known - sentinel mymaster 172.17.0.4 26379 1 fd38d8910b6a7a5ecff59422f5a11bfb4266913 sentinel current - epoch 1 127.0.0.1:6379 > PSUBSCRIBE * Reading messages... (press Ctrl-C to quit) 1) "psubscribe" 2) "*" 3) (integer) 1 1) "pmessage" 2) "*" 3) "__sentinel__:hello" 4) "172.17.0.3, 26379, 7 c23231eea5562ae404c6bc1fbdfe92e2fcf16aa, 1, mymaster, 172.17.0.5, 6379, 1" 1) "pmessage" 2) "*" (3) "__sentinel__ : hello" 4) "172.17.0.5, 26379, c39a7cb614dc1f38394455c4ff21c69efa9081cd, 1, mymaster, 172.17.0.5, 6379, 1" 1) "pmessage"Copy the code

partition

cluster

  • Use the create-cluster script

/create-cluster create // Create a cluster./create-cluster stop // Stop a cluster and node./create-cluster clean // Clean up data files

# cd utils/create-cluster
# ./create-cluster start
# ./create-cluster create
#/ soft/servers/redis/redis - cli - cluster info 127.0.0.1:30001127.0.0.1:30001 (df9ce37f...). - > 0 keys 5461 slots | | 1 slaves. 127.0.0.1:30002 (f446380f...). - > 0 keys 5462 slots | | 1 slaves. 127.0.0.1:30003 (fb288d39...). 5461 slots - > 0 keys | | 1 slaves. [OK] 0 keys in 3 masters. 0.00 keys per slot on business.Copy the code
  • Manually create

For manually started Redis instances, cluster-enabled yes must be enabled in the configuration file

Redis-cli --cluster create 127.0.0.1:30001 127.0.0.1:30002 127.0.0.1:30003 127.0.0.1:30004 127.0.0.1:30005 127.0.0.1:30006 --cluster-replicas 1 redis-cli --cluster reshard 127.0.0.1:30001 redis-cli --cluster info 127.0.0.1:30001 redis-cli --cluster check 127.0.0.1:30001Copy the code

twemproxy

git clone https://github.com/twitter/twemproxy.git
yum install automake   libtool -y
cd twemproxy
autoreconf -fvi
./configure
make
cd scripts/
cp nutcracker.init /etc/init.d/twemproxy
cd /etc/init.d/
chmod +x twemproxy  
mkdir  /etc/nutcracker/
cd  twemproxy /conf
cp ./*     /etc/nutcracker/
cd  twemproxy/src
cp nutcracker  /usr/bin
cd /etc/nutcracker
cp nutcracker.yml nutcracker.yml.back
vi nutcracker.yml
service twemproxy start
Copy the code

nutcracker.yml

alpha:
  listen: 127.0. 01.: 22121
  hash: fnv1a_64
  distribution: ketama
  auto_eject_hosts: true
  redis: true
  server_retry_timeout: 2000
  server_failure_limit: 1
  servers:
   - 127.0. 01.: 6379:1
   - 127.0. 01.: 6380:1
Copy the code

The client directly connects to port 22121

predixy

mkdir predixy cd predixy/ wget https://github.com/joyieldInc/predixy/releases/download/1.0.5/predixy-1.0.5-bin-amd64-linux.tar.gz CD predixy/vi Predixy.conf GENERAL module: Bind 127.0.0.1:7617 to the server module: Include sentinel.confCopy the code

sentinel.conf

SentinelServerPool {
    Databases 16
    Hash crc16
    HashTag "{}"Distribution modula MasterReadPriority 60 StaticSlaveReadPriority 50 DynamicSlaveReadPriority 50 RefreshInterval 1 ServerTimeout 1 ServerFailureLimit 10 ServerRetryTimeout 1 KeepAlive 120 Sentinels {+ 127.0.0.1:26379								// IP+ ports of the three sentinels
        + 127.0. 01.:26380
        + 127.0. 01.:26381
    }
    Group ooxx {						// A set of sentries can monitor multiple sets of master and slave, and a Group is a set of master and slave.
    }									// The data written by the Client is split and stored in the master of the OOXX master and the master of the XXOO master.
    Group xxoo {
    }
}
Copy the code

Configure the sentry

Vi 26379.conf port 26379 Sentinel Monitor OOxx 127.0.0.1 36379 2 // Master ID+ port of the first set of primary/secondary OOxx, Ooxx Sentinel Monitor XXOO 127.0.0.1 46379 2 // Second set of master ID+ Port group xxOO VI 26380.conf Port 26380 Sentinel Monitor ooxx 127.0.0.1 36379 2 Sentinel Monitor xxoo 127.0.0.1 46379 2 vi 26381.conf port 26381 Sentinel monitor OOxx 127.0.0.1 36379 2 Sentinel Monitor XXOO 127.0.0.1 46379 2
#Start the sentry
redis-server 26379.conf --sentinel
redis-server 26380.conf --sentinel
redis-server 26381.conf --sentinel

#Create four folders 36379, 36380, 46379 and 46380 and start corresponding Redis respectively

redis-server --port 36379
redis-server --port 46379

#Start two slave machines and follow the corresponding hostRedis-server --port 36380 --replicaof 127.0.0.1 36379 Redis-server --port 46380 --replicaof 127.0.0.1 46379
#Go to the root directory of the agent predixy./predixy .. /conf/predixy.conf
#Client connection
redis-cli -p 7617
Copy the code

The appendix

Redis is configured by default

bind 127.0.0.1
protected-mode yes
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile /var/log/redis_6379.log
databases 16
always-show-logo yes
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/6379
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
Copy the code

use

Five data structures

string

#Check the value type of k1
type k1
#For strings, there can be computed types, where int refers to the encoding type of k2 value instead of k2 valuetype
object encoding k2
Copy the code

String manipulation

Set k1 value1 xx #xx append k1 "world" # append k1 "world" k3 Mget k1 k2 # obtain two values msetnx k2 c k3 d # Setrange k1 3 XXX # get string length from index strlen k1 # getSet k1 v1New # set the new value and return the old value (this can save one IO communication)Copy the code

Numerical operations

Incrby k2 12 DECr k2 decrby k2 22 incrbyfloat k2 0.5Copy the code

Bitmap operations

#BITPOS key bit [start] [end] Returns the position of the first bit in the bitmap whose value is bit.127.0.0.1:6379> SETBIT bits 3 1 # 1000 (integer) 0 127.0.0.1:6379> BITPOS bits 0 (integer) 0 127.0.0.1:6379> BITPOS bits  1 (integer) 3#Bit operation, and or not
setbit k1 1 1
setbit k1 7 1
get k1
setbit k2 1 1
setbit k2 6 1
#If the and operation has 0, 0, all 1 is 1
#01000000
bittop and andkey k1 k2
#Or operation has 1, then 1, all 0 is 0
#01000011
bittop or orkey k1 k2
Copy the code

list

#A stack can be implemented using the List type:Lpop k1 # e d c b a
#A queue can be implemented using the List type:Lpush k1 a b c d e # rpop k1 a b c d e #
#Get all elements between a range in the List (negative index support)LRANGE k1 0 -1Copy the code

hash

hset sean name 'minato aqua' hmset sean age 18 address hovelive hget sean name hget sean address hmget sean name age Hkeys Sean hvals Sean hgetall Sean HincrByFloat Sean Age 0.5 HincrByFloat Sean AGE-1Copy the code

set

#setFeatures: Disorder + weighthelp @set sadd k1 tom sean peter ooxx tom xxoo smembers k1 srem k1 xxoo ooxx smembers k1 sadd k2 1 2 3 4 5 sadd k3 4 5 6  7 8 sinter k2 k3
#Insert into dest
sinterstore dest k2 k3

sunion k2 k3
#Same as above, there is rstore, union
sdiff k2 k3
sdiff k3 k2

spop k1
Copy the code

sortedset

#How do you want them sorted? The name? The size? The price?
help @sorted_set
 
zadd k1 8 apple 2 banana 3 orange
#The size of the physical memory does not change with commands
#To view
zrange k1 0 -1
#See the score
zrange k1 0 -1 withscores
#Take the range of points
zrangebyscore k1 3 8
#Positive take
zrange k1 0 1
#Reverse take
zrevrange k1 0 1
#Score by value
zscore k1 apple
#Remove the top
zrank k1 apple
#To see all
zrange k1 0 -1 withscores
#increase2.5 banana zincrby k1#Sort updates based on changes in real time
zrange k1 0 -1 withscores
#Set takes union
zadd k1 80 tom 60 sean 70 baby
zadd k2 60 tom 100 sean 40 yiming
zunionstore unkey 2 k1 k2
zrange unkey 0 -1 withscores
#Set weightZunionstore unkey1 2 K1 K2 weights 1 0.5#Constant in K1, times 0.5 in k2
zrange unkey1 0 -1 withscores
#Take one of the biggest
zunionstore unkey2 2 k1 k2 aggregate max
zrange unkey2 0 -1 withscores
Copy the code

Release subscription

help @pubsub
PUBLISH test hello
SUBSCRIBE test
Copy the code

pipeline

The client and server are connected over a network. The connection can be fast (loopback interface) or slow (establishing a multiple hop network connection). Regardless of network latency, packets can always arrive from the client to the server and return data from the server to the client. This Time is called RTT (Round Trip Time). It’s easy to see how this can affect performance when a client needs to perform multiple requests in a batch (for example, adding many elements to the same list, or populating a database with many Keys). For example, if the RTT time is 250 ms (over a slow connection), even if the server can handle 100K requests per second, we can only handle a maximum of 4 requests per second.

A one-time request/response server can be implemented to process new requests even if the old one has not yet been responded to. This allows you to send multiple commands to the server without waiting for a reply, which is finally read in one step.

(printf "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379
+PONG
+PONG
+PONG
Copy the code

The transaction

MULTI, EXEC, DISCARD, and WATCH are Redis transaction related commands. Transactions can execute more than one command at a time with two important guarantees:

  • A transaction is a single isolated operation: all commands in the transaction are serialized and executed sequentially. The transaction will not be interrupted by command requests from other clients during execution.
  • A transaction is an atomic operation: all or none of the commands in a transaction are executed.

The EXEC command is responsible for firing and executing all commands in the transaction:

  • If the client starts a transaction using MULTI and fails to execute EXEC because of a disconnect, none of the commands in the transaction will be executed.
  • On the other hand, if the client successfully executes EXEC after the transaction is started, all commands in the transaction will be executed.

Watch monitors a key in a transaction and does not execute the transaction if it changes

> MULTI
OK
> INCR foo
QUEUED
> INCR bar
QUEUED
> EXEC
1) (integer) 1
2) (integer) 1
Copy the code

Why do redis transactions not support rollback?

  • Redis command will only failed because of wrong grammar (and these problems cannot be found in the team), or command on the wrong type of button above: that is to say, from a practical point of view, failure command is caused by programming errors, and these errors should be in the process of development was found, and should not appear in a production environment.
  • Because rollback support is not required, Redis can be kept simple and fast internally

RedisBloom module – Bloom filter

Github.com/RedisBloom/…

redis-server –loadmodule /path/to/redisbloom.so

Bf. Madd # Add multiple elements at once bf. Mexists # add multiple elements at once bf#There are two values in Redis that determine the accuracy of bloom filters:
#Error_rate: The error rate allowed for a Bloom filter. The lower the value, the larger the size of the filter's bit array and the more space it takes up.
#Initial_size: The number of elements that the Bloom filter can store. When the number of elements stored exceeds this value, the accuracy of the filter decreases.
#There is a command in Redis to set both values:Bf. Reserve the test 0.01 100#The first value is the name of the filter.
#The second value is the value of error_rate.
#The third value is the value of initial_size.
Copy the code

overdue

redis> SET cache_page "www.google.com"
OK
redis> EXPIRE cache_page 30  Set the expiration time to 30 seconds
(integer) 1
redis> TTL cache_page    # Check the remaining time
(integer) 23
redis> EXPIRE cache_page 30000   # Update expiration time
(integer) 1
redis> TTL cache_page
(integer) 29996
Copy the code