Redis transactions

Transactions in Redis guarantee atomic execution of a batch of commands, that is, all commands are either executed or none are executed. No other commands are served during transaction execution and the integrity of transaction commands is guaranteed when Redis restarts to load AOF files

Redis transaction command

The command instructions
multi Show open transaction
exec Perform transactions
discard Cancel the transaction
watch key [key …] Listen for keys that might be changed,If the monitored key changes before exec, the transaction is cancelled
unwatch Cancel listening for keys that may be modified

Redis transaction example

Scenario 1: XM (Xiao Ming) has 1000 yuan and XH (Xiao Hong) has 1000 yuan. Xm needs to transfer 100 yuan to Xiao Hong. Xm has 900 yuan and XH has 1100 yuan

time The client 1
T1 127.0.0.1:6379 > flushall

OK
T2 127.0.0.1:6379 > set xm, 1000

OK
T3 127.0.0.1:6379 > set xh 1000

OK
T4 7.0.0.1:6379 > multi

OK
T5 27.0.0.1:6379 > decrby xm, 100

QUEUED
T6 127.0.0.1:6379 > incrby xh 100

QUEUED
T7 127.0.0.1:6379 > exec

1) (integer) 900

2) (integer) 1100
T8 127.0.0.1:6379 > get xm

“900”
T9 127.0.0.1:6379 > get xh

“1100”

Redis transaction cancelled

Scenario 1: XM prepares to spend 300 yuan (900 yuan minus 300 yuan is 600 yuan), but finally cancels (the result is still 900 yuan)

time The client 1
T1 27.0.0.1:6379 > multi

OK
T2 7.0.0.1:6379 > decrby xm, 300

QUEUED
T3 7.0.0.1:6379 > the discard

OK
T4 27.0.0.1:6379 > get xm

“900”

Scenario 2: XM has been monitored by Watch, and xM is ready to spend 300 yuan, and another client adds 100 yuan to XM

time The client 1 Client 2
T1 127.0.0.1:6379 > flushall

OK
T2 127.0.0.1:6379 > set xm, 1000

OK
T3 27.0.0.1:6379 > watch xm

OK
T4 127.0.0.1:6379 > multi

OK
T5 127.0.0.1:6379 > decrby xm, 300

QUEUED
T6 127.0.0.1:6379 > get xm

“1000”
T7 127.0.0.1:6379 > incrby xm, 100

(integer) 1100
T8 7.0.0.1:6379 > exec

(nil)
T9 7.0.0.1:6379 > get xm

“1100”

Redis transaction exception

An error occurred before exec execution

time The client 1
T1 127.0.0.1:6379 > flushall

OK
T2 127.0.0.1:6379 > multi

OK
T3 127.0.0.1:6379 > set xm, 1000

QUEUED
T4 127.0.0.1:6379 > set xh 3000

QUEUED
T5 127.0.0.1:6379 > hset xl age

(error) ERR wrong number of arguments for ‘hset’ command

// There is a syntax error
T6 127.0.0.1:6379 > exec

(error) EXECABORT Transaction discarded because of previous errors.

// Execute the transaction and find that the transaction has been cancelled and the instruction before the error instruction is invalid
T7 127.0.0.1:6379 > get xm

(nil)
T8 127.0.0.1:6379 > get xh

(nil)

Error after exec execution (runtime exception)

time The client 1
T1 127.0.0.1:6379 > flushall

OK
T2 127.0.0.1:6379 > multi

OK
T3 127.0.0.1:6379 > set xm, 1000

QUEUED
T4 127.0.0.1:6379> hset xm age 19

QUEUED
T5 127.0.0.1:6379 > exec

1) OK1)

2) (error) WRONGTYPE Operation against a key holding the wrong kind of value
T6 127.0.0.1:6379 > get xm

“1000”

// If the transaction is executed, the transaction will be executed successfully

Does this contradict the atomicity of Redis transactions? Why is rollback not supported?

Why Redis does not support roll backs?

​ If you have a relational databases background, the fact that Redis commands can fail during a transaction,==but still Redis will execute the rest of the transaction instead of rolling back==, may look odd to you.

However there are good opinions for this behavior:

​ 1.Redis commands can fail only if called with a wrong syntax (and the problem is not detectable during the command queueing), or against keys holding the wrong data type: this means that in practical terms a failing command is the result of a programming errors, and a kind of error that is very likely to be detected during development, and not in production. ​ 2.Redis is internally simplified and faster because it does not need the ability to roll back.

An argument against Redis point of view is that bugs happen, however it should be noted that in general the roll back does not save you from programming errors. For instance if a query increments a key by 2 instead of 1, or increments the wrong key, there is no way for a rollback mechanism to help. Given that no one can save the programmer from his or her errors, and that the kind of errors required for a Redis command to fail are unlikely to enter in production, we selected the simpler and faster approach of not supporting roll backs on errors.

There are two reasons why Redis transactions do not support rollback.

In practice, Redis errors do not occur in production environments. Redis errors only occur when the wrong parameter type is passed, and these errors can be found in development environments. In short: developers don’t intentionally run bugs in production, and should self-check after the program is written

Redis transaction source code analysis

The transaction start

The multi command only places a CLIENT_MULTI flag on the client structure that represents the connection of the command. Redis transactions cannot be nested, that is, the multi command cannot be invoked to open a new transaction within an open transaction.

  • The source code is as follows:
typedef struct redisClient{
  / /...
  multiState mstate; /*multi/exec state*/
  / /...} redisClient;void multiCommand(client *c) {
    if (c->flags & CLIENT_MULTI) {// If the multi command has already been executed, it cannot be executed again
      addReplyError(c,"MULTI calls can not be nested");
      return;      
		}
    c->flags |= CLIENT_MULTI; // The client structure is marked CLIENT_MULTI
    addReply(c,shared.ok);
 }
Copy the code

The command team

  1. If the client sends one of the four commands exec, discard, watch, or multi, the server executes the command immediately

  2. If the client sends a command other than exec, discard, watch, or multi, the server does not execute the command immediately. Instead, the server places the command in a queue and replies to the client QUEUED

  3. The processCommand function also checks whether the command exists, whether the number of arguments to the command meets the requirements, whether password verification is passed if enabled, and so on. If these checks fail, Redis still places a CLIENT_DIRTY_EXEC flag in the client structure. Exec command execution detects flags on the client side to determine the execution process

  • The source code is as follows:
int processCommand(client *c) {...// If the client has the CLIENT_MULTI flag and is not exec, discard, multi, or watch commands, the command is queued
	if(c->flags & CLIENT_MULTI && c->cmd->proc ! = execCommand && c->cmd->proc ! = discardCommand && c->cmd->proc ! = multiCommand && c->cmd->proc ! = watchCommand)// Queue
						queueMultiCommand(c);
            addReply(c,shared.queued);
     } else {
  		// Otherwise call the call commandcall(c,CMD_CALL_FULL); . }... }Copy the code
  • The execution flow chart is as follows:

Transaction queue

Each client has its own transaction state, which is stored in the mState property of the client state

typedef struct redisClient{
  / /...
  multiState mstate; /*multi/exec state*/
  / /...} redisClient;Copy the code

The transaction state contains a transaction queue and a counter for the queued commands (or the length of the transaction queue)

typedef struct multiState{
  // Transaction queue, FIFO order
  multiCmd *commands; 
  // Queue command count
  int cout; } multiState;Copy the code

A transaction queue is an array of type multiCmd. Each multiCmd structure in the array holds information about a queued command, including Pointers to its implementation function, command arguments, and the number of arguments

typedef struct multiCmd{
  / / parameters
  robj **argv; 
  // Number of arguments
  intArgv;// Command pointer} multiCmd;Copy the code

The transaction queue stores queued commands in a first-in-first-out (FIFO) manner, with the first queued commands placed at the front of the array and the last queued commands placed at the back

Perform transactions

When a client in transactional state sends an exec command to the server, the exec command is immediately executed by the server. The server iterates through the client transaction queue, executes all the commands stored in the queue, and returns all the results of executing the commands to the client

void execCommand(client *c) {...if(! (c->flags & CLIENT_MULTI)) {// Check for CLIENT_MULTI flags
        addReplyError(c,"EXEC without MULTI"); // The transaction is not started and an error is returned
				return; 
    }
		if (c->flags & (CLIENT_DIRTY_CAS|CLIENT_DIRTY_EXEC)) { 
      // Whether there are CLIENT_DIRTY_CAS and CLIENT_DIRTY_EXEC flags
			discardTransaction(c); // Discard transaction...
		}unwatchAllKeys(c); //unwatch all keys.// call each queue command in turncall(c,server.loading ? CMD_CALL_NONE : CMD_CALL_FULL); . }Copy the code

Give up the transaction

Use the discard command in Redis to explicitly discard a transaction. The discard command causes Redis to discard transactions opened with Multi. The return value is OK

When a transaction is abandoned, all enqueue commands will be cleared, flags related to the transaction will be cleared on the client, and all keys monitored will be unlistened

void discardCommand(client *c) {
       if(! (c->flags & CLIENT_MULTI)) {// Whether a transaction has been started
       	addReplyError(c,"DISCARD without MULTI");
    		return;
     }
}
discardTransaction(c); // Discard the transaction
addReply(c,shared.ok);/ / return ok
Copy the code

The WATCH command monitors database keys

The entire Redis service defaults to 0-15 databases. Each Redis database holds a watch_keys dictionary. The watch_keys dictionary is a database key monitored by the WATCH command, and the dictionary value is a linked list of all the clients that monitor the corresponding data key

typedef struct redisDb {
    dict *dict;
    dict *expires;
    dict *blocking_keys;
    dict *ready_keys;
    dict *watched_keys; // Watch key and corresponding client, mainly used for transactions
    int id;
    long long avg_ttl;
    list *defrag_later;
} redisDb;
Copy the code

WATCH Command monitoring mechanism is triggered

All commands that modify data, such as: Set, lpush, sadd, zrem, del, flushall, etc., all call the touchWatchedKey function after execution to check whether the watch_keys dictionary is being monitored by a client. If so, The touchWatchedKey function turns on the REDIS_DIRTY_CAS flag of the client monitoring the modified key, indicating that the transaction security of the client has been broken

void touchWatchedKey(redisDb *db, robj *key) {...// If the client does not have a key to monitor
	if (dictSize(db->watched_keys) == 0) return; 
  	// Check whether the changed key is in the listening state
		clients = dictFetchValue(db->watched_keys, key); 
  	if(! clients)return;
    listRewind(clients,&li);
    while((ln = listNext(&li))) {
			client *c = listNodeValue(ln);
			c->flags |= CLIENT_DIRTY_CAS; // Otherwise iterate through the list and set the flag CLIENT_DIRTY_CAS}... }Copy the code

Redis publish and subscribe

Redis’s publish-subscribe function decouples producers and consumers. Producers can send messages to a given channel without caring if there is a consumer or who the consumer is, while consumers can subscribe to a given channel and receive messages sent to that channel without caring who sent them

Redis issues subscription commands

The command instructions
subscribe channel [channel …] To subscribe to a channel, a channel is not created separately, but with the SUBSCRIBE channel command
unsubscribe [channel [channel …]] unsubscribe
publish Push message

Redis publishing subscription example

Scenario 1: Client 1 subscribes to channel-1, channel-2, channel-3. Client 2 advertises messages to channel-1

time The client 1 Client 2
T1 127.0.0.1:6379 > subscribe channel channel channel – 1-2-3

Reading messages… (press Ctrl-C to quit)

1) “subscribe”

2) “channel-1”

3) (integer) 1

1) “subscribe”

2) “channel-2”

3) (integer) 2

1) “subscribe”

2) “channel-3”

3) (integer) 3
T2 Hello 127.0.0.1:6379 > publish channel – 1

(integer) 1
T3 1) “message”

2) “channel-1” // Which channel the message comes from

3) “hello” // message content
T4 Nihao 127.0.0.1:6379 > publish channel – 2

(integer) 1
T5 1)”message”

2) “channel-2”

3) “nihao”
T6 127.0.0.1:6379 > publish channel – 3 Java

(integer) 1
T7 1) “message”

2) “channel-3”

3) “java”

Subscribe channels according to the pattern

Scene 2:

Channel: news-sport; news-music; news-weather

Client 1, subscribe to sport (*sport), client 2, subscribe to all news (* news), client 3, subscribe to weather (new-weather), client 4, publish information

time The client 1 Client 2 The client 3 The client 4
T1 127.0.0.1:6379 > psubscribe * * sport

Reading messages… (press Ctrl-C to quit)

1) “psubscribe”

2) “*sport”

3) (integer) 1		 127.0.0.1:6379 > psubscribe *news

Reading messages… (press Ctrl-C to quit)

1) “psubscribe”

2) “news“

3) (integer) 1		 127.0.0.1:6379 > psubscribe news – the weather

Reading messages… (press Ctrl-C to quit)

1) “psubscribe”

2) “news-weather”

3) (integer) 1
T2 127.0.0.1:6379 > publish news – sport Nike

(integer) 2
T3 1) “pmessage”

2) “*sport”

3) “news-sport”

4) “nike”		 1) “pmessage”

2) “news*”

3) “news-sport”

4) “nike”
T4 1) “pmessage”

2) “news*”

3) “news-music”

4) “liangbo”		 127.0.0.1:6379 > publish news – music liangbo

(integer) 1
T5 127.0.0.1:6379 > publish news – the weather is sunny

(integer) 2
T6 1) “pmessage”

2) “news*”

3) “news-weather”

4) “sunny		 1) “pmessage”

2) “news-weather”

3) “news-weather”

4) “sunny”

Redis Lua script

The Lua script embedded in Redis is powerful enough not only to run Lua code, but also to ensure that scripts are executed atomically: no other scripts or Redis commands are executed while the script is being executed. This semantics is similar to MULTI/EXEC, which means we can write Lua code with logical operations to be executed by the Redis server, which solves the problem of non-atomicity between Redis command execution

Redis Lua script command

  • eval script numkeys key [key …] arg [arg …]
attribute instructions
eval Indicates executing lua commands
lua-script Represents the contents of the Lua script executed
key-num Note that the key in Redis starts with 1. If there is no key parameter, write 0
key [key …] Key is passed to Lua as an argument, or it can be left blank, but it must correspond to the number of key-num
arg [arg …] Parameters passed to The Lua language can be left blank

Examples of Redis Lua scripts

time The client 1
T1 127.0.0.1:6379> eval “return ‘hello world’ 0

// Simply print hello world
time The client 1
T1 127.0.0.1:6379 > set k1 10

OK
T2 127.0.0.1:6379 > set k2. 3

OK
T3 127.0.0.1:6379> eval “local v1 = redis. Call (‘get’,KEYS[1]); return redis.call(‘incrby’,KEYS[2],v1)” 2 k1 k2 3

(integer) 13
T4 127.0.0.1:6379 > get k2

“13”

Scenario 1: Create the xxx.lua file and execute the Lua script

  1. Creates a Lua file with no parameters
  2. redis-cli –eval /usr/local/soft/lua-redis/lua1.lua
// Lua file contents redis.call('set'.'java'.'100');
return redis.call('get'.'java');
Copy the code
[root@localhost ~]# redis-cli --eval /usr/local/soft/lua-redis/lua1.lua 
"100"
Copy the code

Scenario 2: Create the xxx.lua file with parameters and execute the Lua script

  1. Create a Lua file, limit IP traffic, and check whether the IP address is accessed more than 10 times within 6s
  2. redis-cli –eval /usr/local/soft/lua-redis/lua-arg.lua
// To limit the IP flow of lua script content // count +1(If the key does not exist, you can create it.)local num = redis.call('incr',KEYS[1]);
if tonumber(num)==1 then// For the first access, set the expiration time with the first argument.'expire',KEYS[1],ARGV[1])
   return 1// If this is not the first time, compare with the second parameter to see if the limit is exceededelseif tonumber(num) >tonumber(ARGV[2])then 
   return 0/ / overrunelse
   return 1/ / not overrunend
Copy the code
[root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 1 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 0 [root@localhost ~]#Redis-cli --eval /usr/local/sof/lua-redis-lua-arg. lua app:127.0.0.1, 6 10 (integer) 0 [root@localhost ~]#Redis -cli --eval /usr/local/sof/lua-redis /lua-arg.lua app:127.0.0.1, 6 10 (integer) 0Copy the code

Redis Lua script cache

The command instructions
script load Redis returns a cache ID for the current directive
evalsha sha1 numkeys key [key …] arg [arg …] The eval command is executed in the same way as the eval command, except that the preceding script is replaced with the cached ID 
127.0.0.1:6379 > script load "return" hello wolrd '" "bd98d8093f7488b91dd2650ec646e0c127eeed29" 127.0.0.1:6379 > evalsha bd98d8093f7488b91dd2650ec646e0c127eeed29 0 "hello wolrd"Copy the code

Scenario 1: The ride case

local curVal = redis.call("get",KEYS[1]);
if cuVal == false then
  curVal=0
else 
  curVal = tonumber(curVal)
end;
curVal = curVal*tonumber(ARGV[1]);
redis.call("set",KEYS[1],curVal);
return curVal
Copy the code
127.0.0.1:6379> script load 'local curVal = redis. Call ("get",KEYS[1]); if curVal == false then curVal=0 else curVal = tonumber(curVal) end; curVal = curVal*tonumber(ARGV[1]); redis.call("set",KEYS[1],curVal); Return curVal '" 138375 d6d6b9a8dbe94c0c36fd37ac1b896c8669 "127.0.0.1:6379 > set num OK 127.0.0.1:6379 > evalsha 138375 d6d6b9a8dbe94c0c36fd37ac1b896c8669 num 6 12 127.0.0.1 (integer) : 1 6379 > get num "12"Copy the code

Redis Lua script is interrupted

The execution of lua scripts by Redis is exclusive, that is, no other scripts or Redis commands are executed while the script is executed.

Redis Lua script interrupt provides two commands:

  1. SCRIPT KILL
  2. SHUTDOWN NOSAVE

SCRIPT KILL

Scenario 1: Run the Lua script on client 1. Run the redis command on client 2

time The client 1 Client 2
T1 127.0.0.1:6379> eval ‘while(true) do end’ 0

// Emulated client 1 is executed without exit
T2 127.0.0.1:6379 > set name yy

(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
T3 // Then execute SCRIPT KILL as prompted

127.0.0.1:6379 > SCRIPT to KILL

OK
T4 (error) ERR Error running script (call to f_eec1f08dafc6bfdf256e3820d971514a3a24267e): @user_script:1: Script killed by user with SCRIPT KILL…

(103.37 s)

SHUTDOWN NOSAVE

Scenario 2: Client 1: Run the Lua script. Client 2: run the redis command

time The client 1 Client 2
T1 127.0.0.1:6379> eval “redis. Call (‘set’,’name’,’ Tom ‘) while(true) do end” 0

// Emulated client 1 keeps running, but there are modification instructions in the batch command
T2 127.0.0.1:6379 > set the name jack

(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
T3 127.0.0.1:6379 > SCRIPT to KILL

(error) UNKILLABLE Sorry the script already executed write commands against the dataset. You can either wait the script termination or kill the server in a hard way using the SHUTDOWN NOSAVE command.

// SCRIPT KILL cannot stop SCRIPT execution; Because the key: name value was changed earlier, the lua script had to SHUTDOWN the NOSAVE command to ensure atomicity of the batch command
T4 127.0.0.1:6379 > SHUTDOWN NOSAVE

// when SHUTDOWN is done, the redis service is SHUTDOWN, so this is a destructive command for the online environment

not connected> exit
T5 // The client has run SHUTDOWN NOSAVE

Could not connect to Redis at 127.0.0.1:6379: Connection refused

(41.40 s)

How does SCRIPT KILL differ from SHUTDOWN NOSAVE execution? Why the design?

SCRIPT KILL: In the Lua SCRIPT, if there is no command that changes the key, the command can be used to end the command without affecting the Redis service

SHUTDOWN NOSAVE: In the Lua script, if there is a command that changes the key, the command can be used to stop the Redis service

Lua scripts are designed this way to ensure atomicity of batch commands