Redis transaction, Go+ Lua really sweet to use

Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

Redis is a common tool for dealing with high concurrency, as discussed in common caching tips. However, in some business scenarios, problems will be encountered when using Redis, such as the second kill and inventory deduction in e-commerce.

Taking inventory reduction as an example, there are generally two steps:

The inventory is deducted first, and the inventory value V after deduction is obtained
If V is less than 0, it indicates that the inventory is insufficient, and the deducted value needs to be added back. If V is greater than or equal to 0, perform the following operations

However, the two steps are separate, and it is likely to succeed when deducting, but fail when adding back, resulting in inconsistent inventory.

Another option is:

At t1, check the inventory to determine whether it is sufficient
At t2, the inventory is reduced

However, these two steps are also separate, and there is a time difference between T1 and T2. When inventory is deducted at T2, the real inventory is no longer consistent with t1.

Does Redis have the atomicity of MySQL to solve this problem?

The transaction

To solve the destocking problem, you can leverage Redis’s transactional capabilities.

Basic introduction

The MULTI command and EXEC command are required for Redis’ basic transaction, which allows a client to execute multiple commands without being interrupted by other clients.

Unlike transactions in relational databases, which can be rolled back during execution, in Redis all commands surrounded by MULTI and EXEC commands are executed one after another until all commands have been executed. Redis does not process other client commands until a transaction has completed.

A Redis transaction encounters an error midway through execution and does not roll back, but continues to execute subsequent commands;

Error: If a syntax error occurs in the transaction, the transaction will be successfully rolled back and the commands in the entire transaction will not commit
An exec error occurs after the transaction is executed successfully: If the transaction has an execution error instead of a syntax error, the rollback will not be triggered and only the error command in the transaction will not be committed, other commands will continue to be committed

If a query is executed in Redis, the result will not be returned. If a query is executed in Redis, the result will not be returned.

127.0.0.1:6379> multi
OK
127.0.0.1:6379(TX)> set a aaa
QUEUED
127.0.0.1:6379(TX)> get a
QUEUED
127.0.0.1:6379(TX)> exec
1) OK
2) "aaa"

Copy the code

A quick summary of transactions:

Redis transactions are executed as a whole without interruption by other clients
The Redis transaction encounters an error midway through execution and does not roll back, but continues with subsequent commands
Redis encounters queries that are interspersed with transactions and do not return results

Redis transactions are executed without interruption by other clients, so multiple commands can be packaged and executed as a single command. However, Redis native commands do not provide the ability to perform actions based on Redis query results, so we can use Lua scripts instead.

lua

New feature since Redis2.6, you can use Lua script to manipulate Redis in Redis.

The script will execute multiple commands and operations as one command in Redis, which means that the script will not be interrupted by any other script or command during execution.

It is this atomicity that allows Lua scripts to replace the transaction capabilities of Multi and exec. At the same time, it is not advisable to perform excessive overhead operations in lua scripts to avoid affecting the normal execution of subsequent requests.

The benefits of using Lua scripts:

Lua scripts are executed as a unit, so no other commands can be inserted in between
Multiple commands can be packaged at a time, thus effectively reducing network overhead
Lua scripts can be resident in Redis memory, so when used, they can be reused directly and reduce the amount of code

Lua scripts in Redis look like this:

The instance

scenario

The second kill system has drawn the second kill flow chart, in which the second kill inventory related code is:

 / / add inventory
	goodsKey := getGoodsKey(p.AppLocal, tag, tid, goods_id)
	cacheQty, err := app.Global().RedisStore.Incr(goodsKey)
	iferr ! =nil {
		return nil.11008, fmt.Errorf(SERVICE_BUSY)
	}
	// Compare with the total number of seconds
	if cacheQty > int64(goodsInfo.Qty) {
    // If the condition is not met, the inventory is subtracted
		_, err := app.Global().RedisStore.Decr(goodsKey)
		iferr ! =nil {
			ctx.Warn("%s seckill %s ,incr the cnt but not decr success in count process, err:%s", userId, goods_id, err.Error())
		}
		return nil.11009, fmt.Errorf(NOT_SUCCESS)
	}

Copy the code

It can be seen that the inventory should be increased first, and then compared with the total number of seconds to kill this time. If it exceeds the range, the inventory should be reduced. This separate operation increases the possibility of failure. A similar logic applies to the deduction of goods inventories.

Use lua

In Go language with lua script to achieve inventory reduction operations. We can query first, then compare, and finally subtract. The script packages the operations as follows:

package main

import (
	"fmt"
	"github.com/go-redis/redis"
)

var Client *redis.Client

func init(a) {
	Client = redis.NewClient(&redis.Options{
		Addr:     "localhost:6379",
		Password: "111111".// no password set
		DB:       0.// use default DB})}func useLua(a) {
	Client.FlushAll()
	// Set the initial value
	Client.Set("stock"."10".0)
	// write a script - check if the value is sufficient, then subtract, otherwise return the subtracted result
	var luaScript = redis.NewScript('local value = redis. Call ("Get", KEYS[1]) print(" current value = ".. value); If (value-keys [2] >= 0) then local leftStock = redis. Call ("DecrBy",KEYS[1],KEYS[2]) print(" leftStock ); Return leftStock else print(" Not enough to deduct "); return value - KEYS[2] end return -1 `)
	// Execute the script
	n, err := luaScript.Run(Client, []string{"stock"."6"}).Result()
	iferr ! =nil {
		panic(err)
	}
	fmt.Println("The results", n, err)
}


func main(a) {
	useLua()
}

Copy the code

The client loads Lua into Redis and the script is executed by Redis:

If the subtraction value is set to 25, the return value is observed to be negative after execution, but the value is not changed after query again

➜ myproject go run main.go

The current value is 10

The quantity is not enough to make any deduction

Results 15

➜ myproject redis-cli -p 6379 -a 111111

127.0.0.1:6379 > get stock

“10”

If the deduction value is set to 6, the deduction is successful, and view the result after execution

➜ myproject go run main.go

The current value is 10

The remaining value is 4

Results 4

➜ myproject redis-cli -p 6379 -a 111111

127.0.0.1:6379 > get stock

“4”

Optimize lua usage

When scripts are executed multiple times, consider using ScriptLoad and EvalSha instead of RUN to save bandwidth.

The script is cached to Redis with the ScriptLoad command, which returns an identifier for SHA1
The EvalSha command executes scripts based on SHA1

In this scheme, only identifier SHA1 is transmitted over the network, without luA code blocks, saving traffic, as shown in the following figure:

The specific code is:

package main

import (
	"fmt"
	"github.com/go-redis/redis"
)

var Client *redis.Client
var script string = 'local value = redis. Call ("Get", KEYS[1]) print(" current value = ".. value); If (value-keys [2] >= 0) then local leftStock = redis. Call ("DecrBy",KEYS[1],KEYS[2]) print(" leftStock ); Return leftStock else print(" Not enough to deduct "); return value - KEYS[2] end return -1 `
var luaHash string

func init(a) {
	Client = redis.NewClient(&redis.Options{
		Addr:     "localhost:6379",
		Password: "111111".// no password set
		DB:       0.// use default DB
	})
	luaHash, _ = Client.ScriptLoad(script).Result() // The returned script produces an SHA1 hash value, which can be used directly the next time
}

func useLuaHash(a) {
	n, err := Client.EvalSha(luaHash, []string{"stock"."6"}).Result()
	iferr ! =nil {
		panic(err)
	}
	fmt.Println("The results", n, err)
}

func main(a) {
	useLuaHash()
}

Copy the code

custom

If you don’t think it’s elegant to write lua in Go, we can customize it.

Find Redis operation and maintenance students, let the operation and maintenance personnel through Lua custom commands, these commands resident Redis memory, to achieve reuse effect. Using these custom commands is no different from using the commands provided by Redis itself.

Of course, the premise is that operation and maintenance students agree to do.

The problem

While using Lua to take advantage of Redis transaction capabilities and ensure that no other commands are inserted during execution, it does not solve all problems:

Inconclusive return result: Should I return inventory if the request has timed out?
This situation generally do not return, because it is likely not successful deduction, oversold easy to cause capital loss, less selling problem is smaller
Return failure: the deduction is successful, but the subsequent operation fails, the inventory still needs to be returned, there is always the possibility of return failure
In this case, retries are required and can only be performed if the error is of a certain type. For example, retries cannot be performed if a timeout occurs, and records must be made

This is also the reason why SLA is required to be high and log is often kept, which can solve many problems virtually.

Set up

Record the process of building Redis on a Mac, you can install Redis, execute Go+ Lua code.

The installation

Redis setup is relatively simple, use the following command to complete the installation

brew install redis

Copy the code

password

For security, you can set a password. Open the redis. Conf file and press Command + f to search: # requirePass foobared To: requirepass your password

Server startup

Run the following command to start Redis on the server

/usr/local/bin/redis-server /usr/local/etc/redis.conf

Copy the code

If the redis.conf file does not exist when you run the redis-server command, the system starts according to the default configuration. In this case, auth is not required for login

Cli Login

If auth is set, the command line login is as follows:

Redis -cli -p Port -a PasswordCopy the code

If auth is not set, run the command directly

redis-cli

Copy the code

conclusion

We need to accept the fact that some problems cannot be solved 100 percent. But it doesn’t mean compromise, we have to be able to identify problems and identify solutions, even if it’s a manual solution. The premise is that the frequency of the solution should be low enough that there is no need to focus on 100% automated solutions and do as little as possible to achieve a very low output.

data

Use lua scripts in Golang
Redis login and set password
Why Redis is single threaded and why Redis is so fast!
Go-redis transaction commits
Golang uses Lua scripts to implement redis atomic operations
Redis transactions and Lua
Redis transaction analysis and improvement
Redis series (9), Redis “transactions” and Lua script operations

The last

If you like my article, you can follow my public account (Programmer Malatang)

My personal blog is shidawuhen.github. IO /

Review of previous articles:

Design patterns
recruitment
thinking
storage
The algorithm series
Reading notes
Small tools
architecture
network
The Go