14. Lua scripting for Redis

Introduction to the

Redis from version 2.6 introduced the use of Lua programming language for server-side scripting capabilities, this feature allows users to perform a variety of operations directly within Redis, thereby simplifying code and improving performance. P248

Add new functionality without writing C codeP248

By scripting Redis using Lua, we can avoid some common pitfalls that slow down development or degrade performance. P248

Load the Lua script into RedisP249

• SCRIPT LOADThe script is loaded into Redis by the command, which takes a Lua script as a string, stores it for later use, and returns the SHA1 checksum of the stored script
• EVALSHAThe command invokes the previously stored script, which receives the SHA1 checksum of the script and all the parameters required by the script
• EVALThe command directly executes the specified script, which receives the script string along with all the parameters required by the script. In addition to executing the script, this command also caches the script to the Redis server

Due to Lua’s data incoming and outgoing restrictions, Lua and Redis need to convert each other. Because scripts can produce ambiguous results when returning various types of data, we should try to return strings explicitly. P250

We can find conversion tables between the different types of Redis and Lua in the official documentation:

The Redis type is converted to Lua type

Lua type is converted to Redis type

Create a new status messageP251

• Lua scripts follow a single Redis command as wellMULTI/EXECLike transactions, they are atomic operations
• Lua scripts that have been structurally modified cannot be broken
  • Do not execute any write commands on read-only scripts: scripts can be run over time in pairslua-time-limitOption to execute after the specified timeSCRIPT KILLCommand to kill the script that is running
  • Script with command writing: Killing the script will cause Redis to store data in an inconsistent state. In this case

Use Lua to override locks and semaphoresP254

If we don’t know in advance which keys will be read and written, WATCH/MULTI/EXEC transactions or locks should be used instead of Lua scripts. Therefore, reading or writing a key in a script that is not recorded in the KEYS parameter may cause incompatibility or failure when the application is migrated to the Redis cluster. P254

Lua scripts are no longer needed to obtain locks. You can use SET and use PX and NX options to SET a value with an expiration time when the key does not exist. To ensure that you can obtain the lock when releasing the lock, you need to use the Lua script. Related code has been implemented in the implementation of autocomplete, distributed lock and counting semaphore.

removeWATCH/MULTI/EXECThe transactionP258

In general, if only a few clients attempt to modify the data monitored by the WATCH command, the transaction can usually complete without significant conflicts or retries. However, if an operation requires several communication round trips, or if the probability of an operation colliding is high, or if the network latency is high, the client may need to retry many times to complete the operation. P258

Using Lua scripts instead of transactions not only ensures successful execution of any client attempt, but also reduces communication overhead and dramatically improves TPS. Lua scripts are also significantly faster than fine-grained locking versions because there are no multiple communication round trips.

Lua scripts can provide huge performance benefits and can simplify code significantly in some cases, but run inside Redis but Lua scripts can only access data that is within Lua scripts or within Redis databases. Locks or WATCH/MULTI/EXEC transactions do not have this limitation. P263

Shard the list using LuaP263

The composition of a shard listP263

In order to perform push and eject operations on both ends of the shard list, the ID of the first shard and the ID of the last shard in the list need to be recorded during the construction of the shard list in addition to storing the shards that compose the list. When the shard list is empty, the shard ID stored by the two strings will be the same. P263

Each shard that makes up the shard list is named < listName >:

and assigned in order. Specifically, if the program always pops elements from the left and pushes them in from the right, the last allocated index will grow, and the ID of the new shard will become larger and larger. If the program always pops elements from the right and pushes them in from the left, the index of the first shard will decrease and the ID of the new shard will become smaller and smaller. P264

When a shard list contains multiple lists, the list at either end of the shard may be filled, but the other lists between the two ends are always filled. P264

Push the element into the shard listP265

The Lua script uses LPUSH/RPUSH to find the first or last shard in the list and then pushes the element into the list. If the number of shards has reached the upper limit (the value -1 of list-max-ziplist-entries in the configuration can be used as the upper limit), A new shard is automatically generated, pushed in, and the shard ID of the first or last shard is updated. When the push operation completes, it returns the number of pushed elements. P265

Eject elements from the shardP266

The Lua script finds the first or last shard of the list using the command LPOP/RPOP, and pops an element from the shard if the shard is not empty. If the list does not contain any more elements after the pop operation, Then the program will modify the string key that records the shard information at both ends of the list (note that the string key will be modified only when the shard at the list end is empty, while the whole list is empty) P267

Perform blocking pop-up on the shard listP267

I don’t understand this part of the book, and I don’t know why I need the fancy operation in the book to complete it.

Personally, the blocking pop-up of the shard list does not need the blocking pop-up of the list itself. We can continuously execute the operation of the pop-up element implemented by the above Lua script. If the pop-up succeeds, it will return directly; if the pop-up fails, we will continue to perform the pop-up operation after 1 ms sleep until the pop-up succeeds or reaches the timeout time. So we only operate on Redis in Lua script, atomicity guarantees that the elements at both ends of the shard list will pop up.

This article is published on GitHub: Reading-notes/Redis-in-Action