The “Playing with Redis” series focuses on the basics and advanced applications of Redis. This is the first article in the Redis series [13]. Please visit the public account “Zxiaofan” or search for “Redis zxiaofan” on Baidu.

Key words: Redis, Redis data import, Redis mass insert, Redis data export, Redis export specified wildcard, Redis data delete, Redis batch delete, Redis delete specified prefix key, Redis delete specified wildcard key;

Past picks: Redis- Deleted 2 million keys, why memory still not free?

The outline

• The Redis production environment imports large amounts of data safely and efficiently
  • Construct the test data using shell scripts
  • Redis non-clustered mode imports large amounts of data
  • Redis cluster mode imports large amounts of data
• The Redis production environment can safely and efficiently export large amounts of data
  • Redis imports and exports all data
  • Redis exports the specified prefix (specified wildcard) data
• Redis production environment deletes data safely and efficiently
  • Redis Deletes a known specified key
  • Redis deletes the specified prefix (specified wildcard) data
• Practical tips
  • Redis Counts the number of specified wildcard keys
  • Password-free connection to Redis scripts
  • Question to consider: Can Linux make scripts executable but unreadable

Summary:

This paper will simulate the production environment, construct a large number of test data, and complete efficient and safe data import, data export, data deletion.

As we all know, Redis is single-threaded. Once a command is blocked for a long time, it is easy to cause a large number of commands to block, resulting in an avalanche of production environment. Therefore, this paper focuses on “production environment, safe and efficient” operation.

Let ‘s Go!

1. Redis production environment imports large amount of data safely and efficiently

1.1. Use shell scripts to construct test data

The databuild. sh script is used to generate test data. The input parameter of the script is the amount of test data. The script content is as follows:

#! /bin/sh# init Test Data.
# Build By @zxiaofan.com

echo "Construction Quantity: $1";
rm -f ./testdata.txt;
touch testdata.txt;

starttime=`date +'%Y-%m-%d %H:%M:%S'`;
echo "Start: $starttime";

for((i=0; i< $1; i++))do
	str='set key-';
	name=${str}${i}' value'${i};
	echo  $name>> testdata.txt;
done

endtime=`date +'%Y-%m-%d %H:%M:%S'`;
echo "End: $endtime";

start_seconds=$(date --date="$starttime" +%s);
end_seconds=$(date --date="$endtime" +%s);
echo "This running time:"$((end_seconds-start_seconds))"s"
#echo 'show testdata';
#cat testdata.txt;
Copy the code

You are advised to create a script in Linux. Otherwise, an error $’\r’: command not found may be reported when the shell script is executed. Solution: vim acc.conf.base, run :set Fileformat = Unix, and wq to save the configuration.

Then you can run the script to generate testdata. 30W data is generated here. After the script is executed, the testdata file testdata.txt is generated in the current directory.

# execute script generation30W Test data; root@xxx:/study/redis# ./dataBuild.sh300000Number of structures:300000
Start: 2020- 10- 11 17:30:45
End: 2020- 10- 11 17:34:54Running time:249s
Copy the code

# sample script content,30W data is about7.5M:set key0 value0
set key- 1 value1
set key2 - value2
set key- 3 value3
set key4 - value4
set key- 5 value5
...
Copy the code

1.2 Redis non-cluster mode imports a large amount of data

IO /topics/mass insert Redis /topics/mass… :

Using a normal Redis client to perform mass insertion is not a good idea for a few reasons: the naive approach of sending one command after the other is slow because you have to pay for the round trip time for every command. It is possible to use pipelining, but for mass insertion of many records you need to write new commands while you read replies at the same time to make sure you are inserting as fast as possible.

Only a small percentage of clients support non-blocking I/O, and not all the clients are able to parse the replies in an efficient way in order to maximize throughput. For all of these reasons the preferred way to mass import data into Redis is to generate a text file containing the Redis protocol, in raw format, in order to call the commands needed to insert the required data.

Translation:

• Each Redis client command takes a certain amount of time from execution to return results;
• Only a few clients support asynchronous I/O processing, so even using multiple Redis clients to insert concurrently will not improve throughput;
• The recommended solution is to use Redis protocol to generate text files and import the data to be inserted.

Redis 2.6 and above, Redis – CLI supports pipe Mode, which is specifically designed for performing large-scale inserts.

Pipe mode import command:

cat testdata.txt | redis-cli --pipe
Copy the code

We can add the time command before the command to count the named execution time.

# By @zxiaofan

[redis@xxx]$ time cat testdata.txt | ./redis-cli --pipe
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 300000

real	0m1778.s
user	0m0039.s
sys	0m0016.S # Connect to Redis to confirm whether the data is imported successfully.127.0. 01.:6379> get key0
"value0"
127.0. 01.:6379> get key- 299999.
"value299999"
127.0. 01.:6379> get key- 300000.
(nil)
Copy the code

30W data, 1.7 seconds to complete import. Note that if the redis-cli command is not configured as a global command, go to the directory where redis-cli resides and run the following command:

# redis-cli becomes./redis-cli
cat testdata.txt | ./redis-cli --pipe
Copy the code

To specify other parameters, perform the following operations:

• Specified port: -p 6379
• Specify the remote host: -h 192.168.1.1
• Password authentication is required: -a redisPasword

# # By @ zxiaofan Redis import data complete command: cat testdata. TXT. | / Redis - cli - h192.1681.1. -p 6379 -a redisPasword --pipe
Copy the code

1.3 Redis cluster mode imports a large amount of data

In the non-cluster mode, how to import data in the cluster mode?

cat testdata.txt | redis-cli -c --pipe
Copy the code

As shown in the preceding figure, use -c to start the cluster mode. However, the cluster mode has multiple nodes, and data processing may fail.

The Redis cluster mode has 16384 Hash slots. Assume that the cluster has three nodes. Node1 is allocated to slot 0 5460, Node2 to slot 5461 10922, and Node3 to slot 10923 to 16,383. Each primary node in the cluster only operates on the hash slot corresponding to each key. Calculate the hash slot corresponding to each key according to CRC16(Key) Mod 16384, and then perform operations on the Redis node corresponding to the hash slot.

HASH_SLOT = CRC16(key) mod 16384
Copy the code

Note:

In cluster mode, nodes can be dynamically added or deleted. Therefore, ensure the stability of cluster nodes and check the result.

2. Redis production environment is safe and efficient to export a large amount of data

2.1 Redis imports and exports all data

In this scenario, you can use the Ruby open source tool Redis-dump to export files. The official website is github.com/delano/redi…

# redis-dump example

$ redis-dump -u 127.0. 01.:6371 > db_full.json
$ redis-dump -u 127.0. 01.:6371 -d 15 > db_db15.json

$ < db_full.json redis-load
$ < db_db15.json redis-load -d 15
# OR
$ cat db_full | redis-load
$ cat db_db15.json | redis-load -d 15$redis-dump -u :password@host:port -d0 > db_full.json
$ cat db_full.json | redis-load -u :password@host:port -d 0
Copy the code

2.2 Redis exports the specified prefix (specified wildcard) data

Export a large amount of data in a secure and efficient manner. In production environments, you need to export a large amount of specified Wildcard data for data or service analysis. The difficulty is to export a large amount of specified Wildcard data in a secure and efficient manner. In production environments with large data volumes, obstructive operation instructions cannot be used because Redis is single-threaded.

Complete command example:

# By @zxiaofan

./redis-cli -h 192.1681.1. -p 6379 -a password --raw --scan --pattern 'Custom Pattern' > out.txt
Copy the code

Note that the “keys” command cannot be used here because it is blocking. The “keys” command can be used to delete a library or run away from it.

2.3 redis-cli command –raw parameter function

• The output data is raw and contains no type;
• The output data is normally displayed in Chinese, and the display content is not the encoded content;

Redis – when the cli detects that the standard output is tty, it adds a type to the result, such as (integer); Standard output is not a TTY; for example, when data is redirected to a pipe or file, the — RAW option is automatically turned on by default and the type information is not added. Redis. IO /topics/redi… Line Usage “module.

Note that the result has an integer: $redis-cli incr mycounter
(integer)$redis-cli incr myCounter > / TMP /output. TXT $cat/TMP /output. TXT(integer)
$ redis-cli --raw incr mycounter
9
Copy the code

3. Redis production environment deletes data safely and efficiently

3.1 Redis Deletes the known specified key

If we can construct the script to delete data in advance, we can also use “– PIPE” to complete the efficient deletion of a large amount of data.

# By @zxiaofan

cat deletedata.txt | redis-cli -c --pipe
Copy the code

# deletedata.txt  del key-0 del key-1 del key-2 del key-3 del key-4 del key-5 del key-6 del key-7 del key-8 del key-9 ...Copy the code

3.2. Delete the specified prefix (specified wildcard) data

The command is as follows:

# By @zxiaofan

./redis-cli -h 127.0. 01. -p 6379 -a password --scan --pattern 'key-*' | xargs ./redis-cli -h 127.0. 01. -p 6379 -a password del {}
Copy the code

The output is as follows:

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(integer) 12322
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(integer) 12326
Copy the code

The command core is:

• 【–scan –pattern ‘key-*’】 :
• 【 | xargs 】 : used to pass parameters;
• [del {}] : receive key and delete;

Linux xargs:

Xargs (eXtended ARGuments) is a filter for passing ARGuments to commands and a tool for composing multiple commands. Xargs can convert pipe or standard input (STDIN) data into command-line arguments, and can also read data from the output of files.

Xargs can also convert single – or multi-line text input to other formats, such as multi-line to single-line, single-line to multi-line.

The default xargs command is echo, which means that input piped to Xargs will contain line breaks and white space, which xargs will process instead of white space.

Xargs is a powerful command that captures the output of one command and passes it to another.

Because many command does not support | pipe to pass parameters, so have the xargs command.

The preceding commands are described from: www.runoob.com.

Xargs is typically used with pipes in the following command format:

somecommand |xargs -item  command
Copy the code

4. Practical tips

4.1 Redis Counts the number of specified wildcard keys

We’ve taught you how to export or delete data in production. How do we verify that the amount of data is correct?

With the help of “| wc -l” :

# # By @ zxiaofan command is the core of "| wc -l" >. / redis - cli - h127.0. 01. -p 6379 -a password --scan --pattern 'id:*' | wc -l
66666
Copy the code

4.2 Password-free connection to Redis script

Entering a password for every operation in a production environment must be a very insecure and low operation, so is there any way we can connect to Redis without entering a password?

Save the following information in the openredis.sh script and run the chmod u+x openredis.sh command to grant the script execution permission. The password-free here is actually saving the password to a script, so be aware of the security of this script.

# the first parameter is the port number, the second parameter is the command to execute; If you want to specify a host, add -h; Port=$1
CMD=$2
/usr/local/bin/redis-cli -p $Port -a password $CMD
Copy the code

Usage:

[redis@xxx redis]#./openredis6379 'info Clients'
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Clients
connected_clients:1
client_recent_max_input_buffer:51
client_recent_max_output_buffer:0
blocked_clients:0
Copy the code

[redis@redis]#./openredis.sh6379 'set key1 v1 '
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
OK
[redis@redis]# ./openredis.sh 6379 'get key1 '
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"v1"
Copy the code

# use openredis.sh statistics # Notice that the instruction is wrapped in single quotes,'--scan --pattern 'k*'' [redis@redis]# ./openredis.sh 6379 '--scan --pattern 'id:*'' | wc -l
66666
Copy the code

Use openredis.sh to import data# testdata. TXT contains a large number of "set key-0 value0" data

cat testdata.txt | ./openredis.sh 6379 --pipe
Copy the code

[redis@redis]#./openredis.sh6379 '--scan --pattern 'id:*'' > out.txt
66666
Copy the code

4.3. Question: Can scripts be made executable but unreadable in Linux

“Password-free connection to Redis script” We mentioned that the password is saved in the script “openredis.sh”, so can we set the permission of “openredis.sh” to “executable but unreadable”, so as to ensure the security of private information in the script?

The owner can read, write, and execute; Other users can only execute this command. chmod711 openredis.sh
Copy the code

# redis users only have executable permissions

[redis@redis]$ ll openredis.sh 
-rwx--x--x 1 root root 186 Nov  8 22:20 openredis.sh

[redis@redis]$ cat openredis.sh 
cat: openredis.sh: Permission denied

[redis@redis]$ ./openredis.sh 6379 '--scan --pattern 'id:*'' | wc -l
bash: ./openredis.sh: Permission denied
0
Copy the code

From the above execution, when the script is set to “executable but unreadable”, the script cannot be executed because “the interpreter also needs to read the script”.

Knowledge extension:

If the object whose permission is set to Executable but unreadable is not a script but a binary file, the binary is executable.

# set Java files in JDK bin directory to executable only. [admin@xxx bin]$ ll java ---x--x--x1 10 143 7734 Dec 13  2016 java
[admin@xxx bin]$ java -version
java version "1.8.0 comes with _121"
Java(TM) SE Runtime Environment (build 1.8. 0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
Copy the code

Is there any way that scripts can be “executable but not readable”? Welcome to comment.

Redis- deleted 2 million keys, why is the memory still not free?

Play with the Use and Principle of Bloom Filter in Redis

Exploration of Redis-Hyperloglog Principle

“Playing With Redis-Hyperloglog Statistical Micro-blog Day and month”

How to Play Redis- Jingdong Check-in And Jingdou

---

Good luck!

Life is all about choices!

In the future, you will be grateful for your hard work now!

【CSDN】 【GitHub】 【OSCHINA】 【The Denver nuggets】 【Language finches】 【Wechat official account】