Redis performance testing, Jedis connection principles, weak transactions.

First, performance test, jedis connection principle

1. What is Redis slow query

As with MySQL: When SQL execution time exceeds the time threshold set by the long_query_time parameter (for example, 2 seconds), time command logging occurs.

The communication process is as follows:

Life cycle of a redis command: send, queue, execute, return.

All instructions in Redis are queued, which are obtained and executed by a single thread in sequence.

If an instruction executes slowly, it will block, as shown in the figure above: Redis’ slow query refers to the third operation.

2. How do I set a time threshold in Redis

One of two ways

First, modify the configuration file redis.conf and restart redis.

/ / modifysetSlowlog-log-slower than 10000 (10 ms) slowlog-log-slower than 10000Copy the code

The second: dynamic Settings.

// Set the value to 10 milliseconds 127.0.0.1:6379> configsetSlowlog-log-slower -than 10000 OK // Takes effect if a restart is required 127.0.0.1:6379> config rewrite OKCopy the code

Note: slowlog-log-slower-than 0 records all commands, slowlog-log-slower-than -1 does not record all commands.

3. Working principle of Redis slow query

Slow-max-len Indicates the maximum number of slow query records. For example, if slow-max-len is set to 10, when the 11th slow query command is inserted, the first command in the queue will be exited and the 11th slow query command will be added to the slow query queue.

The default value of slow-max-len is 128. You can configure slow-max-len dynamically or modify redis.conf.

4. Redis slow query command

Get the command of slow query in the queue: slowlog get.

To obtain the current length of the slow query list: slowlog len // Only one slow query is displayed above and 1 is returned.

  1. To clear (reset) the slow query list: slowlog reset // Check slowlog len again at this time return 0 to clear
  2. For the online slow-max-len configuration, you can increase the value of slow-max-len online. Redis will truncate the slow query when storing the long command, which will not occupy a large amount of memory. You can set the value above 1000 online
  3. Recommendation for online slowlog-log-log-slower than configuration: Default is 10 ms, adjusted according to the amount of Redis concurrency, recommended for high concurrency ratio is 1 ms
  4. Slow query is a first-in, first-out queue. Access log records are lost when they are taken out of the column. You need to perform slowlog get periodically to store the results in other devices (such as mysql).

5. How to use the Redis performance testing tool

There are other tools in the redis-CLI directory during installation, such as Redis-benchmark for performance testing.

Redis-benchmark -c 100 -n 10000 // Test server performance redis-benchmark -c 100 -n 10000 // Test server performance redis-benchmark -q -d 100 // Test onlyset, the performance of the get operation redis-benchmark-tset,get -n 100000 -q
Copy the code

6. What is RESP

Refer to the reidis website for a description of the RESP protocol.

Redis clients communicate with the Redis server using a protocol called RESP (REdis Serialization Protocol). While the protocol was designed specifically for Redis, it can be used for other client-server software projects.

RESP is a compromise between the following things:

  • Simple to implement.
  • Fast to parse.
  • Human readable.

The Redis client communicates with the Redis server using a protocol called RESP(Redis Serialization Protocol). While this protocol was specifically designed for Redis, it can be used for other client-server software projects. Has the following characteristics: simple implementation, parsing speed, human readable.

The underlying implementation principle of RESP

RESP uses TCP connection. Data is transmitted through TCP, and then the corresponding information is parsed according to the parsing rules to complete the interaction.

We can write a simple example to test this by first running a serverSocket listener 6379 to receive requests from redis clients.

Server code:

// Write a fake redis
public class ServerRedis {
    public static void main(String[] args) {
        try {
            ServerSocket serverSocket = new ServerSocket(6379);
            Socket socket = serverSocket.accept();
            byte[] result = new byte[2048];
            socket.getInputStream().read(result);
            System.out.println(new String(result));
        } catch(IOException e) { e.printStackTrace(); }}}Copy the code

Client code:

public class ClientTest {
    public static void main(String[] args) {
        Jedis jedis = new Jedis("127.0.0.1".6379);
        jedis.set("name"."SunnyBear"); jedis.close(); }}Copy the code

(Jedis still needs to return, and the fake redis we wrote did not return in the format). The server side prints the following result:

*3
$3
SET
$4
name
$9
SunnyBear
Copy the code

Note:

*3  // * set key value (*3)
$3  // $identifies the length of the SET, so $3
SET
$4  // $identifies the length of name, so $4
name
$9  // $identifies the length of value, so $9
SunnyBear
Copy the code

This is the rule of the RESP protocol

7. Quickly store existing table data to Redis

The process is as follows:

  1. Log in to the database using the user name and password
  2. After successful login, execute the select statement of order. SQL to obtain the query result set result
  3. Use your password to log in to Redis
  4. After Redis login, use PIPE to import the result to Redis. Test operations are as follows:

To establish the order table

CREATE TABLE `order` (
  `orderid` varchar(40) COLLATE utf8_bin DEFAULT NULL,
  `ordertime` varchar(60) COLLATE utf8_bin DEFAULT NULL,
  `ordermoney` varchar(40) COLLATE utf8_bin DEFAULT NULL,
  `orderstatus` varchar(40) COLLATE utf8_bin DEFAULT NULL,
  `version` varchar(40) COLLATE utf8_bin DEFAULT NULL
);


INSERT INTO `order` VALUES('1'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('2'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('3'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('4'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('5'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('6'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('7'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('8'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('9'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('10'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('11'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('12'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('13'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('14'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('15'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('16'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('17'.'the 2019-01-01 14:04:05'.24.0.0);
INSERT INTO `order` VALUES('18'.'the 2019-01-01 14:04:05'.24.0.0);
Copy the code

Create order.sql with the following content

SELECT CONCAT(
 '*10\r\n'.'$', LENGTH(redis_cmd), '\r\n', redis_cmd, '\r\n'.'$', LENGTH(redis_key), '\r\n', redis_key, '\r\n'.'$', LENGTH(hkey1),'\r\n',hkey1,'\r\n'.'$', LENGTH(hval1),'\r\n',hval1,'\r\n'.'$', LENGTH(hkey2),'\r\n',hkey2,'\r\n'.'$', LENGTH(hval2),'\r\n',hval2,'\r\n'.'$', LENGTH(hkey3),'\r\n',hkey3,'\r\n'.'$', LENGTH(hval3),'\r\n',hval3,'\r\n'.'$', LENGTH(hkey4),'\r\n',hkey4,'\r\n'.'$', LENGTH(hval4),'\r\n',hval4,'\r'
)
FROM (
 SELECT
 'HSET' AS redis_cmd,
 CONCAT('order:info:',orderid) AS redis_key,
 'ordertime' AS hkey1, ordertime AS hval1,
 'ordermoney' AS hkey2, ordermoney AS hval2,
 'orderstatus' AS hkey3, orderstatus AS hval3,
 'version' AS hkey4, `version` AS hval4
 FROM `order`)AS t
Copy the code

Operating instructions

mysql -uSunnyBear -p123456 sunny --default-character-set=utf8 --skip-column-names --raw < order.sql | redis-cli -h 192.168.182.130 -p 6379 -a 123456 --pipe
Copy the code

Note: The order. SQL script is best executed in the redis installation directory for easy management

8. PIPELINE solves the network overhead of batch operation

Most of the time, redis will be operated on by a request-appropriate mechanism. A common step for using only this pattern is to get an instance of Jedis and then interact with Redis through Jedis’ get/put methods. Because Redis is single-threaded, the next request must wait for the last request to complete before continuing.

However, in batch operation, the network overhead is very high, resulting in slow speed. In Pipeline mode, the client can send multiple commands at one time without waiting for the return of the server. In this way, the network round-trip time is greatly reduced and the system performance is improved.

Example code, initialize 10000 key, batch delete 5000.

/** * initialize data and set values in batches */
public class RedisTools {
    public static int arraylength = 10000;
    public static String ip = "192.168.192.130";
    public static int port = 6379;
    public static String auth = "123456";
    public static String[] keys = new String[arraylength / 2];

    /** * initialize the data and set the value in batches * redis provides the value in batches mset and set the value in batches mget, but does not delete the mdel instruction in batches */
    public static void initRedisData(a) {
        Jedis jedis = new Jedis(ip, port);
        jedis.auth(auth);
        String[] str = new String[arraylength];

        int j = 0;
        for (int i = 0; i < str.length / 2; i++) {
            str[j] = "key:" + i;
            str[j + 1] = "v" + i;
            j = j + 2;

            keys[i] = "key:"+ i; } jedis.mset(str); jedis.close(); }}public static void delNoPipe(String... keys){
    Jedis jedis = new Jedis(RedisTools.ip,RedisTools.port);
    Pipeline pipelined = jedis.pipelined();
    for(String key : keys){
        // Wrap all keys to be deleted into pipelined
        pipelined.del(key);
    }
    // Send a request to perform the deletion operation
    pipelined.sync();
    jedis.close();
}

public static void main(String[] args) {
    RedisTools.initRedisData();
    long t = System.currentTimeMillis();
    delNoPipe(RedisTools.keys);
    System.out.println(System.currentTimeMillis()-t);
}
Copy the code

Second, Redis weak transactions

Pipeline is a combination of multiple commands, and to keep it atomicity, Redis provides simple transactions.

What is a transaction? A transaction is the execution of a set of actions that either succeed or fail.

But redis transactions are not as powerful as relational databases such as MySQL. Why? The redis transaction only guarantees atomicity of multiple commands with syntax errors, but it does not guarantee that multiple commands are syntactically correct, but other errors occur.

1. Basic use of Redis weak transactions

Place the command to be executed between the multi and exec commands.

127.0.0.1:6379> flushallsetUser :name SunnyBear // Service operation 1 QUEUED 127.0.0.1:6379>setUser :age 22 // Service Operation 2 QUEUED 127.0.0.1:6379> sadd user:list user:1 user:2exec1) OK 2) OK 3) (integer2)Copy the code

2. Two cases of Redis weak transactions

Syntax errors guarantee atomicity

127.0.0.1:6379> flushall 127.0.0.1:6379> multisetThe name SunnyBear QUEUED 127.0.0.1:6379 >setAge 22 QUEUED 127.0.0.1:6379> SEtt name Zhangsan (error) ERR unknowncommandBeginning with: 'name', 'zhangsan', 127.0.0.1:6379>exec
(error) EXECABORT Transaction discarded because of previous errors.
127.0.0.1:6379> get name
(nil)
127.0.0.1:6379> get age
(nil)
Copy the code

But atomicity is not guaranteed if there are other errors

127.0.0.1:6379> flushallsetUser :name SunnyBear OK 127.0.0.1:6379> multi OK 127.0.0.1:6379>setUser :age 22 QUEUED 127.0.0.1:6379> sadd user:name zhangsan QUEUED 127.0.0.1:6379> get user:name QUEUED 127.0.0.1:6379> sadd user:name QUEUED 127.0.0.1:6379>exec
1) OK
2) (error) WRONGTYPE Operation against a key holding the wrong kind of value
3) "SunnyBear"127.0.0.1:6379 > get the user: the name"SunnyBear"
Copy the code

3. Stop the transaction

127.0.0.1:6379> flushall 127.0.0.1:6379> multisetTt 1 QUEUED 127.0.0.1:6379> discard // Stop transaction OK 127.0.0.1:6379>exec(error) ERR EXEC without MULTI 127.0.1:6379 > get tt (nil)Copy the code

4. Watch to invalidate transactions

Three, supplementary points

1. Redis subscription

Redis offers to publish messages, subscribe to channels, unsubscribe, and subscribe and unsubscribe by pattern.

However, it is not commonly used. There is a specialized technique (message queuing).

2. Key migration (understand)

Move key DB // ReIDS has 16 libraries numbered 0-15

setname SunnyBear; Move name 5 // select name 5; // The database switches to the 6th library. Get name can fetch James1Copy the code

This mode is not recommended for production environments, but can be used in the same ReIDS.

dump key

Restore key TTL value// Implement key migration for different Redis instances. TTL =0 indicates no expiration time

Example: 192.168.42.111 on server A

setname SunnyBear; dump name; / / get"\x00\tSunnyBear\t\x00\xe7\xf9 \xe6\x84\x9d\xc0\xab" 
Copy the code

On server B: 192.168.1.118

restore name 0 "\x00\tSunnyBear\t\x00\xe7\xf9 \xe6\x84\x9d\xc0\xab"Get name // Return SunnyBearCopy the code

migrate

Migrate Is used to migrate data between Redis instances. In fact, the migrate command is a combination of dump, restore, and del commands to simplify the operation process.

The migrate command is atomic and has supported migrating multiple keys since Redis3.0.6.

The migrate command transfers data directly to the source Redis and the target Redis. The target Redis will send OK to the source Redis after being restored.

migrat 192.168.182.130 6379 name 0 1000 copy replace keys
instruction The destination IP address to be migrated port The key value migration Object library timeout The original key is not deleted after the migration The migration succeeded regardless of whether the target library did not have the test key Migrating multiple keys

Migrate the name key from 128 to 130.

192.168.182.128:6379>migrat 192.168.182.130 6379 name 0 100 copy
Copy the code

3. Key traversal (understand)

Redis provides two commands to traverse all keys.

Key full traversal

Note: Considering it is single threaded, it is not recommended to use in production environments, and may block if there are many keys.

Keys * // * matches any character. Keys *y // identifies keys ending in y, such as my, yy keys n? me // ? Matches a character, such as name or NDMECopy the code

Progressive traversal

Note: Features do not block processes.

127.0.0.1:6379> mset n11 N2 2 N3 3 N4 4 N5 5 N6 6 N7 7 N8 8 N9 9 N10 10 N11 11 N12 12 N13 13 OK 127.0.0.1:6379> scan 0 match n* count 8 1)"3"
2)  1) "n13"
    2) "n4"
    3) "n1"
    4) "n8"
    5) "n9"
    6) "n11"
    7) "n7"
    8) "n12"
    9) "n6"
   10) "n10"
127.0.0.1:6379> scan 3 match n* count 8
1) "0"
2) 1) "n5"
   2) "n2"
   3) "n3"
Copy the code

Compare the two traversal methods

Scan has the following features compared with keys:

  1. Through cursor distribution, does not block threads.
  2. Provide the limit parameter, you can control the maximum number of results returned at a time, limit is incorrect, the results can be returned more or less.
  3. Like Keys, Scan provides pattern matching.
  4. The server does not need to save state for the cursor. The only state of the cursor is that scan returns the cursor integer to the client.
  5. The scan results may be duplicated. The client needs to remove the duplication.
  6. If data is modified during scan traversal, it is uncertain whether the modified data can be traversed.
  7. The single return of an empty result does not mean the end of the traversal, but rather whether the returned cursor value is zero.

Now that you’ve read this, like, comment, follow, or collect it!

Author: IT wang2 xiao3 er4 starting address: www.itwxe.com/posts/6a029… Copyright notice: The content of the article is subject to the authorship-non-commercial-no deduction 4.0 international license. If you want to reprint, please link to the author in a prominent position on the article page.