Those who do not have time to read the article can send it directly to the source repository -> EF-redis
RESP Protocol document in Chinese
REDIS command complete
Zero, cause
Why did I build the redis wheel? 1. Demystify Redis. 2. "Basic Services Mid-stage" colleagues are meeting to discuss the Redis cloud and redis agents. 3. It is not easy to open a Redis resource. Why not write it in Java and push it directly to the future cloud? With this idea, I began to use my spare time to study the TCP communication principle of Redis and redis command. The starting point is to write a cloud management software like Redis cloud agent, but I still could not help writing the Java version of Redis. This article mainly shares the writing process of Redis.Copy the codeRedis communications and Netty
1, TCP
The client connected to the Redis server establishes a TCP connection to port 6379.
Although RESP is not technically specific to TCP, in the context of Redis, the protocol is only used for TCP connections (or similar stream-oriented connections, such as Unix sockets).
Use Netty as the communication framework.
2, the agreement
The Redis client communicates with the server using a Redis Serialization Protocol (RESP). Although this protocol is specifically designed for Redis, it can also be used on other client-server communication software. The RESP protocol was introduced in Redis1.2 and did not become the standard for communicating with Redis servers until Redis2.0. This protocol needs to be implemented in your Redis client.
RESP is a serialization protocol that supports multiple data types: Simple Strings, Errors, Integers, Bulk Strings, and Arrays.
RESP is used in Redis as a request-response protocol in the following ways:
The client sends commands to the server as a large RESP array of strings. The server returns a RESP data type based on the implementation of the command. In RESP, the type of data depends on the first byte:
Errors: The first byte of the response is “-” Integers: the first byte of the response is “:” Multi-line Bulk Strings: The first byte of the response is “*”. Additionally, RESP can use large-capacity strings or special variables of array types to represent null values, as explained below. Different parts of the RESP agreement always end with “\ R \ N “(CRLF). The string “foobar” is encoded as follows:
"$6\r\nfoobar\r\n"
Copy the codeWhat does the actual redis command look like, say SET LHJLJH LHJKJHKH
*3\r\n$3\r\nSET\r\n$6\r\nlhjljh\r\n$8\r\nlhjkjhkh
Copy the codeRESP Protocol details document in Chinese
3. Encoding and decoding
Because RESP is inherently command-oriented, there is no way to serialize and deserialize redis messages directly like GRPC or Dubbo. And each content is limited in length, which is good for serialization in time, zero copy, deserialization and serialization directly against the input stream, which is very similar to the design of the Protostuff serialization protocol. So serialization directly converts the stream received by the server directly into values.
4. Command processing
To decode the message into RESP, you also need to convert the RESP into a Command object, which is easier to write and understand because the method is bound to the class in the Java language. But it will add some overhead.
Second, the data structure of Redis
1. Underlying main structure
The underlying main tree is implemented using the hop table ConcurrentSkipListMap. The reason why the hash class map is not used is that after the server is in a cluster, the client may use hash routes, which results in serious hash conflicts on the server and greatly reduces performance
2, the key
The reason for using encapsulated values as values is to facilitate unified management
3, the list
The underlying reason for using LinkedList is that LinkedList implements a variety of interfaces, implementing commands that call methods directly from their existing implementations
4, the set
The underlying HashSet is used, and the set in Redis is not that special
5, the hash
The underlying use of HashMap does not conflict with HashMap. Why not use a skip watch? The compressed list is very clever, basically means that the array received by communication is directly filled into the list, the list is directly used as a map in order, mainly the idea of 0 copy, no need to create new resources, high performance, but note that the compressed list has nothing to do with compression. Interested can view the link:Redis compressed list
6, zset
You first need to encapsulate an object with a value and a score
Redis AOF persistence
1. Aof thread is decoupled from TCP thread, i.e., write buffer
When the redis command is parsed, the redis write command is added to the queue for writing aOF logs
2. Aof Persistence protocol
In a word, aOF adds write commands to logs, reads the commands at first, and executes them as if they were received from the network. Because the protocol is too simple, I won’t post the link here. The format of aOF day is as follows:
3. Aof loading and storage implementation
Here read and write memory are used memory file mapping, the advantage is good read and write performance, the disadvantage is that there may be memory leakage, debugging is more trouble.
4. Memory file mapping and object orientation
The code here to store and load aOF files is procedural-oriented and seems quite complex. In fact, it was written in accordance with object-oriented, encapsulated into a row object, calling the drop drive and pick method can write and read commands in AOF, but TPS is only 10W /s, later changed to process-oriented, throughput increased to more than 100W TPS.
Fourth, redis clustering features
1, master-slave
It is easy to think of mysql’s slave only, many scenarios use mysql’s master-slave read-write separation, or ZK’s master-slave. But in fact, the principal and subordinate of Redis do not guarantee consistency. Personally, I think the principal and subordinate of Redist mainly considers the distributed fault tolerance of CAP. Because redis does not guarantee consistency between master and slave, using Redis to separate read and write may cause some inconsistent problems. Writing is consistent, but reading is inconsistent. You can make choices according to the needs of the project.
2. Master/slave replication
The author did not understand the main slave copy of Redis, so he did not write it out.
3. Sharding clusters
Redis clusters are divided into several unique types: master/slave, partitioned cluster, and agent. Generally, from the perspective of the Redis client, the cluster is mainly partitioned. Based on the key sent to redis, operations such as hash and MD5 are performed. A consensus value is obtained for all clients and the key and value are sent. That is, the client routing distributed software cluster implementation of REDis jingdong cluster design to redis a specific sharding.
Fifth, the pressure measurement and tuning of REDis
1. Aof memory leak
When you start the aof pressure test, you find that there is a memory leak, which is later found to be caused by the frequent creation of memory pools, so the pool is pooled, i.e., there is only one bytebuff memory pool in the aof object.
2. Memory overcommitment improves performance
There is no memory copy, only a New BytesWrapper object is created, but the data stored is stored using BytesWrapper object. There is very little overhead to create/destroy memory.
Check whether the message delay exceeds 200ms
The following figure shows the redis pressure test data of C language version:
4. Performance
The performance of the original Redis is about 4-5W for E5 series CPUS. The data in the figure above is tested using AMD chips. Using redis’s own pressure measuring tool, maintaining 100 client connections, Java version performance is about 75-90% of the original C language performance, performance is still strong.