Different Redis

What do you think of Redis?

High concurrency, KV storage, in-memory database, rich data structure, single thread (prior to version 6)

So, next, the above mentioned, will one by one for you to answer, take you a bit of the charm of Redis, the article will be relatively long, part of the nonsense, please skip, thank you! ~

Why does caching occur?

Under normal circumstances, the data are in the database, the application system directly operates the database. When tens of thousands of visits, database pressure increases, this time, how to do?

Some friends will say, divide the database divide the table, read and write separation. Yes, these are solutions for a relatively high number of visits, but what about a larger number, 100,000, a million? No amount of splitting seems to solve the problem, so we need to use other methods to solve the database stress caused by high concurrency.

This time, the cache, the cache, as the name implies, is the data cached in memory in the first part, when visiting, we will access to memory data first, if the data in memory does not exist, this time, we’ll read the database, the data in the database backup again after a into memory, so next time you read requests to come over, Or it’s going to be accessed from memory first, and once it’s accessed from memory it’s going to be returned. The perfect way to do it to reduce the pressure of the database, and may be thousands of requests come in, all access to the backup data in memory, and didn’t go to access the database, or only a small amount of request access to the database. This really is greatly reduce the pressure of the database, and but also improve the system response, do you think about it, The memory read and write speed is much faster than the hard disk read and write speed, a request to read memory can be much faster than the hard disk read, user experience will be very high.

What is caching?

Cache originally refers to a kind of high speed memory on the CPU, it is before memory and CPU exchange data, speed is very fast

Now generally refers to a set of copies of raw data stored on a computer for quick access.

In the Internet technology, cache is one of the key technologies for the system to respond quickly.

Cache read/write mode

There are three read and write modes for the cache

Cache Aside Pattern

Cache Aside Pattern is the most classic Cache + database read/write Pattern

When reading, first read the cache, if the cache is not available, read the database, then take the data and put it into the cache, and return the response.

When updating, first update the database and then delete the cache

Why delete the cache instead of updating it?

1. The cached value is a structure, hash, list, and updates need to be traversed

2. Lazy loading, updating the cache when it is used, or filling the cache asynchronously

There are three cases of high concurrent dirty reads

1. Update the database first and then the cache

If the cache is updated between update and COMMIT, the DB data is inconsistent with the cache data

2. Delete the cache and update the database

Between update and COMMIT, there are new reads, the cache is empty, and the DB data is read to the cache, but the old data is the data

After the commit, the DB is new data

Then the DB is inconsistent with the cached data

3. Update database first, then delete cache (recommended)

Between update and COMMIT, there are new reads, the cache is empty, and the DB data is read to the cache, but the old data is the data

After the commit, the DB is new data

Then the DB is inconsistent with the cached data

The delay deduplication policy is adopted

Read/Write Through Pattern

The application only operates on the cache, and the cache operates on the database

Read-through (Read Through/direct Read mode) : The application reads from the cache, but the cache does not, the cache returns to the source database, and writes to the cache

Write-through: the application writes to the cache, and the cache writes to the database. This mode needs to provide database handler, the development is more complicated

Write Behind Caching Pattern

The application only updates the cache

The cache updates data to the DB asynchronously in bulk or after merging

It can’t be synchronized all the time, or even lose data

And what about Redis?

Redis is a high-performance, open source, C written NoSQL (non-relational database) also known as a cache database, where data is stored in memory. Redis is stored in key-value form, which is different from traditional relational database. It does not necessarily follow the basic requirements of a traditional database. For example, non-compliance with SQL standards, transactions, table structures, etc. Redis has a variety of data types, such as String, list, set, zset, hash, etc

What can Redis do?

1. This can reduce database stress, increase concurrency, and improve system response time

2. Do the Session separate

   Traditional sessions are maintained and managed by their own Tomcat. In cluster and distributed scenarios, different Tomcats need to manage different sessions. Session replication can only be carried out among different Tomcats through network and I/O, which greatly affects system performance

   Redis solves this problem by storing session information in Redis, so that multiple Tomcat users can share session information

3. Do distributed locking

   General Java lock is multi-threaded lock, is in a process, multiple processes in concurrent will also produce problems, but also to control the timing, this time Redis can be used to do distributed lock, use Redis setnx command to achieve

There are many advantages to using Redis as a cache, but are there many disadvantages?

1. Additional hardware expenditures

   Caching is a space-for-time technique in software systems that requires additional disk space and memory space to store data

2. High concurrency cache invalidation

   In the case of high concurrency, cache invalidation (cache penetration, cache avalanche, cache breakdown, etc.) can cause an instantaneous increase in database access or even crash, so these issues must be addressed

3. The cache is synchronized with the database

   The cache and database cannot synchronize data all the time

4. Cache concurrency contention

   Concurrency issues when multiple Redis clients set a key at the same time due to order of execution

Redis installation here do not say, MAC, Windows, Linux online various installation tutorials, a lot of, we go to the Internet search search follow to do ok, relatively simple, next, with you to analyze, Redis in some common data types.

The data structure of Redis

Redis is a key-value storage system. The key type is a string

There are five common data types of value in Redis: string, list, hash, set, and zset

String String type

String is good for single value caching, object caching, distributed locking, etc

Name of the command	The command format	Command description
set	set key value	The assignment
get	get key	The values
getset	getset key value	Value and assign a value
setnx	setnx key value	Assign when value does not exist Set key value NX PX 3000 atomic operation, PX set the number of milliseconds
append	append key value	Append a value to the tail
strlen	strlen key	Get string length
incr	incr key	Increasing Numbers
incrby	incrby key increment	Increments the specified integer
decr	decr key	Decreasing Numbers
decrby	decrby key decrement	Reduces the specified integer
mset	mset key value key value	Batch assignment
mget	mget key key	Bulk values

Next, we execute these commands from Redis

The set command:

127.0.0.1:6379> set name liuxixi
OK

Copy the code

The get command:

127.0.0.1:6379> set name liuxixi
OK
127.0.0.1:6379> get name
"liuxixi"
Copy the code

Getset command:

127.0.0.1:6379> getset name lixixi" liuxixi" 127.0.0.1:6379> get name lixixiCopy the code

Setnx command:

127.0.0.1:6379> setNx age 12 (INTEGER) 1 // If 1 is returned on the first attempt, the setting is successful. 127.0.0.1:6379> setNx age 13 (INTEGER) 0 // If 0 is returned on the second attempt, the setting is not successfulCopy the code

Append the command:

127.0.0.1:6379> Append name xi (INTEGER) 8 127.0.0.1:6379> Get name "lixixixi"Copy the code

Strlen command:

127.0.0.1:6379> strlen name
(integer) 8
Copy the code

Incr command:

(integer) 14 127.0.0.1:6379> get age "14"Copy the code

Incrby command:

127.0.0.1:6379> incrby age 3
(integer) 17
127.0.0.1:6379> get age
"17"
Copy the code

Decr command:

127.0.0.1:6379> decr age
(integer) 16
127.0.0.1:6379> get age
"16"
Copy the code

Decrby command:

127.0.0.1:6379> decrby age 3
(integer) 13
127.0.0.1:6379> get age
"13"
Copy the code

Hash Indicates the hash type

Name of the command	The command format	Command description
hset	hset key field value	Assignment, no difference between new or modified
hmset	hmset field1 value1 field2 value2	Batch assignment
hsetnx	hsetnx key field value	Assignment, does not operate if filed exists
hexists	hexists key filed	Check whether a field exists
hget	hget key field	Gets a field value
hmget	hmget key field1 field2 …	Gets multiple field values
hgetall	hgetall key
hdel	hdel key field1 field2..	Delete a specified field
hincrby	hincrby key field increment	Specifies the increment of the field INCREMENT
hlen	hlen key	Get number of fields

Application scenario: Can be used as e-commerce shopping cart

E-commerce shopping cart:

1. The key is the user ID
2. The product ID is Field
3. The quantity of goods is value

Shopping cart operation:

1. Add product: hset CART :1001 10088 1
2. Add quantity: HINCRby CART :1001 10088 1
3. Hlen Cart :1001
4. Delete item: hdel Cart :1001 10088
5. Get all items in the cart: hGEtall Cart :1001

Pros and cons of the hash structure

advantages

1. Similar data is classified and consolidated to facilitate data management
2. Compared to string operations, they consume less memory and CPU
3. More space saving than string storage

disadvantages

1. Expired functions cannot be used on fields, only on keys
2. Redis cluster architecture is not suitable for large-scale use

List List type

List The list type can store ordered, repeatable elements

Retrieving records near the head or tail is extremely fast

List has at most 2^31-1 elements (4 billion)

Common operation commands are as follows:

Name of the command	The command format	Command description
lpush	lpush key v1 v2 v3 …	Insert the list from the left
lpop	lpop key	Take it from the left side of the list
rpush	rpush key v1 v2 v3 …	Insert the list from the right
rpop	rpop key	Take it from the right side of the list
lpushx	lpushx key value	Inserts the value into the list header
rpushx	rpushx key value	Inserts the value at the end of the list
blpop	blpop key timeout	From the left side of the list, block when the list is empty, you can set the maximum blocking time, in seconds
brpop	blpop key timeout	Block when the list is empty. You can set the maximum blocking time (unit: second)
llen	llen key	Gets the number of elements in the list
lindex	lindex key index	Get a list element with index starting at 0
lrange	lrange key start end	Returns the elements of the specified range in the list, as specified by start and end
lrem	lrem key count value	Removes the element equal to value from the list When count>0, lREM is deleted from the left side of the list; When count<0, lREM is deleted from the list; When count=0, LREM deletes all elements with value
lset	lset key index value	Set the element at the index position of the list to the value of value
ltrim	ltrim key start end	Trim the list to keep only the start to end range
rpoplpush	rpoplpush key1 key2	Pops from the right side of the KEY1 list and inserts to the left side of the key2 list
brpoplpush	brpoplpush key1 key2	Popping from the right side of the KEY1 list and inserting to the left side of the key2 list blocks
linsert	linsert key BEFORE/AFTER pivot value	Insert value into the list before or after the value pivot

Common data structures

Stack = LPUSH + LPOP (Stack)

The Queue (Queue) = LPUSH + RPOP

BlockingMQ(block queue)=LPUSH+BRPOP

List Application scenarios:

Weibo and wechat public account news flow

Microblogs and public accounts are where new messages are posted at the top

1. MacTalk is tweeting with message ID 10018

   LPUSH msg:{ID} 10018

2. The spare tire said the car tweeted, message ID 10086

   LPUSH msg:{ID} 10086

3. Check the latest tweets

   LRANGE msg:{ID} 0 4

If weibo big V, or wechat big V, attention is relatively high, thousands, tens of thousands, can be sent in batches, such as the first to the online hair, so very fast

Set Set type

Set: Unordered, unique element

The largest number of members in the set is 2^32-1

The following table lists the common operation commands:

Name of the command	The command format	Command description
sadd	sadd key mem1 mem2 ….	Adds a new member to the collection
srem	srem key mem1 mem2 ….	Deletes the specified member from the collection
smembers	smembers key	Gets all the elements in the collection
spop	spop key	Returns a random element in the collection and removes the element
srandmember	srandmember key	Returns a random element of the collection without removing it
scard	scard key	Gets the number of elements in the collection
sismember	sismember key member	Determines whether the element is in the set
sinter	sinter key1 key2 key3	Find the intersection of multiple sets
sdiff	sdiff key1 key2 key3	Find the difference set of multiple sets
sunion	sunion key1 key2 key3	Find the union of sets

Application scenario:

Apply to data structures that cannot be repeated and do not require sequence

For example: follow the user, but also through spoP random lottery

Wechat Lucky draw applet:

1. Click to join the draw to join the collection

   SADD key {userId}

2. View all users who participated in the draw

   SMEMBERS key

3. Draw count the winners

   SRANDMEMBER key [count] / SPOP key [count]

Like, collect and tag wechat and Weibo

1. give a like

   SADD like:{message ID} {user ID}

2. Cancel the thumb up

   SREM like:{message ID} {user ID}

3. Check to see if the user has clicked a “like.

   SISMEMBMR like:{message ID} {user ID}

4. Get a list of the likes

   SMEMBERS like:{message ID}

5. Get the number of likes

   SCARD like:{message ID}

Set operation to achieve microblog and wechat concern model

1. The people you follow

   Xx -> {x, XXX}

2. People I care about

   Ll -> {xx , xxx}

3. Me and the people you care about

   SINTER xx LI -> {xxx}

4. The people I follow follow him:

   SISMEMBER xx LI

5. People I might know:

   SDIFF xx LI -> {xx}

Zset Ordered collection type

The elements themselves are unordered and not repeated

Each element is associated with a score

Can be sorted by score, score can be repeated

The following table lists the common operation commands:

Name of the command	The command format	Command description
zadd	zadd key score1 member1 score2 member2 …	Adds a new member to an ordered collection
zrem	zrem key mem1 mem2 ….	Deletes the specified member from an ordered collection
zcard	zcard key	Gets the number of elements in an ordered collection
zcount	zcount key min max	Return the number of elements in the collection whose score value is in the interval [min, Max]
zincrby	zincrby key increment member	Add increment to the member score of a set
zscore	zscore key member	Gets the score of member in the collection
zrank	zrank key member	Get the ranking of members in the collection (from smallest to largest)
zrevrank	zrevrank key member	Get the ranking of members in the collection (from largest to smallest)
zrange	zrange key start end	Gets the members of the specified interval in the collection, sorted by increasing fraction
zrevrange	zrevrange key start end	Gets the members of the specified interval in the set, sorted by decreasing score

Application scenario:

Since you can order by point, it is suitable for all kinds of leaderboards. Like leaderboards, sales leaderboards, follow leaderboards, etc.

For example:

127.0.0.1:6379> zadd hit:1 100 Item1 20 Item2 45 Item3 (INTEGER) 3 127.0.0.1:6379> zcard hit:1 (integer) 3 127.0.0.1:6379> zrevrange hit:1 0-1 1) "item1" 2) "item3" 3) "item2" 127.0.0.1:6379 >Copy the code

Zset set operation to achieve leaderboard

1. Click on the news

   ZINCRBY hotNews:20190819 1 Protecting Hong Kong

2. Show the top 10 of the day

   ZREVRANGE hotNews:20190819 0 9 WITHSCORES

3. 7 days search list calculation

   ZUNIONSTORE hotNews:20190813-20190819 7

   hotNews:20190813 hotNews:20190814… hotNews:20190819

4. Show the top 10 of the seven days

   ZREVRANGE hotNews:20190813-201908109 0 9 WITHSCORES

Redis single threaded and high performance

Is Redis single threaded?

The single thread of Redis mainly means that the network IO and key-value pair read and write of Redis are completed by one thread, which is also the main process of Redis to provide the external key-value storage service. But other features of Redis, such as persistence, asynchronous deletion, cluster data synchronization, etc., are performed by additional threads.

Why is Redis single threading so fast?

Here we test Redis’ support for concurrency locally

Run this command:./redis-benchmark get result: ====== get ====== 100000 requests completed in 1.02 seconds 50 PARALLEL clients 3 bytes payload keep alive: 1 host configuration "save": 900 1 300 10 60 10000 host configuration "appendonly": no multi-thread: No 0.00% <= 0.1 milliseconds 13.00% <= 0.2 milliseconds 55.85% <= 0.3 milliseconds 80.60% <= 0.4 milliseconds 92.57% <= 0.5 milliseconds 97.12% <= 0.6 milliseconds 99.06% <= 0.7 milliseconds 99.68% <= 0.8 milliseconds 99.86% <= 0.9 Milliseconds 99.90% <= 1.0 milliseconds 99.90% <= 1.1 milliseconds 99.90% <= 1.2 milliseconds 99.91% <= 1.3 milliseconds 99.93% <= 1.4 milliseconds 99.95% <= 1.5 milliseconds 99.97% <= 1.6 milliseconds 99.98% <= 1.7 milliseconds 99.99% <= 1.8 milliseconds 99.99% <= 1.9 milliseconds 100.00% <= 2 milliseconds 100.00% <= 2 milliseconds 98328.42 requests per secondCopy the code

As you can see here, in a second, you can support about 100,000 less concurrency, which is a pretty scary number

Because all of its data is in memory, all of its operations are memory-level operations, and a single thread avoids the performance cost of multithreaded switching. Because Redis is single-threaded, use the Redis command with caution. Use time-consuming commands (such as keys) with caution.

How does Redis single thread handle so many concurrent client connections?

Redis IO multiplexing: Redis uses epoll to implement IO multiplexing, putting connection information and events in a queue and once in a file event dispatcher, which distributes events to event handlers.

Some other advanced commands for Redis

keys

The full traversal key is used to list all keys that meet the rules of a specific regular string. Avoid using this key when the Redis data volume is large, which leads to poor performance

Scan: progressive traversal key

SCAN cursor [MATCH pattern] [COUNT count]

The scan parameter provides three parameters: the first is an integer value of cursor (the index value of the hash bucket), the second is the regular pattern of key, and the third is the number of keys traversed at one time (reference value, the number of underlying traversal is not necessarily), not the number of results that meet the condition. On the first walk, the cursor value is 0, and the first integer value in the returned result is used as the cursor for the next walk. Iterate until the returned cursor value is 0.

127.0.0.1:6379> scan 0 match key* count 3 1) 2) 1) "key4" 127.0.0.1:6379> scan 12 match key* count 3 1) "26" 2) 1) "key1" 2) "key3"Copy the code

Note: However, scan is not perfect. If keys are changed (added, deleted, modified) during scan, the traversal effect may encounter the following problems: The newly added keys may not be traversed, or duplicate keys may be traversed. In other words, scan does not guarantee that all keys are traversed by the complete key, which is something we need to consider during development.

Redis core design principles

Redis serves as a key-value storage system. The data structure is as follows:

A Redis instance contains a number of DB’s and a number of key’s. A Redis instance contains a number of DB’s and a number of key’s. A Redis instance contains a number of DB’s and a number of key’s. Are encapsulated by Redis into something called RedisObjcet, which type is it? This is a pointer to what type it is

Why do you do this, mainly to improve the performance of Redis

PS: By the way, why is using Pointers better than using objects themselves?

Here are two points:

The first is dynamic allocation, and one of the things about Pointers is that you only need to declare the type to which the pointer points (whereas you need to define it if you want to use actual objects). In this way, you can reduce the coupling between your compilation units and thus reduce compilation time

RedisDB structure

Redis does not have the concept of table. The DB of Redis instance is identified by number. DB itself is the namespace of key

For example, if user:1000 is used as the key value, it indicates the element with id 1000 in the user namespace, which is similar to the row with ID =1000 in the user table

SDS string

For example, char data[]=” XXX \0″. For example, when a client sends a set command to Redis, it can send any String. There is no validation. So if we send a string xx\0xx, the xx after the \0 is not read, only the xx before the \0 is read (C uses “\0” to indicate the end of the string, if the string itself has a “\0” character, the string is truncated).

So Redis self-implemented a string called SDS, which contains len and a char buf[]. Len is used to record the length of buf. For example, char buf[] = “xx\0xx”, then len is 5. It’s how much is left

For example, if “xxx1234” is changed to “xxx123456”, then (len+addlen) *2=18, then len becomes 9, and free becomes 9

Such as:

char buf[] = "xxx1234"to"xxx123456" // Where buf is a flexible array
free:12becomefree:10
len:8Become a len:10
Copy the code

What are the benefits of Redis designing SDS this way:

1. Binary secure data structure
2. Provides a memory preallocation mechanism to avoid frequent memory allocation
3. C compatible function library
4. With separate statistical variables len and free, it is convenient to get the length of the string, thus avoiding the risk of an incomplete read.
5. The contents are stored in the flexible array BUf. The pointer exposed by SDS to the upper layer is not a pointer to the structure SDS, but a pointer directly to the flexible array BUF. The upper layer can read the contents of SDS as C strings, compatible with various functions of C language processing strings.

Here’s what soft arrays are:

A flexible array is an array of undetermined size. In C, the last element of a structure can be an array of unknown size, which is called length 0, so we can use a structure to create a flexible array. The main purpose of flexible array is to meet the needs of variable length structure, in order to solve the use of array memory redundancy and array out of bounds

This was also implemented prior to Redis3.2.