Redis advanced features Bitmap posture and application scenario introduction

Redis advanced features Bitmap posture and application scenario introduction

The five basic redis data structures, such as String, List, Set, ZSet, and Hash, are relatively common. On top of these basic results, Redis also provides some more advanced functions, such as GEO, Bitmap, Hyperloglog, pub/ Sub. This paper will mainly introduce the posture of using bitmap and its applicable scenarios, and the main knowledge points include

Bitmap is basically used
Bitmap usage posture in daily active statistics application scenarios
Thumbs up to re-use bitmap gestures in application scenarios
Bloomfilter basic principle and experience case

I. Basic use

1. The configuration

We used SpringBoot 2.2.1.RELEASE to set up the project environment and added redis dependencies directly to pom.xml

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Copy the code

If our Redis is the default configuration, we can add no additional configuration. It can also be directly in the application.yml configuration, as follows

spring:
  redis:
    host: 127.0. 01.
    port: 6379
    password:
Copy the code

2. Use posture

Bitmap consists of three operations: setbit, getbit and bitcount

A. Set the tag

Setbit, which basically means setting an index to 1(0 means erasing the mark). The syntax is as follows

Note that the index must be a number and the value must be 0/1
setbit key index 0/1
Copy the code

In SpringBoot, RestTemplate can be relatively easy to implement, usually there are two ways to write, both can

@Autowired
private StringRedisTemplate redisTemplate;

/** * sets the flag bit **@param key
 * @param offset
 * @param tag
 * @return* /
public Boolean mark(String key, long offset, boolean tag) {
    return redisTemplate.opsForValue().setBit(key, offset, tag);
}

public Boolean mark2(String key, long offset, boolean tag) {
    return redisTemplate.execute(new RedisCallback<Boolean>() {
        @Override
        public Boolean doInRedis(RedisConnection connection) throws DataAccessException {
            returnconnection.setBit(key.getBytes(), offset, tag); }}); }Copy the code

The key difference between the two methods is the serialization of keys. The first method uses the default JDK string serialization method, which differs from the getBytes() method

B. Check whether it exists

Getbit key index, if it returns 1, it exists otherwise it does not exist

/** * checks whether ** is marked@param key
 * @param offest
 * @return* /
public Boolean container(String key, long offest) {
    return redisTemplate.opsForValue().getBit(key, offest);
}
Copy the code

C. count

Bitcount key, statistics and

/** * Statistics **@param key
 * @return* /
public long bitCount(String key) {
    return redisTemplate.execute(new RedisCallback<Long>() {
        @Override
        public Long doInRedis(RedisConnection redisConnection) throws DataAccessException {
            returnredisConnection.bitCount(key.getBytes()); }}); }Copy the code

3. Application scenarios

The basic usage of bitmap is relatively simple. As mentioned in the introduction to String data structures, we need to focus on the use of bitmap. What can it be used for, and in what situations would it have significant advantages

Day live statistics
give a like
bloomfilter

Although the above three scenarios have similarities, the actual application scenarios are slightly different, which will be explained one by one

A. Daily active statistics

Statistics application or website daily activities, this is a relatively common case, if it is to use Redis to do this thing, the first thing we can most easily think of is the Hash structure, the general logic is as follows

Set the key based on the date, for example, today2020/10/13, then key can beapp_20_10_13
Second, when the user accesses, set field to userId and value to true
To judge the daily activity, count the number of Mapshlen app_20_10_13

What’s wrong with this logic? Of course there is no problem, but think about it when our application is doing very well, with millions or tens of millions of jobs per day, the memory overhead is a little daunting

Now let’s see what bitmap can do

Also set key based on date
When the user accesses, index is set to userId,setbit app_20_10_13 uesrId 1
Day live statisticsbitcount app_20_10_13

A quick comparison of the two schemes above

When the amount of data is small and the distribution of userids is uneven, the use of bitmap is a bit of a loss, because the userID is used as index, then the bitmap length needs to accommodate the maximum userID, but the actual daily activity is very small. Note There is a large amount of blank data in the bitmap

On the contrary, when the data volume is large, such as millions/tens of millions, and the userId is continuously increasing, bitmap has two advantages: 1. Low storage overhead. 2. Fast total statistics

C. thumb up

The main thing about a “like” business is that once a user has liked it, they can’t like it again (except for certain business scenarios, of course), so we need to know if we can like it again

Of course, the above hash can also be implemented, so we will mainly discuss the implementation logic of bitmap

For example, if we want to count likes for an article, we generate redisKey= based on the articleIdlike_1121UserId as index
The first is throughgetbit like_1121 userIdTo determine whether the user has liked, thus limiting whether the user can operate

Hash and bitmap options are similar in scope

D. Bloomfilter

Bloem filters are pretty well known, so let’s take a quick look at what they are

The underlying storage is a bitmap
When you take a number, you go through n hash functions to get n values
Map n values from hash to bitmap, marking the corresponding position is 1

If n values have bitmaps of 1 after hash calculation, the data may exist. If one of them is not 1, the data must not exist

Note: when it does not exist, it certainly does not exist; In existence, not necessarily

As we know from the above description, the underlying data structure of BloomFilter is bitmap. Of course, the key point of BloomFilter is hash algorithm. It is very suitable for solving the problem of cache breakdown because it must not exist when it is not hit

How ‘

Redis bloom filter for >=4.0, through the form of a plug-in, the project source address is github.com/RedisBloom/… According to readme’s instructions, here is a simple experience of redis bloomfilter using posture

# Docker install
docker run -p 6379:6379 --name redis-redisbloom redislabs/rebloom:latest

# Redis-cli access
docker exec -it redis-redisbloom bash

# Start using
# redis-cli
127.0.0.1:6379> keys *
(empty array)
127.0.0.1:6379> bf.add newFilter hello
(integer) 1
127.0.0.1:6379> bf.exists newFilter hello
(integer) 1
127.0.0.1:6379> bf.exists newFilter hell
(integer) 0
Copy the code

Bloomfilter is simple to use. It uses two commands: bf.add to add elements

4. Summary

Bitmap is a relatively delicate data structure, usually in the case of a large amount of data, there will be a performance effect; Redis itself based on the String data structure to achieve bitmap function support, the use of the way is relatively simple, basically the following three commands

setbit key index 1/0Set:
getbit key index: Checks whether it exists
bitcount key: Count statistics

This article also presents three common application scenarios for Bitmap

Daily active statistics: mainly by means ofbitcountTo get the total number (as described later, hyperLogLog is more elegant for more than 100,000 million daily lives)
Likes: mainly with the help ofsetbit/getbitTo judge whether users like, so as to achieve heavy
Bloomfilter: Bitmap-based Bloomfilter, widely used in business scenarios for de-weighting (such as cache penetration, crawler URL de-weighting, etc.)

In general, Bitmap is a data structure that is easy to use and clever to use. Good use can save memory and improve efficiency, while bad use does not seem to cause too much problem

II. The other

0. Project

Series of blog posts

【DB series 】Redis pipeline Pipelined using posture
DB series Redis cluster environment configuration
【DB series 】 Build a simple site statistics service with Redis (Application)
【DB series 】 Using Redis to achieve the ranking function (Application)
Redis ZSet data structure using posture
【DB series 】Redis Set data structure using posture
Redis Hash data structure using posture
Redis List data structures use postures
【DB series 】Redis String data structure read and write
【DB series 】Redis Jedis configuration
DB series basic configuration of Redis

Engineering source

Project: github.com/liuyueyi/sp…
Project source: github.com/liuyueyi/sp…

1. An ashy Blog

As far as the letter is not as good, the above content is purely one’s opinion, due to the limited personal ability, it is inevitable that there are omissions and mistakes, if you find bugs or have better suggestions, welcome criticism and correction, don’t hesitate to appreciate

Below a gray personal blog, record all the study and work of the blog, welcome everyone to go to stroll

A grey Blog Personal Blog blog.hhui.top
A Grey Blog-Spring feature Blog Spring.hhui.top

Redis advanced features Bitmap posture and application scenario introduction

I. Basic use

1. The configuration

2. Use posture

A. Set the tag

B. Check whether it exists

C. count

3. Application scenarios

A. Daily active statistics

C. thumb up

D. Bloomfilter

4. Summary

II. The other

0. Project

1. An ashy Blog

Related Posts

Go Advanced 11, Hand touch takes you deep into the implementation of Mutex

Kubeflow pit records

To learn About Spring IOC, just finish reading this article