Learning is endless, and you are encouraged.
Related series
- A distributed lock
- Bitmap of actual combat
- HyperLogLog
Introduction to the
HyperLogLog is an advanced data structure in Redis. It is mainly used for cardinality statistics (de-recalculation) on massive data (2^64 data). It is characterized by high speed and small footprint (12KB). But the calculation is in error, with a standard error of 0.81%. HyperLogLog only calculates cardinality from the input elements, but does not store the input elements themselves, so it cannot determine whether a given element already exists.
Basic instructions
Pfadd (key, value)…
Adds the specified element to HyperLogLog. Multiple elements can be added
public void pfAdd(String key, String... value) {
stringRedisTemplate.opsForHyperLogLog().add(key, value);
}
Copy the codePfcount (key)…
Returns the cardinality estimate for the given HyperLogLog. When multiple HyperLogLog statistics are collected at a time, you need to compare multiple HyperLogLog structures and put the combined result into a temporary HyperLogLog, which has low performance and should be used with caution
public Long pfCount(String... key) {
return stringRedisTemplate.opsForHyperLogLog().size(key);
}
Copy the codePfmerge (destkey, sourcekey…).
Merges multiple Hyperloglogs and puts the result of the union into a specified HyperLogLog
public void pfMerge(String destKey, String... sourceKey) {
stringRedisTemplate.opsForHyperLogLog().union(destKey, sourceKey);
}
Copy the codeError of the test
Error test based on SpringBoot, initialize 5 HyperLogLog, randomly add 10000 elements to each, then call pfcount to see the error:
@RestController
@RequestMapping("/redis/hll")
public class HyperController {
private final RedisService redisService;
public HyperController(RedisService redisService) {
this.redisService = redisService;
}
@GetMapping("/init")
public String init(a) {
for (int i = 0; i < 5; i++) {
Thread thread = new Thread(() -> {
String name = Thread.currentThread().getName();
Random r = new Random();
int begin = r.nextInt(100) * 10000;
int end = begin + 10000;
for (int j = begin; j < end; j++) {
redisService.pfAdd("hhl:" + name, j + "");
}
System.out.printf("Thread [% s] completes data initialization, interval [%d, %d)\n", name, begin, end);
},
i + "");
thread.start();
}
return "success";
}
@GetMapping("/count")
public String count(a) {
long a = redisService.pfCount("hhl:0");
long b = redisService.pfCount("hhl:1");
long c = redisService.pfCount("hhl:2");
long d = redisService.pfCount("hhl:3");
long e = redisService.pfCount("hhl:4");
System.out.printf("hhl:0 -> count: %d, rate: %f\n", a, (10000 - a) * 1.00 / 100);
System.out.printf("hhl:1 -> count: %d, rate: %f\n", b, (10000 - b) * 1.00 / 100);
System.out.printf("hhl:2 -> count: %d, rate: %f\n", c, (10000 - c) * 1.00 / 100);
System.out.printf("hhl:3 -> count: %d, rate: %f\n", d, (10000 - d) * 1.00 / 100);
System.out.printf("hhl:4 -> count: %d, rate: %f\n", e, (10000 - e) * 1.00 / 100);
return "success";
}
}
Copy the codeInitialize the data, call interface: http://localhost:8080/redis/hll/init
Thread [4] completes data initialization, interval [570000, 580000]
Thread [2] completes data initialization, interval [70000, 80000]
Thread [0] completes data initialization, interval [670000, 680000]
Thread [1] completes data initialization, interval [210000, 220000)
Thread [3] completes data initialization, interval [230000, 240,000]
Copy the codeTo view detailed statistics, calculation error: http://localhost:8080/redis/hll/count
hhl: 0 -> count: 10079, rate: 0790000.
hhl: 1. -> count: 9974, rate: 0260000.
hhl: 2 -> count: 10018, rate: 0180000.
hhl: 3 -> count: 10053, rate: 0530000.
hhl4: -> count: 9985, rate: 0150000.
Copy the codeIn actual combat
For example, you need to count the popularity of articles and the number of valid user clicks. Heat can be counted by the Reis counter, and incR is executed each time. Use HyperLogLog to collect statistics on the number of valid users.
Implementation approach
Use AOP and custom annotations to count articles that need to be counted:
- Annotate article interfaces that require statistics
- Set the custom annotation value to
HyperLogLogThe correspondingkey - Set AOP pointcuts to custom annotations
- Get annotation values in AOP
- In AOP, user information is judged by token or cookie
- Accumulated popularity and audience
pom
Introduce Redis and AOP
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<! Oracle mode connection pool -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
Copy the codeDefine custom annotations
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Article {
/ * *
* Is the key of the corresponding HyperLogLog
* /
String value(a) default "";
}
Copy the codeDefine the AOP
@Aspect
@Component
public class ArticleAop {
private static final String PV_PREFIX = "PV:";
private static final String UV_PREFIX = "UV:";
@Autowired
private RedisService redisService;
/ * *
* Define pointcuts
* /
@Pointcut("@annotation(org.ylc.note.redis.hyperloglog.annotation.Article)")
private void statistics(a) {
}
@Around("statistics()")
public Object doAround(ProceedingJoinPoint proceedingJoinPoint) throws Throwable {
// Get the annotation
Method method = ((MethodSignature) proceedingJoinPoint.getSignature()).getMethod();
Article visitPermission = method.getAnnotation(Article.class);
String value = visitPermission.value();
// Get the request information
ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();
HttpServletRequest request = attributes.getRequest();
// This is used to simulate, passing in the parameters directly. In actual projects, it can be implemented based on tokens or cookies
String userId = request.getParameter("userId");
/ / heat
redisService.incr(PV_PREFIX + value);
/ / users
redisService.pfAdd(UV_PREFIX + value, userId);
// Implement specific methods
return proceedingJoinPoint.proceed();
}
}
Copy the codeDefines the interface
Annotate @article () on interfaces where statistics are required
@RestController
@RequestMapping("/redis/article")
public class ArticleController {
@Autowired
private RedisService redisService;
@Article("it")
@GetMapping("/it")
public String it(String userId) {
String pv = redisService.get("PV:it");
long uv = redisService.pfCount("UV:it");
return String.format("Current users: [%s], current IT popularity: [%s], number of accessed users: [%d]", userId, pv, uv);
}
@Article("news")
@GetMapping("/news")
public String news(String userId) {
String pv = redisService.get("PV:news");
long uv = redisService.pfCount("UV:news");
return String.format("Current user: [%s], current news category popularity: [%s], number of accessed users: [%d]", userId, pv, uv);
}
@GetMapping("/statistics")
public Object statistics(a) {
String pvIt = redisService.get("PV:it");
long uvIt = redisService.pfCount("UV:it");
String pvNews = redisService.get("PV:news");
long uvNews = redisService.pfCount("UV:news");
redisService.pfMerge("UV:merge"."UV:it"."UV:news");
long uvMerge = redisService.pfCount("UV:merge");
Map<String, String> result = new HashMap<>();
result.put("it", String.format("It popularity: [%s], number of users: [%d];", pvIt, uvIt));
result.put("news", String.format("News category Popularity: [%s], number of users: [%d]", pvNews, uvNews));
result.put("merge", String.format("[%d]", uvMerge));
return result;
}
}
Copy the codeAccess to the source code
All codes are uploaded to Github for easy access
>>>>>> Redis Combat — HyperLogLog <<<<<<
Daily for praise
Creation is not easy, if you feel helpful, please support
Please focus on
Wechat official account: Yu Daxian
