Recently, I encountered a scenario requiring batch pop in my business. At first, I thought it was a very simple solution. However, there were many detours in the implementation process

1. Simple looping pop

This is the simplest scenario, and the pseudocode looks like this:

var values []string for(i:=0; i<num; i++){ values=append(values,redis.rpop()) }Copy the code

This was also our original plan. Later, with the increase of business volume, a large amount of network overhead resulted in serious time consuming, which was gradually unacceptable to the business

conclusion

advantages

  • The simplest solution

disadvantages

  • Serial, time-consuming

2. Easy concurrent pop

Through concurrent POP, thus shortening the network time, the pseudo-code is as follows:

var values []string locker:=new(sync.mutux) wg:=new(sync.WaitGroup) for(i:=0; i<n; i++){ wg.Add(1) go func(){ locker.Lock() values=append(values,redis.rpop()) locker.Unlock() wg.Done() } } wg.Wait()Copy the code

Final Time ≈ Slowest request in the network

conclusion

advantages

  • The easiest optimization to think of
  • Significant performance improvement

disadvantages

  • Need to handle concurrency
  • There is a lot of network overhead, and the server side is under great pressure to handle network connections

3.lrange+ltrim+txpipeline

At first, some colleagues mentioned using Lrange to batch fetch messages from the queue, but it became a headache when we discovered that lrange could only be viewed and not simultaneously removed messages

Then use Ltrim and txpipelline to ensure atomicity, as shown below:

P :=redis.txpipeline(); p:=redis.txpipeline(); p:=redis.txpipeline(); Trim (key, n, -1) values:=p.execute()Copy the code

At first glance it does look perfect, a packaging command that implements bulk pop and avoids a lot of network overhead, but is it really?

Txpipeline atomicity

pipeline

First of all, pipeline itself is non-atomic, and it only sends a batch of commands to the server in one go. The server processes these commands in turn, and the commands from other clients may be executed in between. Therefore, it is impossible to ensure the correct operation of POP using only pipeline, and there is a risk of pop having the same element. Or concurrency risks like deleting too many elements

transaction

Second, let’s take a look at Redis transactions, which consist mainly of multi and exec and encapsulate an atomic execution unit, but are not really transactions per se:

  1. Cannot be rolled back
  2. If some commands fail, the command can still be executed

The transaction can explicitly check some syntax errors, so as to avoid the execution of the transaction; But it is impossible to check for errors that are syntactically indistinguishable, such as popping a set, which results in the above 2 being sent

txpipeline

So TXPipeline is in the pipeline before and after the transaction operation, so as to ensure the atomicity of this batch of commands. The above implementation is fine from this point of view, but the server is slow to respond because the transaction is expensive and blocks other commands

First in first out

When using lrange to batch fetch elements, we should first understand how elements are arranged in the queue, which can be divided into rpush and Lpush:

We can see that in rpush, index is the same as the order in which we push the elements. In this case, when fetching elements from lrange, we can fetch them directly from scratch:

p.lrange(key, 0, n – 1)

The order (A, B,c) is also the order in which we push it, ensuring first-in, first-out

But in lpush, index is the opposite of the element we push, so when we fetch the element through lrange, we need to fetch it from the tail:

p.lrange(key, -n, -1)

First of all, index is not written in the same way as Rpush. Second, the order of extraction will be (C, B, A). In order to ensure fifO, we need reverse consumption

conclusion

advantages

  • One network request with low network overhead

disadvantages

  • Transaction operations are required and the server is at risk of blocking
  • According to the difference between RPUSH and LPUSH, the batch processing methods are different, with more details, which need to be considered clearly

4.pipeline+pop

After understanding the pipeline, we naturally thought of this scheme:

p:=redis.txpipeline() for(i:=0; i<n; i++){ p.rpop() } values:=p.execute()Copy the code

The solution essentially packages a bunch of POPS to the server in one go and lets the server process them in turn, avoiding network overhead and transaction problems

In essence, there is a performance balance between this scheme and scheme 3, that is, the performance loss caused by a large number of POP in this scheme is equal to the transaction operation of scheme 3

conclusion

advantages

  • Easy to implement

disadvantages

  • You need to consider the number of POPS, as unlimited growth would incur a significant performance loss (as opposed to scenario 3, because every POP is lookup + write).

reference

How do I pop multiple pieces of data from a Redis list at once?

Redis series ten: Pipeline details

What is the difference between pipeline and transaction in Redis?

How to use Redis Pipeline well

Redis transaction and Pipeline functions