The premise

The author recently used Redis to store the order list displayed by the client in a project. The list needs to be paginated. Since I was not familiar with the usage scenarios of various Redis data types, I first saw Hash types:

USER_ID:1
   ORDER_ID:ORDER_XX: {"amount": "100","orderId":"ORDER_XX"}
   ORDER_ID:ORDER_YY: {"amount": "200","orderId":"ORDER_YY"}
Copy the code

It feels that the Hash type fully meets the requirements. You then take it for granted to use the HSCAN command for paging, causing the problems you encounter later.

SCAN and HSCAN commands

The SCAN command is as follows:

SCAN cursor [MATCH pattern] [COUNT COUNT] [TYPE TYPE] // Returns the following values: // 1. The result set of the traversal, listCopy the code

The SCAN command is added in Redis2.8.0. The time complexity is calculated as follows: The time complexity of each round of traversal is O(1). The time complexity of all elements traversal until the cursor returns 0 is O(N), where N is the number of elements in the set. SCAN progressively traverses all keys in the Database and does not block Redis. In other words, the SCAN command performs better than the KEY * command in traversing keys. There is a derivative of the Hash command HSCAN specifically for traversing fields of the Hash type and its associated attribute (Field) :

HSCAN key cursor [MATCH pattern] [COUNT COUNT] // Returns the following values: // 1. The result set of the traversal is a mappingCopy the code

Without looking closely at the official Redis documentation, I assumed that hashing pages would be as simple as this (assuming only 1 data per page) :

USER_ID:1 1 COUNT 1 <= = USER_ID:1 1 COUNT 1 <= =Copy the code

In fact, the result is as follows:

HSCAN USER_ID:1 0 COUNT 1 // result 0 ORDER_ID:ORDER_XX {"amount": "100","orderId":"ORDER_XX"} ORDER_ID:ORDER_YY {"amount": "200","orderId":"ORDER_YY"}Copy the code

That is, all field-values corresponding to the KEY are returned in the first iteration. The author tries to add elements in the hash set KEY = USER_ID:1, but the expected paging effect is still not achieved when the data volume is relatively large. On the other hand, I tried to change the COUNT value in the command and found that changing the COUNT value in any way would have no effect on the result of the traversal (that is, returning all the results in the first iteration). Scratching their heads, they could only pore over official documents to find a solution. The cause was found in the COUNT attribute description of the SCAN command:

A simple translation to understand:

The SCAN command and its derivatives do not guarantee the number of elements returned in each iteration, but you can use the COUNT attribute to empirically adjust the behavior of the SCAN command. COUNT specifies the number of elements that should be traversed per call to facilitate traversal of the collection, essentially a hint value.

  1. COUNTThe default value is 10.
  2. When traversing the targetSet,Hash,Sorted SetorKeyThe space is large enough to be represented by a hash table and not usedMATCHUnder the premise of the property,RedisThe server will returnCOUNTOr better thanCOUNTLarge traversal of the element result set.
  3. When traversal only containsIntegerThe value of theSetSet (also known asintsets), orzipliststype-codedHashorSorted SetSet (indicating that the space occupied by the elements of these sets is small enough), thenSCANThe command returns all elements in the collection, ignoring themCOUNTProperties.

Note point 3, which is the root cause of the invalidation of the HSCAN COUNT attribute in the Hash collection. The Redis configuration has two associated values for Hash ziplist encoding:

hash-max-ziplist-entries 512
hash-max-ziplist-value 64
Copy the code

The encoding of the Hash set will change from ziplist to dict if one of the following two conditions is met:

  • whenHashData items in a collection (i.eField-ValueYes) when the number exceeds 512.
  • whenHashAny insert into the collectionField-ValueFor theValueThe length exceeds 64.

The HSCAN COUNT attribute takes effect only when the Hash set encoding is changed from ziplist to dict, and Redis’ memory footprint optimization for Hash has failed and is downgraded to a dictionary encoding that uses more memory.

Case validation

To briefly verify the conclusion of the previous section, write a test data as follows:

/ / 70 X HSET USER_ID: 2 ORDER_ID: ORDER_XXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX / / 70 Y  HSET USER_ID:2 ORDER_ID:ORDER_YYY YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYCopy the code

Next, test the HSCAN command:

Object encoding USER_ID:2 // Encoding result hashtable // first iteration HSCAN USER_ID:2 0 COUNT 1 // First iteration result 2 ORDER_ID:ORDER_YYY YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY / / the second iteration HSCAN USER_ID: 2 2 1 0 COUNT ORDER_ID:ORDER_XXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXCopy the code

The test case deliberately makes two values 70 and larger than 64, which converts the Hash set to the dict(hashtable) type so that the COUNT attribute takes effect. However, this does away with Redis memory optimization for Hash collections. Obviously, the HSCAN command is not designed for data paging naturally, but for incremental iteration (that is, it does not block the Redis service if the set of iterations is large). Therefore, the author finally gave up using HSCAN command and looked for other Redis commands that are more suitable for data paging query.

summary

Through this simple step pit case, the author got some experience:

  • Avoid preconceptions and use middleware in context.
  • Read the user manual carefully before using the tool.
  • Use examples to verify your guesses or inferences.

Redis has a very rich API, and you should expect more experience with stomping in the future.

The attachment

  • Making Page: www.throwable.club/2019/08/12/…
  • Coding Page: throwable. Coding. Me / 2019/08/12 /…
  • Markdown file: github.com/zjcscut/blo…

(E-A-20190812 C-1-D)

Technical official account (Throwable Digest), push the author’s original technical articles from time to time (never plagiarize or reprint) :

Entertainment public account (” Sand carving every Day “), select the interesting sand carving text and video push irregularly, relieve the pressure of life and work: