Message access requirements for message queues

  • Message queue function: serves as a buffer for components of different processing speeds

  • Message queue requirements

    • Message preservation \

    • Repeat message processing \

    • Message reliability assurance

\

List-based message queue solution

  • Message preservation \

    • List Natural Guarantee
    • Write message lpush
    • Read rPOP or BRPOP (block client wait notify, do not consume CPU resources)
  • Duplicate message processing

    • Message queues should be able to provide globally unique IDS (snow Flake) \ for each message

    • The consumer program records the ID number of the message that has been processed

    • Ensure idempotency of consumption

  • Message reliability assurance

    • BRPOPLPUSH inserts the message into another List backup List
  • Possible problem: Producers send messages quickly and consumers process them slowly, causing messages in the List to pile up and strain Redis memory

  • List does not support multiple consumers. Steam can be used instead

\

A Stream-based message queue solution

  • Streams is a Redis data type designed specifically for message queues (memory-based message queues) \

  • operation

    • XADD: Insert message, ensure order, can automatically generate global unique ID\

    • XREAD: used to read messages, which can read data by ID (set block value) \

    • XREADGROUP: reads messages as a consumer group \

    • The XPENDING command can be used to query for all messages \ that have been read but not confirmed by consumers within each consumer group

    • The XACK command is used to confirm to the message queue that message processing has completed

    • XGROUP Creates consumer groups

conclusion

  • Memory message queues use Streams after Redis 5.0

  • Message queue requirements

    • Message sequencing Sequential access to message data

    • Duplicate message processing message data has a globally unique number \

    • Message reliability Ensures that message data is deleted after consumption