Message queues are important components of information exchange between distributed applications. Message queues can reside in memory or on disk, and queues can store messages until they are read by applications.

Message queues allow applications to process messages independently without knowing each other’s locations, or without waiting to receive the message before processing it.

Therefore, message queue can solve the problems of application decoupling, asynchronous messaging, traffic cutting and so on, and is an indispensable part of achieving high performance, high availability, scalability and ultimate consistency architecture.

Now the more common message queue products mainly include ActiveMQ, RabbitMQ, ZeroMQ, Kafka, RocketMQ and so on.

1, ActiveMQ



ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS Provider implementation that fully supports JMS1.1 and J2EE 1.4 specifications. Although JMS specifications have been issued for a long time, JMS still plays a special role in today’s J2EE applications.

ActiveMQ features are as follows:

⒈ Compile client in various languages and protocols. Languages: Java,C,C++,C#,Ruby,Perl,Python,PHP. Application protocols: OpenWire,Stomp REST,WS Notification,XMPP,AMQP

⒉ fully supports JMS1.1 and J2EE 1.4 specifications (persistence, XA messaging, transactions)

(b) With Spring support, ActiveMQ can be easily embedded into systems that use Spring and also supports Spring2.0 features

It has passed testing on common J2EE servers such as Geronimo,JBoss 4,GlassFish,WebLogic, and JCA 1.5 Resource Adaptors. ActiveMQ can be automatically deployed to any J2EE 1.4 compliant commercial server

⒌ supports multiple transfer protocol: the in – VM, TCP, SSL, NIO, UDP, JGroups, JXTA

Pictures of high-speed message persistence via JDBC and Journal

7. From the design, it ensures high performance cluster, client-server, point-to-point

⒏ support Ajax

Its advantages are integration with Axis

The embedded JMS Provider for the test can be easily called

2, the RabbitMQ



RabbitMQ is a popular open source message queue system developed in the Erlang language. RabbitMQ is a standard implementation of AMQP (Advanced Message Queuing Protocol). Supports multiple clients, such as Python, Ruby,.NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., supports AJAX and persistence. It is used to store and forward messages in distributed systems, and has good performance in ease of use, scalability, and high availability.



A few key concepts:

Broker: Simple message queue server entity.

Exchange: message switch, which specifies the rules by which messages are routed to which queue.

Queue: Message Queue carrier, each message is put to one or more queues.

Binding exchanges and queues according to routing rules.

Routing Key: The Key used by Exchange to deliver messages.

Vhost: Virtual host. Multiple vhosts can be set up within a broker to separate permissions between different users.

Producer: A program that delivers messages.

A consumer is a program that receives messages.

Channel: Message channel. In each connection of the client, multiple channels can be established. Each channel represents a session task.

The process of using message queues is as follows:

(1) The client connects to the message queue server and opens a channel.

(2) The client declares an Exchange and sets related properties.

(3) The client declares a queue and sets related properties.

(4) The client uses the routing key to establish a binding relationship between exchange and queue.

(5) The client delivers messages to Exchange.

When exchange receives a message, it routes the message to one or more queues based on the key and the binding that has been set.

3、 ZeroMQ



Called the fastest message queue in history, it is actually a series of interfaces similar to sockets. The difference between sockets is that normal sockets are end-to-end (1:1 relationship), while ZMQ can N: What is known about BSD sockets is point-to-point connections. Point-to-point connections require explicit connection establishment, connection destruction, protocol selection (TCP/UDP), error handling, etc. ZMQ shields these details to make your network programming easier. ZMQ is used to communicate between nodes, which can be hosts or processes.

ZMQ(ZeroMQ for short ZMQ) is a simple transport layer, like a socket library framework, which makes socket programming simpler, simpler, and higher performance. Is a message processing queue library that scales flexibly between multiple threads, the kernel, and the host box. The stated goal of ZMQ is “to become part of the standard network protocol stack and then into the Linux kernel.” We haven’t seen their success yet. However, it is certainly a promising, and much more desirable, layer of encapsulation on top of “traditional” BSD sockets. ZMQ makes writing high-performance web applications extremely easy and fun.”

Features are:

  • High-performance, non-persistent

  • Cross-platform: Supports Linux, Windows, and OS X

  • Multilanguage support; C, C++, Java,.NET, Python and more than 30 other development languages

  • It can be deployed independently or integrated into an application

  • Can be used as a Socket communication library

Compared to RabbitMQ, ZMQ is not a traditional message queue server. In fact, it is not a server at all. It is more of an underlying network communication library, wrapped around the Socket API, abstracting network communication, process communication and thread communication into a unified API. Support “request-reply”, “publisher-subscriber”, “Parallel Pipeline” three basic model and extension model.

Key points of ZeroMQ high-performance design:

1. Lockless queue model

For pipe, the data exchange channel between the cross-thread interaction (client and session), a lock-free queue algorithm CAS is used. Asynchronous events are registered on both ends of the PIPE. Read and write events are automatically triggered when messages are read or written to the pipe.

2, batch processing algorithm

For traditional message processing, each message in the sending and receiving time, need to call the system, so for a large number of messages, the system overhead is relatively large, zeroMQ for batch messages, adaptive optimization, can receive and send messages in batch.

3, multi-core thread binding, no CPU switch

Unlike traditional multithreaded concurrent modes, semaphores, or critical sections, zeroMQ takes full advantage of multiple cores, running one worker thread per core binding, avoiding CPU switching overhead between multiple threads.

4 、Kafka



Kafka is a high-throughput distributed publish-subscribe messaging system that processes all action flow data in consumer-scale websites. This action (web browsing, searching and other user actions) is a key factor in many social functions on the modern web. This data is usually addressed by processing logs and log aggregation due to throughput requirements. This is a viable solution for logging data and offline analysis systems like Hadoop, but with limitations that require real-time processing. Kafka is designed to unify online and offline message processing through Hadoop’s parallel loading mechanism, and to provide real-time consumption through clustered machines.

Kafka is a high-throughput distributed publish-subscribe messaging system with the following features:

  • Message persistence is provided through the O(1) disk data structure, which can maintain stable performance over long periods of time even with terabytes of message storage. (Data is written in the way of file appending, and expired data is deleted periodically)

  • High throughput: Even very modest hardware Kafka can support millions of messages per second

  • Support for partitioning messages through Kafka servers and consumer machine clusters

  • Hadoop supports parallel data loading

Kafka related concepts

  • Broker

A Kafka cluster consists of one or more servers, known as brokers. [5]

  • Topic

Each message published to the Kafka cluster has a category called Topic. (Physically, messages from different topics are stored separately. Logically, messages from one Topic are stored on one or more brokers, but users can produce or consume data by specifying the Topic of the message, regardless of where the data is stored.)

  • Partition

Parition is a physical concept, and each Topic contains one or more partitions.

  • Producer

Responsible for publishing messages to Kafka Broker

  • Consumer

Message consumers, clients that read messages to Kafka Broker.

  • Consumer Group

Each Consumer belongs to a specific Consumer Group (you can specify a Group name for each Consumer, or the default Group if you do not specify a Group name).

It is generally used in big data log processing or scenarios that have low requirements on real-time performance (a little delay) and reliability (a little data loss)

5、 RocketMQ



Pache ActiveMQ is a very popular, powerful, open source messaging and Integration Patterns server that is fast and supports multiple cross-language clients and protocols. Easy to use Enterprise Integration Patterns, with many advanced features and full support for JMS 1.1 and J2EE 1.4 specifications. ActiveMQ is licensed under Apache 2.0.

Apollo is a faster, more reliable, and easier to maintain message broker tool based on the ActiveMQ prototype. Apache claims Apollo is the fastest and most robust STOMP (Streaming Text Orientated Message Protocol) server. Apollo features are as follows:

  • Supports Stomp 1.0 and Stomp 1.1 protocols
  • Topics and queues
  • Queue browser
  • Topic persistent subscription
  • Mirrored queue
  • Reliable messaging
  • Message expiration and exchange
  • Message selector
  • The JAAS authentication
  • Acl-based authorization
  • Support SSL/TLS, certificate verification
  • REST Management API

Which middleware to choose?

Which program should be, or to see the specific needs. In our design, THE functionality of MQ is business agnostic, so the preference is to build with existing middleware. So which middleware do you choose? Let’s start with our requirements for MQ:

The functional requirements

As mentioned earlier, in addition to the most basic production consumption model, MQ is required to support the Request-Reply model to provide support for synchronous invocation. In addition, if MQ could provide a publish-subscribe model, the implementation of the event broker could be simpler.

Performance requirements

Considering the development of the product in the next one to two years, throughput in the message queue is not expected to exceed 1W QPS, but the single-message latency requirement is high and it is expected to be as short as possible.

Availability requirements

Because it is an online service, high availability is required, but a small amount of message loss is allowed.

Ease of use requirements

Including learning cost, initial development and deployment cost, daily operation and maintenance cost, etc.

Horizontal contrast

ActiveMQ is similar to RabbitMQ in many ways, but ActiveMQ does not support the same non-Java ecosystem as RabbitMQ and has limited energy, so this article focuses on RabbitMQ.


ActiveMQ is similar to RabbitMQ in many ways, but ActiveMQ does not support the same non-Java ecosystem as RabbitMQ and has limited energy, so this article focuses on RabbitMQ.



conclusion

The message queue selection depends on the specific application requirements: ZeroMQ is small and beautiful, RabbitMQ is large and stable, Kakfa and RocketMQ are fast and powerful.

RocketMQ is still in its infancy, but once it hatches into a top-level project at Apache, it has a huge future.