NSQ in-depth and practical messaging middleware

1). Distributed NSQ provides a Distributed, decentralized, and no single point of failure topology, stable message transmission and release guarantee, with high fault tolerance and HA (high availability) features. NSQ supports horizontal scaling without centralized brokers. The built-in discovery service simplifies adding nodes to a cluster. Both pub-sub and load-balanced message distribution are supported. 3). Ops Friendly NSQ is very easy to configure and deploy, and comes naturally bundled with an administrative interface. Binary packages have no runtime dependencies. Docker Image is official. Official Go and Python libraries are available. Libraries are also available for most languages.

NSQ recommends using co-located publishers through their corresponding NSQD instances, meaning that messages are kept locally until they are read by a consumer, even in the face of network partitions. More importantly, publishers don’t have to discover other NSQD nodes; they can always publish messages to local instances.

NSQ architecture diagram

First, a publisher sends a message to its local NSQD. To do this, first open a connection and then send a publish command containing the topic and the message body. In this case, we publish the message to the event topic to spread out among our different workers.

Event topics copy these messages and queue them on the channels of each connected topic, in our case three channels, one of which is the archive channel. The consumer gets these messages and uploads them to S3.

nsqd

Messages for each channel will be queued until a worker consumes them. If the queue exceeds the memory limit, messages will be written to disk. Nsqd nodes will first broadcast their location information to NSQLookup. Once they are registered successfully, the worker will find all Nsqd nodes containing event topics from the NSQLOOKUP server node.

nsqlookupd

Each worker then subscribs to each NSQD host to indicate that the worker is ready to receive messages. We do not need a complete connected graph here, but we must ensure that each individual NSQD instance has enough consumers to consume its messages, otherwise channels will be queued up.

NSQ guarantees that the message will be delivered at least once, although the message may be duplicated. Consumers should be aware of this and either delete duplicate data or perform actions such as Idempotent. This guarantee is part of the protocol and workflow and works as follows (assuming the client successfully connects and subscribes to a topic) : 1) The client indicates that it is ready to receive the message 2) NSQ sends a message and temporarily stores the data locally (in re-queue or timeout) 3) The client replies with FIN (end) or REQ (re-queue) indicating success or failure, respectively. If the client does not reply, NSQ will automatically re-queue messages at a set time. This ensures that the only possible case of message loss is an abnormal termination of the NSQD process. In this case, any information that is in memory (or any buffer not flushed to disk) will be lost. How to prevent message loss is the most important, even if this unexpected situation can be mitigated. One solution is to make copies of the same parts of the received messages (on different hosts) for redundant NSQD pairs. Because the consumers you implement are idempotent, processing these messages in twice the time has no downstream impact and allows the system to withstand any single node failure without loss of information.

NSQ is designed to be used in a distributed manner. The NSQD client connects (via TCP) to all producer instances of a given topic. There are no middlemen, no message brokers and no single points of failure. This topology eliminates single chain, aggregation, and feedback. Instead, your consumers have direct access to all producers. Technically, it doesn’t matter which client connects to which NSQ, as long as there are enough consumers connecting to all producers to satisfy a large number of messages, guaranteeing that everything will eventually be processed. With NSQlookupd, high availability is achieved by running multiple instances. They do not communicate directly with each other and data is considered ultimately consistent. Consumers poll all configured NSQlookupd instances and merge Response. Nodes that fail, are inaccessible, or otherwise fail do not bring the system to a standstill.

NSQ’s TCP protocol is push oriented. After establishing the connection, handshake, and subscription, the consumer is placed in an RDY state of 0. When the consumer is ready to receive a message, it updates the RDY status to the number of messages it is ready to receive. The NSQ client library constantly manages, behind the scenes, the results of message control flow. Every once in a while, NSQD will send a heartbeat line connection. The client can configure the interval between heartbeats, but the NSQD will expect a response before it sends the next heartbeat. Combining application-level heartbeat and RDY states to avoid head blocking and possibly make heartbeats useless (i.e., if the consumer is in the receive buffer for later processing message flows, the operating system will fill up and block heartbeats) to ensure progress, all network IO time caps must be associated with the configured heartbeat interval. This means that you can literally unplug the network connection between NSQD and the consumer, and it will detect and handle errors correctly. When a fatal error is detected, the client connection is forcibly closed. Messages in transit time out and are requeued for delivery to another consumer. Finally, errors are recorded and accumulated to various internal metrics.

Unlike other queue components, NSQ does not provide any form of replication or clustering, which is what makes it so easy to run, but it does not provide sufficient guarantees for some highly guaranteed and reliable message publishing. We can partly avoid this by reducing the time it takes to synchronize files, simply by configuring a flag to support our queue through EBS. But there is still a case where a message dies immediately after being published, missing a valid write.

For the timeout system, NSQ uses the heartbeat detection mechanism to test whether the consumer is alive or dead. There are many reasons why our consumer could not complete the heartbeat detection, so there must be a separate step in the consumer to ensure idempotency.

topology

Three NSQD services and two LOOKUPD services were used in the experiment. Adopt the official recommended topology, message publishing service and NSQD on one host. There are five machines. NSQ basically does not have configuration files. You can specify parameters through the command line. The main commands are as follows: LOOKUPD command

bin/nsqlookupdCopy the code

NSQD command

Bin/NSQD - lookupd - TCP - address = 172.16.30.254:4160 - broadcast - address = 172.16.30.254Copy the code

Bin/nsqadmin -- - address = 172.16.30.254 lookupd - HTTP: 4161Copy the code

Utility class, stored in a local file after consumption.

Bin /nsq_to_file --topic=newtest --channel=test --output-dir=/ TMP --lookupd-http-address=172.16.30.254:4161Copy the code

Post a message

The curl - d '5' hello world 'http://172.16.30.254:4151/put? topic=test'Copy the code

At its core, NSQ is simple. It’s a simple queue, which means it’s easy to fault reason and easy to find bugs. Consumers can handle failure events without affecting the rest of the system.

In fact, simplicity was the primary factor in our decision to use NSQ, which was easy to maintain with many of our other software, enabled us to perform perfectly by introducing queues, and even allowed us to increase throughput by several orders of magnitude through queues. More and more consumers need a set of strict reliability and sequential guarantees, which have gone beyond the simple functions provided by NSQ.

In combination with our business system, we are relatively sensitive to the invoice messages we need to transmit. We cannot tolerate the breakdown of an NSQD or the unusable disk, and the accumulated messages on this node cannot be retrieved. This is the main reason we didn’t choose this message-oriented middleware. Simplicity and reliability do not seem to be sufficient. Ops is responsible for more operations than Kafka. On the other hand, it provides us with better service by having a reproducible, ordered log. However, for other consumers suitable for NSQ, it has served us quite well, and we are looking forward to consolidating its solid foundation.

Ps: This article was first published on my CSDN blog.