With the rapid popularization and development of Internet technology in all walks of life, the call relationship between various layers of applications is more and more complex, and the cost of architecture, development, operation and maintenance is higher and higher. High cohesion, low coupling, scalability and high availability have become industry requirements.

When it comes to Message Queue (MQ), we can think of many application scenarios, such as Message notification, user score increase or decrease, lottery win, etc. MQ can be used for: Process asynchronization, code decoupling, traffic peak clipping, high availability, high throughput, and broadcast distribution achieve data consistency and meet the requirements of specific business scenarios.

Nsq is a lightweight distributed message queue developed by Go language. It is suitable for small projects to use and learn the realization principle of message queue. It is a very good entry project for learning the principle and usage of Go Channel and how to write distributed message in Go language.

Nsq module

NSQ is a simple and easy to use distributed messaging middleware developed by Bitly. It can be used for real-time messaging services in large-scale systems and can process hundreds of millions of messages per day.

NSQ features

NSQ has the following characteristics:

  • Distributed: It provides a distributed, decentralized and single point of failure topology, stable message transmission and publication guarantee, and high fault tolerance and high availability.
  • Easy to scale: It supports horizontal scaling, there is no centralized message Broker, and built-in discovery services make it easy to add nodes to a cluster.
  • Convenient operation and maintenance: it is very easy to configure and deploy, high flexibility.
  • Highly integrated: There are now official Golang, Python, and JavaScript clients, and the community has client libraries in various other languages for easy access and easy customization.

NSQ components

First, take a look at the project structure of NSQ:

The core packages are NSQD, nsqadmin, and nsQlookupd. The apps package houses the various entry methods.

  • NSQD: NSQD is a daemon that receives, queues, and delivers messages to clients. It can run on its own, but it is typically configured by the cluster in which the NSQlookupd instance resides.
  • Nsqlookupd: Nsqlookupd is a daemon that manages topology information. The client finds the producers of a given topic by querying nsQlookupd, and the NSQD node broadcasts topic and channel information. There are two interfaces: the TCP interface, which NSQD uses to broadcast. HTTP interface that clients use to discover and manage.
  • Nsqadmin: NSQadmin is a set of WEB UI used to assemble real-time statistics for a cluster and perform different administrative tasks.

In addition, there are some basic concepts involved in the figure:

  • Topic: A Topic is a logical key for an application to publish messages, and is created when the application first publishes a message.
  • Channels: Channel is related to consumers and is a load balancer among consumers. In a sense, channel is a “queue”. Every time a publisher sends a message to a topic, the message is copied to all channels to which the consumer is connected, and the consumer reads the message through this particular channel, in effect creating the channel the first time the consumer subscribes. Channels arrange messages, and if no consumer reads the message, the message is first queued in memory and saved to disk when the equivalent is too large.
  • Messages: Messages form the backbone of our data flow, and consumers can choose to end Messages to indicate that they are being processed normally, or re-queue them for later processing. Each message contains the number of delivery attempts, and when messages exceed a certain threshold number of delivery attempts, we should either discard them or process them as additional messages.

Common tools:

  • Nsq_to _file: consumes the specified topic/channel and writes it to a file, optionally scrolling and/or compressing the file.
  • Nsq_to _HTTP: consumes the specified topic/channel and executes HTTP requests (GET/POST) to the specified endpoint.
  • Nsq_to _NSQ: The consumer specifies the topic/channel and republishes the message to the destination NSQD over TCP.

summary

This paper mainly introduces the modules, features and components of NSQ. Designed for distribution, clustering, NSQ has the advantage of handling SPOF(single point of failure), high availability, and ultimate consistency.