What is the NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, It is based on the Go language and is highly regarded for its light weight. It is a simple and easy-to-use messaging middleware developed by Bitly and distributed under the MIT open source protocolCopy the code

The term is introduced

When an application first publishes a message, it creates a Topic Channels. Topic Channels are related to consumers. They are load balancers among consumers. Nsqlookupd is a daemon that receives, queues, and delivers messages to clients. Nsqlookupd: Nsqlookupd is a daemon that manages topology information. Nsqadmin is a set of WEB UI used to gather real-time statistics of a cluster and perform different administrative tasks nsq_to _file: consume specified topics/channels and write them to files, optionally scroll and/or compress files nsq_to _http: Consume specified topics/channels and perform HTTP requests (GET/POST) to specified endpoints NSQ_to-_NSQ: consumer specified topics/channels and republish messages to destination NSQD via TCPCopy the code

Application scenarios of NSQ

Process asynchronization, code decoupling, traffic peak clipping, high availability, high throughput, and broadcast distribution achieve data consistency and meet the requirements of specific business scenariosCopy the code

What do you get out of this project?

Golang entryCopy the code

The theme

• NSQ system architecture

  It consists of three components with different roles: NSQ > NSQLOOKUP > NSD Admin

  1. NSQ
  2. NSQLOOKUP
  3. NSD Admin

  And just to make sense of it, you can think of it as a plane ride. NSQ is regarded as an airplane, NSQLOOKUP is regarded as the dispatching console of an airplane, NSQ Admin is the app of booking tickets, and we ourselves are the message in the whole process. NSQ will carry us to wherever we want to go, and the flight number is TOPIC.

  

• NSQ code structure

  This is also roughly divided into three parts
  
  1. Entry function, build app
  
  2. Common abstractions
  
  3. Service layer of NSQ, NSQlookup and Admin to realize the actual func, which will be focused on later

• Component analysis of NSQ

• nsqd

  NSQD is a daemon that receives, queues, and delivers messages to clients. It can run on its own, but it is typically configured by the cluster in which the NSQlookupd instance resides

  NSQD execution logic

  Producer –> message –> topic –> channels –> Consumer

  NSQD object

  TCP service HTTP service diagram as can be seen, there are two core objects, topicMap and Listener topicMap store topic information. TcpListener and httpListener listen for TCP and HTTP services. Other objects are built around this, for example: context with guaranteed context, sync.RWMutex with atomization, poolSize with message pool management, etcCopy the code

  

  NSQD startup process

  1. It starts TCP server and HTTP Server and listens in daemon mode2. The system continuously sends the registration heartbeat to NSQlookupd to ensure successful registration and survival in NSQdLookup

  Tips: NSQD manages daemon processes with the help of third-party SVC packages. “github.com/judwhite/go-svc” is a service running framework

  Message passing

  How does NSQD deliver messages? What should I pay attention to in the process of distribution?

  Let’s first look at how NSQD handles messages
  
  The producer, which delivers the message to Topics to A, then distributes the specified message to the specific Channel through which the message will be consumed, thus balancing the load of the consumer

  1. Memory optimization of messages
  
  Sync. Pool is used to save temporary objects for reuse to avoid storage redistribution and reduce the GC pressure of programs, so as to avoid: large concurrency – large memory footprint – slow GC – Reduced concurrency – large concurrency

  2. Message transfer process Object uses Byte for message transfer to avoid performance degradation caused by frequent format transformation

  3. Message format
  
  Timestamp (8byte) +msgID (16byte) +body (*byte)

  4. High reliability of messages
  
  If the value exceeds the threshold, some memory will be converted to disks for persistence

  5. Queue channels through the min heap

  NSQD receives messages

  1, a coroutine listens to TCP server 2, and a coroutine processes the requested client and keeps the long connection. When CONN fails, the coroutine also actively cedes CPU 3, and a coroutine listens to TCP socket and continues to process messagesCopy the code

  [TCP Server Listening]

  [Continuously processing requests from clients]

  

• NSQLOOKUP

  Nsqlookupd is the daemon responsible for managing topology information. The client finds the producers of a given topic by querying nsQlookupd, and the NSQD node broadcasts topic and channel information. There are two interfaces: the TCP interface, which NSQD uses to broadcast. HTTP interface that clients use to discover and manage

  The NSQLOOKUP object is simpler with one atomic object and six objects listening for TCP and HTTP services, respectively

  • TCP services:
  • The HTTP service:

  

• NSQAdmin

  Nsqadmin is a set of WEB UI, used to gather real-time statistics of the cluster, and perform different administrative tasks because this content is relatively simple, not too much parsingCopy the code

Bright spot

Performance optimization method

- Sync. pool Implements memory overcommitment and fragmentation persistence to avoid GC excessively high due to frequent collection - Byte stream transfer to avoid performance degradation due to frequent serialization - CPU allocation, for example, when conn err, Active SURRENDER CPU - Minimum Heap sort - Concurrent dynamic adjustment - Probabilistic expiration algorithm - Custom datagramCopy the code

Development means

- Design mode: Factory methods, decorators, etc. - Go concurrent processing, dynamic tuning goroutiune - elegant OOP thinkingCopy the code

conclusion

Although NSQ is not perfect and the overall structure is not complex, and the content of this paper is not comprehensive, please pay attention to the details

The appendix

NSQ’s Git: github.com/nsqio/nsq, as a 20K star messaging middleware