This is the 13th day of my participation in the August More Text Challenge
The message queue
Why use message queues
- Decoupling (like Spring’s IOC)
Allow you to extend or modify both processes independently, as long as they adhere to the same interface constraints.
- redundant
In some cases, the process of processing data will fail. Unless data is persisted, it will be lost. Message queues persist data until it has been fully processed, thus avoiding the risk of data loss. In the insert-retrieve-delete paradigm used by many message queues, your processing system needs to make it clear that a message has been processed before it can be removed from the queue, ensuring that your data is stored safely until you are done using it.
- recoverability
The failure of a component does not affect the entire system. Message queuing reduces coupling between processes, so that even if a process that processes messages dies, messages that are queued can still be processed after the system recovers.
- The buffer
It helps to control and optimize the speed at which data flows through the system, resolving the inconsistency between the processing speed of production and consumption messages.
- Flexibility & Peak handling capability (peak clipping)
Applications need to continue to function when traffic surges, but such bursts of traffic are rare. It would be a huge waste to invest resources in being able to handle these spikes. Using message queues enables key components to withstand sudden access pressures without completely collapsing under sudden overload of requests.
- Asynchronous communication
Many times, users do not want or need to process messages immediately. Message queues provide asynchronous processing, allowing users to queue a message without processing it immediately. Put as many messages on the queue as you want, and then process them as needed.
Two modes of message queuing
Message queues are divided into
- Point-to-point mode
- Publish/subscribe
(1) Point-to-point mode (one-to-one, consumers take the initiative to pull data, and messages are cleared after they are received)
The point-to-point model is typically a pull or polling based messaging model that requests information from queues rather than pushing messages to clients. The characteristic of this model is that messages sent to queues are received and processed by one and only one receiver, even if there are multiple message listeners.
The producer produces the message and sends it to the Queue, and the message consumer retrieves and consumes the message from the Queue.
Once a message is consumed, it is removed from the queue, so the consumer cannot consume it again
Each message is produced by a producer and consumed by only one consumer (even if there are multiple consumers in the queue).
There is a one-to-one model between producer and consumer
(2) Publish/subscribe mode (one-to-many, after the data is produced, it is pushed to all subscribers, and the message will not be cleared after consumers consume the data)
Publish subscribe model is a push – based messaging model. The publish-subscribe model can have many different subscribers, with a temporary subscriber receiving messages only when actively listening for a topic, and a persistent subscriber listening for all messages for a topic, even if the current subscriber is unavailable and offline.
All consumers who subscribe to the topic receive the same message
Zookeeper profile
Zookeeper is a distributed coordination service
Zookeeper provides configuration management, name service, distributed synchronization, and cluster management.
Configuration management
Once the configuration information changes, each application receives notification from Zookeeper, and then obtains new configuration information from Zookeeper and applies it to the system.
The naming service
Register discovery Center
In fact, from the functions of configuration management and naming services, Zookeeper is like a file system (similar to a Linux file system). In other words, Zookeeper is the brain of distribution.
A distributed lock
In a distributed environment, the same services are deployed on every server in our cluster to improve reliability. However, if every server in a cluster does one thing, it has to be coordinated with each other and can be very complicated to program. If we have only one service operating, then there is a single point. Another common approach is to use distributed locks, where only one service is working at a time, and when that service fails, the lock is released and immediately fails over to another service.
Cluster management
In a distributed storage system, there is a central controller node responsible for allocating storage. When new storage comes in, we allocate storage nodes according to the current state of the cluster. At this point we need to dynamically perceive the current state of the cluster.
Kafka profile
Kafka three-tier messaging architecture
- The first layer is the theme layer, where each theme can be configured with M partitions, and each partition can be configured with N replicas.
- The second layer is the partition layer. Only one of the N copies of each partition can act as a leader and provide services externally, while the other N-1 copies are followers and knowledge provides data redundancy.
- The third layer is the message layer, which contains several messages in the partition, and the displacement of each message starts from 0 and increases successively.
Finally, the client program can only interact with the leader copy of the partition.
The basic concept
- He is a producer.
- -Penny: Consumer.
- Topic = topic
- Broker: a node; Consumers can subscribe to one or more topics and pull data from the Broker to consume these published messages.
- Consumer Offset: represents the consumption progress of consumers. Each Consumer has its own Consumer Offset.
- Consumer Group: A Group of Consumer instances that simultaneously consume multiple partitions to achieve high throughput.
- A topic’s message S can only be consumed by one person in the same group.
- There can be multiple groups to consume.
- But only one person in a group can make a purchase.
It’s easy to understand that a group is a cluster, and it looks like the same person on the outside.
Three mechanisms
There are three mechanisms for ack of Kafka Producer. Producerconfig can be initialized by setting different values of request. Required.
-
0: This means that the producer is not waiting for confirmation from the broker to complete the synchronization and continues to send the next (batch) message. This option provides the lowest latency but the weakest durability guarantee (some data can be lost in the event of a server failure, such as when the leader is dead but the producer does not know about it and the sent information cannot be received by the broker).
-
1: This means that the producer sends the next message after the leader has successfully received the data and confirmed it. This option provides better durability for clients waiting for the server to confirm that the request was successful (the only message written to the dead leader but not yet copied will be lost).
-
-1: This means that the producer does not complete a send until the follower copy confirms that it has received the data.
This option provides the best durability and we guarantee that no information will be lost as long as at least one synchronous copy remains alive.
In the three mechanisms, the performance decreases (producer throughput decreases) and the data robustness increases.
auto.offset.reset
- Earliest: Automatically resets the offset to the earliest offset
- Latest: Automatically resets the offset to the latest offset (default)
- None: Throws an exception to the consumer if the consumer Group did not find the previous offset.
- Other arguments: Throw an exception to consumer (invalid argument)
reference
Juejin. Cn/post / 684490… Segmentfault.com/a/119000001… Segmentfault.com/a/119000001… Segmentfault.com/a/119000001… www.cnblogs.com/pursue339/p…