This is the 9th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

Apache Kafka is a distributed Messaging System. It is written in Java and Scala. It started on LinkedIn and has since been open-source. In 2014, several former LinkedIn engineers founded Confluent with an eye on Kafka, named after author Franz Kafka.

Message system is we often say the message queue, its working mechanism can be described simply, the system sends A message to A message system, system B read messages from the message system, so as to realize the asynchronous invocation of services and systems between decoupled, ease the upstream flow surge to the downstream system pressure peak (cut), etc. The message system is commonly known as message queue because, I think, A first-in, first-out queue data structure can implement the simplest message system, system A pushes elements into the queue, system B pulls elements out of the queue; It may also be that when using a messaging system, it also looks like a first-in, first-out queue structure from the outside. However, the messaging model of a mature messaging system goes beyond that.

There are two ways to pass messages in common messaging systems:

  • The first is the point-to-point model. This method is similar to making A phone call. After A call is placed by system A, only System B can answer the call. Other systems cannot answer the call. That is, messages can only be sent from system A to system B, with no third system involved in the process.
  • The second is the publish/subscribe model. This approach is more like broadcasting, and it has the concept of a radio station (called Topic in Kafka). In this pattern, there are anchors (message sender) and listeners (message receiver). Multiple senders can send messages to a Topic, and multiple receivers can retrieve messages from a Topic. Just as multiple anchors can broadcast content on a single station, many listeners can listen to content from the same station.

Apache Kafka supports both modes.

With the pattern of sending and receiving messages, a messaging system also needs to standardize the language of communication between the recipients and the senders because they belong to different systems. Common common data formats are XML, JSON, and so on. Apache Kafka uses a pure binary sequence.

For information on when a messaging system is needed, or exactly what problems messaging systems can solve, see this article: Why Message queues