The leader throws out a question, Zhang, to integrate Kafka into your microservice to send some data.

From the actual start, the first concept analysis, and then the application of the concept of kafka integration use

Concept paper

What is Kafka?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Apache Kafka is an open source distributed event streaming platform used by thousands of companies to implement high-performance data pipelines, flow analysis, data integration, and mission-critical applications.

In this task, we mainly use the message queue function of Kafka to apply Kafka as the message middleware of microservices

The pattern of kafka

What is the difference between a publish-and-subscribe messaging system and a point-to-point messaging system

Here we can understand that many users subscribe to the computer newspaper (exposed age), and the computer newspaper is full of messages, which is our message queue

Individual editors publish articles to newspapers, meaning that there can be more than one producer

And many subscribers can see, that is, consumers can be multiple

Topic

Continue just model, we subscribe to the PC, you can receive the news of the PC, that at this point I would like to subscribe to the reader, the youth abstract, meaning, Lin is mapped to a company in the system, if there is no unified management middleware, so will appear for extending the publisher, whenever you need a kind of new information, it needs new development a publisher module, eventually lead to system entropy Rapidly expanding

So Kafka introduced topics, which are collectively managed by Kafka. Computer news messages are sent to Computer news topics, and consumers are consumed by Computer news topics, while Youth Digest messages are sent to Youth Digest Topics, and consumers are consumed by Youth Digest Topics, avoiding the need for companies to maintain multiple lines for message queues With Kafka, we have a single centralized system

partition

A topic can be divided into multiple partitions, which can be understood as a mysql branch concept. Ideally, messages are distributed evenly to each partition without any special Settings

About order: Because there are multiple partitions within a topic, the order of messages cannot be guaranteed across the topic, but the order of messages within a single partition can be guaranteed, that is, the single partition is ordered

The message

Kafka’s data unit is called a message, which can be easily understood as a row of data in a traditional relational database.

model

Schemas, combined with Apache Avro, provide a compact serialization format. Let’s take a look at a simple Avro schema

{
	"fields": [
		{ "name": "name", "type": [
        "null",
        "string"
      ],
      "default": null },
		{ "name": "company", "type": [
        "null",
        "string"
      ],
      "default": null }
	],
	"name": "test",
	"type": "record"
}
Copy the code

The schema named Test has two fields, name and company. The field types are both compound types, which can be regarded as NULL or String. Messages are sent according to the schema and parsed with it when receiving messages

Producers and consumers

The producer creates the message and the consumer reads it

After clarifying the concepts above, in Part2 we will look for mappings between concepts and code to get us started on kafka development