Welcome to github.com/hsfxuebao/j… , hope to help you, if you feel ok, please click on the Star

1. Kafka producer source directory

01 Kafka source code in-depth analysis – Preface

02 Kafka source code in-depth analysis – source code reading environment build

03 Kafka source code in-depth analysis – source analysis ideas

04 Kafka source code in-depth analysis – start with a demo

05 Kafka source code in-depth analysis -Producer core process introduction

06 Kafka source code in-depth analysis -Producer initialization

“07 Producer Source Code metadata Management at Producer End”

08 Kafka source code in-depth analysis -Producer core process exploration

09 Kafka source code in-depth analysis -KafkaProducer load metadata

10 Kafka source code in depth – Select the appropriate partition

Kafka Accumulator (Kafka accumulator)

12 Kafka source code in-depth analysis – improve the performance of the design of data structure CopyOnWriteMap

13 Kafka source code in-depth analysis – write data to the corresponding batch (block lock)

Kafka source code in-depth analysis – memory pool design

Kafka source code in depth analysis -sender thread send message flow

Kafka source code depth analysis – Batch send conditions

17 Kafka source code in depth analysis – Filter can send message brokers

18 Kafka source code in-depth analysis -Kafka network design

KafkaProducer will send messages if the network is not established

Kafka source code -KafkaProducer finally establish a connection with the Broker

KafkaProducer finally sends a request

Kafka source code in-depth analysis -KafkaProducer solution to sticky package problem

23 Kafka source code in-depth analysis -KafkaProducer processing unpacking problem textbook code

KafkaProducer: How to handle transient state response messages

25 Kafka source code in depth -KafkaProducer Finally how to handle the response

Kafka source code in depth – how is memory processed after the response is processed?

27 Kafka source code deep analysis – message exception is how to handle?

Kafka source code in depth analysis – send a message timeout is how to handle?

29 Kafka source code in depth analysis – how to handle a request for a long time

30 Kafka source code in-depth analysis – producer source code essence summary

Ultra-concurrent network architecture design:

2. Conclusion:

(1) Kafka’s network design is definitely a highlight. Kafka encapsulates its own network framework based on NIO, which supports a client to establish connections with multiple brokers.

(2) deal with unpacking and sticky package ideas and code, is absolutely textbook level, you can copy and paste down the code directly use their own online projects.

(3) RecordAccumulator encapsulates the message batchs, using its own encapsulated data structure CopyOnWriteMap, using the idea of read and write separation, in the face of high concurrency (read more, write less) to improve the performance of the whole process.

(4) Memory buffer pools designed to encapsulate messages at the same time, which greatly reduces the number of GC. The biggest fear of our Java projects is that 60% of the problems in GCjava projects are caused by GC.

(5) If you still remember the code of RecordAccumulator when it encapsulated lots, the code used the idea of segmental locking when it encapsulated lots, which greatly improved the performance. It can be seen that the author really had deep programming skills. Lock not where you can, but only where you must. Try to improve the performance of your code, or reduce the use of locks. Because once we use locks in our code: 1. Deadlocks 2. Lock contention

(6) In fact, I personally feel that Kafka’s exception system is clearly designed to catch exceptions in the core process and throw exceptions at the bottom. If you have less programming experience, you can learn from it.

(7) The producer provides two schemes when sending data: 1. Synchronous sending: performance will be poor; 2. Asynchronous sending: Kafka’s performance is greatly improved when combined with callback functions. If we go to design a system by ourselves, this design idea is also worth learning.

Reference Documents:

Kafka 0.10.2.0-src kafka 0.10.2.0-src kafka 0.10.2.0-src

Kafka technology insider – Graphic details kafka source code design and implementation

Kafka source code analysis series