Welcome to github.com/hsfxuebao/j… , hope to help you, if you feel ok, please click on the Star

First, server design

1.1 How to Handle Server Requests?

1.2 Reactor design model

1.3 High performance high concurrency design

1.4 Sequential Read and Write – High performance

Kafka persists message logging to a local disk. It is generally considered that disk read/write performance is poor, and the reliability of Kafka is questioned.

In fact, whether memory or disk is fast or slow depends on the way it addresses. Disk is divided into sequential and random reads and writes, and memory is divided into sequential and random reads and writes. While disk-based random read/write is slow, sequential disk read/write performance is generally three orders of magnitude higher than random disk read/write performance. In some cases, sequential disk read/write performance is higher than random memory read/write performance

1.5 Jumper design

Log file: message store index file: Index information comes in pairs

1.6 Sparse Index

1.7 zero copy

1.7.1 Non-zero copy

1.7.2 zero copy

1.8 Summary of Server Design

High concurrency and high performance network design

In order to read and write

Skip table design sparse index zero copy

Second, Producter design

2.1 the batch

2.2 Memory Pool Design

2.3 summarize

The batch

Memory pool design

Encapsulate the same server request

Iii. Consumer design

3.1 P2P model and publish and subscribe model

P2P model: Also known as peer-to-peer model, a message can only be consumed by one consumer, that is, if a message is consumed by this consumer, the rest of the consumers can not consume, traditional messaging system is used in this way.

Publish subscribe model: Allows messages to be consumed by multiple consumers, but one Consumer needs to subscribe to all partitions of the topic.

3.2 Design of Consumer Group

In P2P mode, one consumer can only be consumed by one consumer in the same group

Different groups are subscriptions, and a message can be consumed by different consumer groups

A partition can only be consumed by one consumer in the same group at a time

3.3 Offset Storage

3.3.1 Architecture of Earlier Versions (0.8)

ZK is not good at high concurrency

ZK is not suitable for high-frequency read and write operations

3.3.2 Architecture scheme of the new version

Kafka naturally supports high concurrency, high availability, and high performance

3.4 summarize

Consumer Group design

Offset storage modification