Kafka study notes
The profile
- Kafka is a distributed commit logging platform, or distributed streaming platform
- Persist in a certain order
- Provides data fault protection and performance scaling capabilities
For efficiency, messages are sent in batches.
partial
producers
By default producers distribute messages evenly across all partitions within a topic, or they can be published to specific partitions in a specified manner. Specify partition: key of message, custom partition
consumers
Consumers pass message offsets, which are stored on Zookeeper or Kafka. A node in a consumer cluster can bind consumption to only one partition
The Broker and the cluster
- Receive a message
- Sets the offset for the message
- Save to disk
- Inform consumers
Cluster controllers, automatically elected from active brokers, are responsible for management, including assigning partitions to brokers, and automatic monitoring
Partition leader Partition replication
Retention time: time, certain size