Original: Taste of Little Sister (wechat official ID: XjjDog), welcome to share, please reserve the source.

Celebrate! The latest version of Kafka, Kafka 2.8.0, removes the dependency on Zookeeper and manages its own cluster through KRaft. Good, good, good, good, good, good.

When we hear KRaft, we think of the Raft protocol. Raft protocol is the most popular distributed coordination algorithm today and is the basis for systems such as Etcd and Consul. Now Kafka has one.

The feature is so new that ZooKeeper will be used by default in version 2.8.0, but that doesn’t stop us from trying it out. Also, don’t get too excited, some of the features are officially not perfect, so don’t use it online.

Welcome to star: github.com/xjjdog/bcma… . It includes ToB complex business, Internet high concurrency business, cache applications; DDD, microservices guidance. Model driven, data driven. Understand the evolution of large-scale services, coding skills, Learning Linux, performance tuning. Docker/ K8S power, monitoring, log collection, middleware learning. Front-end technology, back-end practices, etc. Main technology: SpringBoot+JPA+Mybatis-plus+Antd+Vue3.

1. How to start KRaft?

Kafka replaces ZooKeeper with embedded KRaft, which is a big step forward because in distributed systems like ES, the synchronization of cluster meta information is self-cyclic.

But how do you start with KRaft? Many students directly dizzy, this aspect of the data is relatively little, but it is very simple to use.

We noticed that, under the Config directory, there was a new directory called Kraft, which contains a new set of configuration files that directly eliminates ZK dependency.

With the following three lines, you can start a single broker without ZK from start to finish.

# ./bin/kafka-storage.sh random-uuid
# ./bin/kafka-storage.sh format -t TBYU7WMiREexuZqrjKG60g -c ./config/kraft/server.properties
# ./bin/kafka-server-start.sh ./config/kraft/server.properties
Copy the code

After a crackling run, Kafka for No ZK is up and running.

It’s that simple.

2. How is it configured?

Kafka has added an internal theme called @metadata to store this meta information.

Now let’s look at some key configuration information. You can use vimdiff config/server. The properties of config/kraft/server properties have a look at these main difference.

First, Kraft added a configuration called Process.Roles. In our configuration file it looks like this.

process.roles=broker,controller 
Copy the code

It actually has three values.

  • brokerThis machine will act only as a broker
  • controller: starts as one of Raft Quorum’s controllers
  • broker,controller: Contains the functions of both

Students familiar with ES can see that these divisions are just like master and Node of ES, so the concept of distribution is actually interlinked to a certain extent.

The next step is to listen for address changes, because our server has two functions, so we need to open two ports.

listeners=PLAINTEXT://:9092,CONTROLLER://:9093
Copy the code

There’s also something called Node. id. Unlike the original broker. Id, this nodeID is used for voting.

node.id=1   
Copy the code

Because of the nature of the RAFT protocol, our voting configuration uses node.id above. It’s weird to write, isn’t it? But it’s a lot better than Zk’s. So these configurations are subject to change in later versions.

controller.quorum.voters=1@localhost:9093
Copy the code

This is the main difference in configuration files. So let’s look at the set.

process.roles=broker,controller 
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
node.id=1   
controller.quorum.voters=1@localhost:9093
Copy the code

3) Why kill ZK?

It’s a joke that Kafka, as a message queue, relies on a heavyweight coordination system called ZooKeeper. Also as a message queue, RabbitMQ is self-managing early on.

Zookeeper is cumbersome and requires an odd number of nodes to be configured in a cluster, making it difficult to expand or shrink the cluster capacity. Zk is configured in a completely different way from Kafka, and it’s frustrating to tune Kafka with another system.

In order for Kafka to evolve into lightweight, out-of-the-box Kafka has to kill Zk.

In addition, since Zk and Kafka are not in the same storage system after all, data synchronization issues become significant as the number of topics and partitions increases in size. Zk is reliable, but it’s slow, and nowhere near as good as in Kafka’s log storage system, which Kafka has to get around because of its reputation for speed.

Those of you who have used Kafka-admin are familiar with the slow synchronization of monitoring data. It needs to go around zK to get some metadata information, and then pull data from Kafka’s JMX interface. One swing at a time, it almost killed a large cluster.

4. What will change?

Deployment is simpler.

First, deployment is much simpler. For less high-availability systems, even a single process can get kafka up and running. We don’t need to apply for ZooKeeper-friendly SSDS anymore, and we don’t need to worry about whether zK has enough capacity.

Monitoring is easier.

Second, because of the centralization of information, it’s easy to get monitoring information from Kafka without having to go all the way to ZK. Integration with grafana/ Kibana/Promethus systems is just around the corner.

Faster.

Of course, the most important thing is speed. Raft is easier to understand and more efficient than ZK’s ZAB protocol, the partition primary election will be faster, and the controller scheduling speed will be improved.

There will never be a connection like this again.

zookeeper.connect=zookeeper:2181	
Copy the code

Instead, only the bootstrap connection is left. Kafka’s nodes are more and more like peers.

bootstrap.servers=broker:9092
Copy the code

Kafka also provides a tool called kafka-metadata-shell.sh that allows you to see the distribution of topics and partions, information that was previously available through ZK, now available from the command line.

$ ./bin/kafka-metadata-shell.sh  --snapshot /tmp/kraft-combined-logs/\@metadata-0/00000000000000000000.log
>> ls /
brokers  local  metadataQuorum  topicIds  topics
>> ls /topics
foo
>> cat /topics/foo/0/data
{
  "partitionId": 0."topicId" : "5zoAlv-xEh9xRANKXt1Lbg"."replicas": [1]."isr": [1]."removingReplicas" : null,
  "addingReplicas" : null,
  "leader" : 1,
  "leaderEpoch": 0."partitionEpoch" : 0
}
>> exit
Copy the code

Finally, do not enable this function on the online environment at present, or old honest use ZK bar. Features are the reason, because the infrastructure for those features is not in place and the code is not comfortable enough. If you do, chances are you’ll be devastated by incomplete tools or stubborn bugs.

Welcome to star: github.com/xjjdog/bcma… . It includes ToB complex business, Internet high concurrency business, cache applications; DDD, microservices guidance. Model driven, data driven. Understand the evolution of large-scale services, coding skills, Learning Linux, performance tuning. Docker/ K8S power, monitoring, log collection, middleware learning. Front-end technology, back-end practices, etc. Main technology: SpringBoot+JPA+Mybatis-plus+Antd+Vue3.

But the brave first step has been sold, the direction has been pointed, and all we have left is to wait. Either way, killing Zk is a good thing.

Xjjdog is a public account that doesn’t allow programmers to get sidetracked. Focus on infrastructure and Linux. Ten years architecture, ten billion daily flow, and you discuss the world of high concurrency, give you a different taste. My personal wechat xjjdog0, welcome to add friends, further communication.