The upcoming 2.8 release of Apache Kafka will replace ZooKeeper with Kafka’s internal Quorum controller. So for the first time, users can execute Kafka without ZooKeeper at all. This will save computing resources. It also makes Kafka more efficient and supports larger clusters.
In the past, Apache ZooKeeper was the key to distributed systems such as Kafka. ZooKeeper played the role of coordinating the proxy. When all the proxy servers started up, they would connect to ZooKeeper for registration. ZooKeeper is a powerful tool, but ZooKeeper is a stand-alone piece of software that complicates Kafka’s entire system, so officials decided to use an internal Quorum controller instead of ZooKeeper.
The work began in April last year, and now, with partial results, users will be able to run Kafka without ZooKeeper in version 2.8, officially known as Kafka Raft Metadata Schema (KRaft). In the KRaft model, metadata previously operated by the Kafka controller and ZooKeeper will be incorporated into the new Quorum controller and executed within the Kafka cluster, or on dedicated hardware if the consumer has a special usage situation.
The KRaft protocol uses an event-driven mechanism to track metadata for an entire cluster. Tasks that previously had to rely on RPC now benefit from event-driven and actual log transport. The benefit of these changes is that Kafka still supports more segmentation.
In the past Kafka was considered a clunky infrastructure with ZooKeeper, but with ZooKeeper removed Kafka is lighter and more suitable for small workloads, with lightweight singleton applications suitable for edge and lightweight hardware solutions.
