This is the 16th day of my participation in the August More Text Challenge. For details, see: August More Text Challenge
Recommend a very good Kafka management platform, Kafka soul mate Didi open source Logi-Kafkamanager one-stop Kafka monitoring and control platform
Technical communication
If you want to enter didi LogI open source user group, add my personal wechat: jjdLmn_ group (note: group). The group mainly exchanges Kakfa, ES, Agent, Logi-Kafka-Manager, and other related technologies; There are people in the group to answer your questions. There are people in the related technical field to answer your questions. Everything you ask will be answered
If you want to enter didi LogI open source user group, add my personal wechat: Jjdlmn_ group (note: group). The group mainly exchanges kakFA, ES, Agent, and other technology groups. There are specially-assigned people to answer your questions, and you can get the response
@[TOC]
Project address: Didi/Logi-Kafkamanager: one-stop Apache Kafka cluster indicator monitoring and operation and maintenance management platform
Operational controls
The operation and maintenance control menu is mainly for the operation and maintenance personnel to manage all clusters.
Cluster list
Logi-kafkamanger is a soulmate of Kafka
The cluster operations
Migration task
Kafka migration scenarios generally include data migration in the same cluster and data migration across clusters. We mainly talk about the same cluster data migration here; Data migration between clusters. For example, when a new Broker node is added to the existing cluster, the existing Topic data in the original cluster needs to be migrated to the new cluster to relieve the cluster pressure.
Before looking at KM migration, let’s take a look at how migration normally works;
The manual migration process is implemented
The partition reallocation tool can be used to migrate some topics from the current Broker node to the newly added Broker. This is often useful when extending an existing cluster, as it is easier to move an entire Topic to a new Broker, rather than one partition at a time. When doing this, the user needs to provide the Topic list of the existing Broker nodes and the list of brokers to the new node (the mapping between the source Broker and the new Broker). The tool is then evenly distributed across all partitions of a given Topic list within the new Broker. The replication factor for the Topic remains the same during the migration.
As an example, move all partitions from Broker1 with Topic KE01 and KE02 to the new Broker2 and Broker3. Since the tool accepts the input list of topics as JSON files, you need to specify the Topic to be migrated and create the JSON file, as shown below:
> cat topic-to-move.json
{"topics": [{"topic": "ke01"},
{"topic": "ke02"}],
"version":1
}
Copy the code1. Prepare a JSON file and use the partition reallocation tool to generate candidate allocations.
> bin/kafka-reassign-partitions.sh --zookeeper dn1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "1, 2," --generate
Copy the codeAfter the command is executed, the following information appears on the console:
{"version":1,"partitions":[{"topic":"ke02","partition":0,"replicas":[2]},{"topic":"ke02","partition":1,"replicas":[1]},{ "topic":"ke02","partition":2,"replicas":[2]},{"topic":"ke01","partition":0,"replicas":[2]},{"topic":"ke01","partition":1 ,"replicas":[1]},{"topic":"ke01","partition":2,"replicas":[2]}]}Copy the code2. Run the following command:
> ./kafka-reassign-partitions.sh --zookeeper dn1:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
Copy the code3. Finally, the — Verify option can be used with the tool to check the status of partition reassignments. Note that the same expa-cluster-reassignment. Json (used with the –execute option) should be used with the –verify option, executing the following command:
> ./kafka-reassign-partitions.sh --zookeeper dn1:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
Copy the codeThe execution result is as shown in the figure below:
Several considerations for data migration
Reduce the amount of data to be migrated: If the Topic to be migrated has a large amount of data (Topic retains data for 7 days by default), you can temporarily and dynamically adjust retention. Ms to reduce the amount of data before the migration. Kafka will actively purge data up to an hour ago;
> bin/kafka-topics --zookeeper localhost:2181 --alter --topic sdk_counters --config retention.ms=3600000
Copy the codeDo not be careful to revert to the original Settings after the migration is complete
Kafka provides traffic limits for replicas between brokers, limiting the bandwidth limit for replicas to travel from one machine to another, which is useful when rebalancing the cluster, introducing new brokers, and adding or removing brokers. Because it limits these intensive data operations, such as the migration we did above, the impact on users can be ensured
> ./kafka-reassign-partitions.sh --zookeeper dn1:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
Copy the codeYou can see that you are throttling all the data you are throttling 50000000 B/s
The throttle limit was set to 50000000 B/s
Successfully started reassignment of partitions.
Copy the codeThe throttle parameter is used to limit the traffic that is transmitted between brokers. It limits the bandwidth limit that can be used to transmit copies from one machine to another. But you should be aware that under normal circumstances, replicas directly have replicas of synchronized traffic; If the limit is lower than the value of the normal replica synchronization, the replica synchronization will be abnormal and the speed that cannot keep up with the Leader will be kicked out of the ISR.
When the migration is complete, the traffic restriction will be removed: if we add migration, we need to verify the execution status and the traffic restriction will be removed. Otherwise, the traffic of periodic replication operations may be limited.
> ./kafka-reassign-partitions.sh --zookeeper dn1:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
Copy the codeSee OrcHome for details on how Kafka limits bandwidth use during data migration
Logi-kafkamanager Implements data migration
Once you’ve seen the manual migration process, let’s look at the KM migration action, and you’ll love it; Because the migration operation is greatly simplified;
The following figure shows the operation of creating a migration task. Explain some of the parameters;
Retention time of the Topic after migration: As explained in the migration precautions above, it is necessary to reduce the amount of data to be migrated; If you keep 7 days’ worth of data by default, the amount of migrated data is likely to be very large, and a lot of the data has already been consumed. So we need to clean up so much expired data before starting migration; This parameter specifies how long the data has been saved; Delete outdated data; After the migration, the time will be changed back to the original time.
Initial current limiting: current limiting online: current limiting line: you may see these parameters will be very strange, the current limiting is not a certain value, fill a current limiting value on the line, do so much is to do what;
In fact, WHAT KM wants to do is dynamically adjust the traffic limiting, which can be dynamically adjusted according to different time and cluster status. For example, in idle time, I can allow your traffic to reach 100 mbit /s(traffic limiting online). But if you are likely to be stressed during migration, I don’t want you to start with this 100M/s limit; Initial flow limiting is used at the beginning of the migration, but the flow limiting cannot be too small, because the traffic in the normal case of replica synchronization should be considered, so there is a flow limiting line. The KM then checks the migration status at regular intervals (1 minute) and dynamically adjusts the current limits.
Now, of course, KM is actually using the initial current limit as the current limit; There is no dynamic adjustment of the flow rate; This is a point that needs to be modified in the future;
After the migration task is created, the KM timer detects that the start time has reached, and the migration begins formally. The process is the same as the migration process mentioned above, but the program does it for us automatically;
If the data volume is large, you are advised to migrate the data during an idle period
Clustering task
This module is used to automate kafka cluster upgrades, but needs to be used with the Nightingale system (mainly to send upgrade packages to the server at KM). This feature is great for large clusters, automatic online upgrades; No need to manually operate;
Take a quick look at the usage diagram
How to connect the nightingale system? I will add the nightingale system when I have time. TODO
Version management
When creating a cluster task, you need to upload the Kafka upgrade package and the configuration file set
Platform management
Operation and maintenance of Logi-Kafkamanger (5) — Platform Management (User Management and Platform configuration)
Column list
Kafka’s soulmate logi-Kafkamanger a cluster access and related concepts
The logi-Kafkamanger is the soulmate of Kafka. The logi-Kafkamanger is the soulmate of Kafka. The Logi-Kafkamanger is the soulmate of Kafka.
Logi-kafkamanger is a soulmate of Kafka
Didi Open source Logi-Kafkamanager one-stop Kafka monitoring and control platform
Welcome to add personal wechat to pull you into the development of technology exchange group, the group dedicated to answer technical questions (please note: technology) WX: JJDlmn_ or WX: mike_zhangliang
If articles are helpful to you, please give bloggers three even a key ah, the original is not easy to your support is my motivation for output ✌ 🏻