This is the 17th day of my participation in the August Challenge

Recommend a very good kafka management platform, Kafka soul mate didi open source Logi-Kafkamanager one-stop Kafka monitoring and control platform

Project Address: DIDI/Logi-Kafkamanager: one-stop Apache Kafka cluster indicator monitoring and operation and maintenance control platform

Operational controls

Operation and Maintenance Management This menu is mainly for operation and maintenance personnel to manage all clusters.

Cluster list

Kafka’s soul mate Logi-Kafkamanger (3) operations management – Cluster list

The cluster operations

Kafka logi-Kafkamanger (4) operation and Maintenance management — Cluster operation and Maintenance (data migration and online cluster upgrade)

Platform management

Application management

Manage all applications that use Kafka, Topic creation needs to be managed to the corresponding system (which system); Here are all the applications; If you want to see your own app; Check Topic Management -> Application Management

Application to apply for

Users with application application permission can apply for new applications. And then byOperations staffThe examination and approval; The place to apply isTopic Management -> Application Management -> Application Application There are at least two app owners

Application details

Show some basic application information, including AppId and key used in Topic authentication.

Operation and maintenance personnel and application responsible person can edit information, such as adding new responsible person;

Apply for offline

If the application is no longer in use, you can apply for offline application. In addition to here operation and maintenance management personnel can apply for offline; Application owners can apply for referrals at Topic Management -> Application Management.

This is shown hereThe connection informationYou need to configure didikafka-gatewayComponents are displayed; Otherwise, you can’t get the corresponding information;kafka-gatewayNo open source, please contact the official if necessary;

Demo TODO..

inApplying for an Application to be OfflineMake sure that all topics created under the application are offline, otherwiseOperations staffWhen you log off, you will be prompted:Before we go offline topic, we can go offline ~It should also be noted that if you apply for permission to use other topics, you need to cancel the permission first

The person in charge of the application needs to apply for Topic offlineTopic Management -> My Topic-> More -> Apply for referralNote that you can only apply for a referral if you are the app owner; If you only have partial access, you cannot apply for a referral.Again, this is shown hereThe connection informationYou need to configure didikafka-gatewayComponents are displayed; Otherwise, you can’t get the corresponding information;kafka-gatewayNo open source, please contact the official if necessary;

User management

O&m personnel manage users

User roles

KafkaManager has three user roles:

Operation and maintenance personnel: have all permissions on the platform. R&d personnel: have all permissions except for expert services. Ordinary users: ordinary developers, with only permissions for Topic management, cluster management, alarm monitoring and so on

In fact, the name of the character permissions is a bit confusing to understand, and at first I thought that the developers were just normal developers as we understand them; But in reality the average user: this character is the developer that we write the code to use Kafak; You only need to care about your Topic module; Research and development personnel: it has the rights of ordinary users, but it also has the rights of operation and maintenance control. The usage scenario is that the role may be the TeamLeader of a team. Or technical experts, he needs to understand the monitoring status of the entire physical cluster on the basis of ordinary users, and find some problems; The user roles and permissions of KM are still relatively rough. I have reported to the community, and the community responded that in the future, this section will be greatly changed to make a unified set of permissions and resources management.

Platform configuration

Internal configuration of some systems

Configure button: ADMIN_ORDER_HANDLER_CONFIGThe specified account has the approval permissionConfiguration values Demo: [ "shirc_10", "shirc1" ] Description:Many approvals need to be approved by operation and maintenance personnel; If o&M personnel are too busy to spend time on approval, they can assign approval permission to some accounts. In lieu of approval; At this time, operation and maintenance personnel do not have permission to approve, and the approval button is hiddenApplicants can see who can be approved through the details here; You can find the corresponding person to help approve;

REGION_CAPACITY_CONFIG Configures the default maximum flow capacity of the cluster Broker. REGION_CAPACITY_CONFIG Configures the default maximum flow capacity of the cluster Broker

The system tries to refresh the topics that have not been disaggregated (for example, the existing topics have not been disaggregated just after KM was connected) into DB every 2 minutes. Task.op. sync-topic-enabled: true By default,topic is a topic that is not logged into the DB. It’s not tied to an application. If you want it to be tied to a default application when you swipe DB, you can use the following configuration;

Whether SYNC_TOPIC_2_DB_CONFIG_KEY is bound to specific applications and permissions when it periodically flushes existing topics to DB; [ { “clusterId”: 4, “defaultAppId”: “dkm_admin”, “addAuthority”: true }, { “clusterId”: 5, “defaultAppId”: “dkm_admin”, “addAuthority”: true } ]

ClusterId: ID of the physical cluster defaultAppId: ID of the application bound to the topic by default addAuthority: Specifies whether to grant the read and write permissions to the application.

Usage Scenarios:It feels useless. Even if we are not bound to the application at this time, we can still bind the corresponding application for Topic one by one later.

Some other configuration is not very important not to list, have interest can directly look at the source code

The gateway configuration

Used with The Kafka-gateway component of Didi; The open source version is not a concern

Gateway configuration details TODO

Column list

Kafka’s soul mate Logi-Kafkamanger (1) cluster access and related concepts explained

Logi KafkaManger(2) Kafka For Topic granularity quota management (limiting)

Kafka’s soul mate Logi-Kafkamanger (3) operations management – Cluster list

Kafka logi-Kafkamanger (4) operation and Maintenance management — Cluster operation and Maintenance (data migration and online cluster upgrade)

Didi open source Logi-Kafkamanager one-stop Kafka monitoring and control platform

Welcome to add personal wechat to pull you into the development of technical exchange group, the group of specials to answer technical questions (please note: technology) WX: JJDLmn_ or WX: Mike_zhangliang

✌🏻 if this article can help you, please send bloggers with one button! Our initiative is not easy