“This is the 19th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021”

server.properties

  • broker.id=0

    • A Kafka cluster is a cluster of nodes. Each node is called a broker. Each broker has a different brokerId, specified by broker.id, which is an integer not less than 0. Brokerids must be different, but they do not have to be consecutive. If we want to extend the Kafka cluster, we can simply introduce new nodes and assign a different broker.ID.
    • When a Kafka cluster is started, each broker instantiates and starts a kafkaController and registers the Broker’s brokerId with the appropriate zooKeeper node. Each broker in the cluster selects one broker as the leader according to the election mechanism, that is, the leader kafkaController. The leader kafkaController is responsible for creating and deleting themes, managing partitions and replicas, and so on. When the leader kafkaController is down, another broker elects a new leader kafkaController.

  • log.dir = /export/data/kafka/

    • Where does the broker persist messages to, the data directory

  • log.retention.hours = 168

    • Minimum log file lifetime. The default value is 168 hours, that is, 7 days. The same is true of log.retention. Minutes and log.retention. Ms. Retention means retention.
    • After the maximum amount of time the data store is stored, data is processed according to a policy set by log.cleanup.policy, which is how long the consumer can consume data.
    • Either log.retention. Bytes or log.retention. Hours will be deleted and overridden by the parameters specified at topic creation time.

  • log.retention.check.interval.ms

    • How often do you check if there are log files to delete? The default value is 300000ms, that is, 5 minutes.

  • log.retention.bytes

    • Limit the maximum number of log files in a single partition, beyond which old logs will be deleted to satisfy that the number of log files does not exceed this value. The default is -1, that is, no limit.

  • log.roll.hours

    • How long does it take to generate a new log segment? The default is 168h, or 7 days. Log.roll. Ms, segment. Ms

  • log.segment.bytes

    • A new log segment is generated when the segment is old. The default value is 1073741824, that is, 1 GB.

  • log.flflush.interval.messages

    • Specify that the broker flush messages from memory to hard disk every few messages it receives. The default is 9223372036854775807.
    • Kafka officially does not recommend using this configuration, and recommends using duplicates and the operating system’s background refresh feature, as this is more efficient. This configuration allows you to set different values for different topics, that is, when you create a topic.
    • Supplementary notes:
    In Linux, when data is written to the file system, it is stored in the operating system's Page cache and not flushed to disk. If the operating system crashes, the data is lost. 1. Kafka is multi-replica, when you configure synchronous replication. Multiple copies of data are stored in the Page cache. The probability of multiple copies failing at the same time is much lower than that of one copy failing. 2. If your application calls fsync every time it writes data, the performance cost is high, so it is often a trade-off between performance and reliability. Because for an application, although the application hangs, as long as the operating system does not hang, the data will not be lost.Copy the code

  • log.flflush.interval.ms

    • Specifies how many milliseconds it takes for the broker to flush messages from memory to disk. Default value with the flflush. Interval. The messages, 9223372036854775807.
    • With the flflush. Interval. The messages, kafka’s official does not recommend using this configuration.

  • delete.topic.enable=true

    • Whether topics are allowed to be physically deleted

Kafka – eagle overview

In a production environment, in a Kafka cluster, message data variation is a concern. When the business premise is not complex, we can use the Kafka command to provide a tool with a Zookeeper client tool to do our job easily. As the complexity of the business increases, with groups and topics, we feel powerless to use Kafka to provide command tools. Kafka monitoring systems are now particularly important, and we need to look at the details of consumer applications.

To make it easier for developers and service engineers to maintain Kafka clusters, there is a monitoring and management tool called Kafka-Eagle. This management tool makes it easy to discover which topics are distributed unevenly across the cluster, or partitions are distributed unevenly across the cluster. It supports managing multiple clusters, selecting replicas, reassigning replicas, and creating topics. Also, the admin tool is a great way to quickly browse the cluster,

Install kafka-Eagle

Environment requirements: Install JDK and start ZK and Kafka services

# start the Zookeeper zkServer. Sh # start start Kafka nohup. / Kafka - server - start. Sh/export/servers/Kafka/config/server properties 2 > &1 &Copy the code

Modify the local host file

192.168.200.20 kafka1
192.168.200.20 kafka2
192.168.200.20 kafka3
192.168.200.11 node1
192.168.200.12 node2
192.168.200.13 node3
Copy the code

Setting up steps:

  • Download kafka-Eagle source code package

    • Kafka-eagle.org/ kafka-eagle.gz kafka-eagle.eagle.gz kafka-eagle.eagle.gz kafka-eagle.eagle.gz kafka-eagle.eagle.gz kafka-eagle.eagle.gz
    • Code hosting address: github.com/smartloli/E…

  • Upload the installation package and decompress it:

    • Here we choose to install Kafak-Eagle on the third machine
    • If you want to decompress the zip file, you need to install command support.
      • yum install unzip
      • unzip xxxx.zip
    Upload the installation package to /export/softwares on node01 server. Then decompress CD /export/softwares/ unzip kafka-eagle-zip CD kafka-eagle-web/target/ tar -zxf kafka-eagle-web-2.0.1-bin.tar.gz  -C /export/serversCopy the code

  • Preparing the database:

    • Kafka-eagle uses a database to store some metadata information. For example, the mSYQL database can be used to store metadata information
    Create database eagle;Copy the code

  • Modify the kafka-Eagle configuration file

CD /export/ Servers /kafka-eagle-bin-1.3.2/kafka-eagle-web-1.3.2/conf vim system-config.properties kafka.eagle.zk.cluster.alias=cluster1 cluster1.zk.list=node1:2181,node2:2181,node3:2181 Kafka. Eagle. Driver = com. Mysql.. JDBC driver kafka. Eagle. Url = JDBC: mysql: / / 10.1.192.208:3306 / eagle kafka. The eagle. The username = root kafka.eagle.password=wu7787879Copy the code

By default, MySQL only allows native connections to MySQL instances, so if you want remote access, you must open permissions:

Update user set host =' %' where user ='root'; // Refresh the configuration of FLflush PRIVILEGES;Copy the code
  • Configuring environment Variables
    • Kafka-eagle must be configured with environment variables. The node03 server runs the following command to configure environment variables
Vi /etc/profile Export KE_HOME=/export/servers/kafka-eagle-bin-1.3.2/kafka-eagle-web-1.3.2 export PATH=:$KE_HOME/bin:$PATH # Implement the source/etc/profileCopy the code
  • Start the kakfa – eagle
CD kafka-eagle-web-1.3.2/bin chmod u+x ke.sh./ke.sh startCopy the code
  • Access the main interface:http://node03:8048/ke/account/signin?/ke/
    • User name: admin
    • Password: 123456