This is the 10th day of my participation in the August More text Challenge. For details, see: August More Text Challenge

How can didi open source LogiKM one-stop Kafka monitoring and control platform be absent for troubleshooting daily operation and maintenance problems

1. The Leader epoch is outdated

The leader epoch in the request is older than the epoch on the broker

--

Partition $topicPartition marked as failed
Copy the code

The solution

The epoch of the current partition Leader is older than that of the Broker

So follow to fetchleader is an error; Just have the Leader election happen again;

2. An exception occurs when modifying Broker


Configured broker.id 0 doesn't match stored broker.id 1 in meta.properties.
 If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).

Copy the code

In this case, you may have changed the Broker’s configuration. Or the log.dir path is changed and the path already exists. You can check out meta-properties under log.dir

#Wed Jun 23 17:59:02 CST 2021
broker.id=0
version=0
cluster.id=0
Copy the code

This is the previous configuration. If you change the broker. Id to be different from this, you will raise an exception.

The solution

If the log.dir belongs to the Broker, then change the Broker. Id of server.properties to be more consistent with the meta. Then you need to put broker. Id in meta-.properties; The ultimate goal is to make meta-.properties consistent with the broker. Id in server.properties;

If log.dir is used for old data, you may want to change the path; Change the path to log.dir in server.properties

Modify the Broker. Id possible exceptions

BrokerId is not really recommended; BrokerId changes can be problematic, for example

  1. Data is being migrated. The ones on zK are still the original onesbroker.Id; This causes the Broker migration to fail
  2. When you modifybroker.Id; If dynamic configuration is configured, it will not take effect; So remember to add the old dynamic configuration back; The ZK node is:/config/brokers/{brokerID}
  3. other

meta.propertiesrole

You can also understand why the meta. Properties file exists; It is used to keep the Broker. Id and cluster.id=0 and version before log.dir; Because you can change this configuration in server.properties, errors are inevitable. Kafka will compare your configuration to this meta. Properties information and alert you that your configuration is incorrect.

Failed to acquire lock on file. Lock in

 Failed to acquire lock on file .lock in /Users/xxxx/work/IdeaPj/xxx/kafka/kafka-logs-0. A Kafka instance in another process or thread is using this directory.

Copy the code

Cause of exception:

The Broker starts with a file lock on log.dirs to prevent other programs from tampering with it. This exception indicates that a program has locked the folder; So get failed;

The solution

Check that the Broker is already started or that log.dirs is configured in the same folder in both Broke. If you are sure there are no problems, you can delete the.lock file in the corresponding folder. Force lock file removal; (Not recommended)

4. The message “UNKNOWN_TOPIC_OR_PARTITION” is sent

WARN [Producer clientId=console-producer]  Error while fetching metadata with correlation id :  {test80=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
Copy the code

Cause of exception:

Description The TopicPartition to be sent does not exist. Either the Topic does not exist or the Partition sent to it does not exist

The solution

  1. Check to see if Topic does not exist
  2. Check that the sending Partition’s Broker is down, causing the sending to fail (especially if the Partition number is specified when sending the message)
  3. Check whether all the Topic brokers are down;

5. Error while reading checkpoint file xxxx/cleaner-offset-checkpoint

 Error while reading checkpoint file /Users/shirenchuang/work/IdeaPj/didi_source/kafka/kafka-logs-2/cleaner-offset-checkpoint
Copy the code

6. InconsistentBrokerMetadataException

kafka.common.InconsistentBrokerMetadataException: BrokerMetadata is not consistent across log.dirs. This could happen if multiple brokers shared a log directory (log.dirs) or partial data was manually copied from another broker. Found:
- kafka-logs-0 -> BrokerMetadata(brokerId=0, clusterId=0)
- kafka-logs-1 -> BrokerMetadata(brokerId=1, clusterId=0)
Copy the code

Cause of exception:

If multiple log.dirs log folders are configured on the same Broker, but the two folders belong to different brokers, an exception will be thrown. Assume that the configuration file log.dirs=kafka-logs-1,kafka-logs-0 is configured with two folders. BrokerMetadataMap object is created using broker. Id,cluster. Id, and broker. Normally, these values are the same, but if multiple brokers are deployed on the same machine and you want to share the same dir, then you cannot.

The solution

If you want to configure more than one dir, find out which one is already used by other brokers.

Log. dir Failed to load XXX during broker startup

Failed to load ${dir.getAbsolutePath} during broker startup
Copy the code

Cause of exception:

When reading the meta. Properties file in log.dirs folder during startup, an IOException was thrown

The solution

Check whether there is an exception in the meta. Properties file in the corresponding dir.

Duplicate log directory found: xxxx
Copy the code

Cause of exception:

Log. dirs set folder is duplicate; For example: the dirs = kafka – logs – 0, kafka – logs – 0

The solution

Check to see if the Settings are repeated

 Found directory /xxxx/kafka/kafka-logs-0/test, 'test' is not in the form of topic-partition or topic-partition.uniqueId-delete (if marked for deletion).
Kafka's log directories (and children) should only contain Kafka topic data.
Copy the code

Cause of exception:

The log.dirs folder contains folders that do not meet the conditions. Generally, the folders in the folder format are TOPic-partition ID, topic-partition ID – Future, and topic-partition ID -delete

The solution

Check the folders that fail

8. Meta. Properties Version information is incorrect

[the 2021-07-21 13:38:19, 246] [ERROR] [the main] : Failed to create or validate data directory /Users/xxx/kafka/kafka-logs-0 java.io.IOException: Failed to load /Users/xxxx/kafka/kafka-logs-0 during broker startupCopy the code

Cause of exception:

The version information in meta. Properties is abnormal. Normally, it is 0.

The solution

Try to delete the meta. Properties directly, it will be regenerated upon startup

About the author: Shi Zhenzhen’s grocery store focuses on knowledge sharing in the Java field and the big data field. The content is mainly principle, source code, actual practice and so on. It insists on output dry goods, and the written content must be verified, and in-depth source code analysis to ensure the accuracy of the content. For a long time in CSDN, and the public number [Shi Zhenzhen’s grocery store] published original articles, welcome to pay attention! If you have questions related to the technical field, welcome to the group communication, each field has a person to answer, you ask, will get a response!




Didi Open source Logi-Kafkamanager one-stop Kafka monitoring and control platform