7. [Kafka operation] Kafka-consumer-groups. sh Consumer group management

This is the 12th day of my participation in the August More Text Challenge. For details, see: August More Text Challenge

@[TOC]

How can didi open Source LogiKM one-stop Kafka monitoring and control platform be absent from daily operation and maintenance and troubleshooting

Consumer group management kafka-consumer-groups.sh

1. Look at the list of consumers--list

sh bin/kafka-consumer-groups.sh --bootstrap-server xxxx:9090 --list

A MetadataRequest is called to get a list of all online brokers and then a ListGroupsRequest is sent to each Broker requesting consumer group data

2. View details about the consumer group--describe

DescribeGroupsRequest

View consumer group details--group 或 --all-groups

–group sh bin/kafka-consumer-groups.sh –bootstrap-server XXXXX :9090 –describe –group test2_consumer_group

---

–all-groups sh bin/kafka-consumer-groups.sh –bootstrap-server XXXXX :9090 –describe –all-groups All the topics consumed, and where the partitions are located, the latest consumption offset, Log the latest data offset, the amount of Lag not yet consumed, the consumer ID, and so on

Example Query information about a consumer member--members

Sh bin/kafka-consumer-groups.sh –describe –all-groups –members –bootstrap-server XXX :9090 bin/kafka-consumer-groups.sh –describe –members –group test2_consumer_group –bootstrap-server xxxx:9090

Example Query consumer status information--state

Status information about all consumer groups sh bin/kafka-consumer-groups.sh --describe --all-groups --state --bootstrap-server xxxx:9090 Specifies the status of a consumer group sh bin/kafka-consumer-groups.sh --describe --state --group test2_consumer_group --bootstrap-server xxxxx:9090

3. Delete the consumer group--delete

DeleteGroupsRequest

Delete a consumer group –delete

Sh –delete –group test2_consumer_group –bootstrap-server XXXX :9090 –all-groups sh bin/kafka-consumer-groups.sh –delete –all-groups –bootstrap-server XXXX :9090

PS: If you want to delete a consumer group, all clients of the consumer group have stopped consuming/offline. Otherwise, the following exception is reported

Error: Deletion of some consumer groups failed:
* Group 'test2_consumer_group' could not be deleted due to: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.GroupNotEmptyException: The group is not empty.

Copy the code

4. Reset the offset of the consumer group--reset-offsets

A prerequisite for successful execution is that the consumer group will be unavailable;

The following example uses the following parameters: –dry-run; This parameter represents pre-execution, which prints out the results to be processed; Use the excute parameter when you want to actually execute it;

The following examples of reset modes are all to-earliest reset;

Please refer to the following related reset Offset mode to another mode as required;

Resets the offset of the specified consumer group--group

–all-topic sh bin/kafka-consumer-groups.sh –reset-offsets — to-groups –group Test2_consumer_group –bootstrap-server XXXX :9090 –dry-run –all-topic Resets the offset of the specified topic for the specified consumer group –topic sh bin/kafka-consumer-groups.sh –reset-offsets –to-earliest –group test2_consumer_group –bootstrap-server xxxx:9090 –dry-run –topic test2

Resets offsets for all consumer groups--all-group

–all Topic sh bin/kafka-consumer-groups.sh –reset-offsets — to-groups –all-group –bootstrap-server XXXX :9090 –dry-run –all-topic Resets the offset for the specified topic in all consumer groups –topic sh bin/kafka-consumer-groups.sh –reset-offsets –to-earliest –all-group –bootstrap-server xxxx:9090 –dry-run –topic test2

–reset-offsets need to be followed by the reset mode

Related reset Offset mode

parameter	describe	example
--to-earliest :	Reset the offset to the original offset(find the earliest offset that has not been deleted)
--to-current:	Reset offset directly to the current offset, which is LOE
--to-latest:	Reset to the last offset
--to-datetime:	Reset to offset at the specified time; Format for:YYYY-MM-DDTHH:mm:SS.sss;	- to - a datetime "2021-6-26 T00:00:00. 000"
--to-offset	Reset to the specified offset, but more often than not, if multiple partitions are matched, here reset all partitions that are matched to this value; If 1. Target maximum offset<--to-offset, reset to the target maximum offset; 2. Target minimum offset>--to-offset, reset to minimum; 3. Otherwise, it will reset to--to-offsetThe target value of;You don’t usually use this	--to-offset 3465
--shift-by	How many offsets to increase or decrease in terms of the offset; Positive is increased forward; Negative goes backwards; And of course it matches all of them;	--shift-by 100 、--shift-by -100
--from-file	Reset according to the CVS document; I’ll talk about it separately here

--from-fileLet me focus on that

The other patterns above reset all matched partitions; Cannot reset to a different offset for each partition; **–from-file** allows us to be a little more flexible;

1. Configure the CVS document first

The format is: Topic: partition number: reset target offset CVS test2,0,100 test2,1,200 test2,2,300 2. Run the >sh bin/kafka-consumer-groups.sh –reset-offsets –group test2_consumer_group –bootstrap-server XXXX :9090 command –dry-run –from-file config/reset-offset.csv

5. Delete the offsetdelete-offsets

A prerequisite for successful execution is that the consumer group will be unavailable;

After the offset is deleted, the Consumer Group will consume from the beginning the next time it starts.

sh bin/kafka-consumer-groups.sh --delete-offsets --group test2_consumer_group2 --bootstrap-server XXXX:9090 --topic test2

This parameter is optional

More

Kafka column continues to be updated… (source code, principle, actual combat, operation and maintenance, video, interview video)

【 Kafka operation 】Kafka network most complete most detailed operation and maintenance command collection (boutique strongly recommended collection!! _ Shi Zhenzhen’s grocery store -CSDN blog

[Kafka actual] partition redistribution may appear problems and troubleshooting problems (production environment actual combat, dry goods!! Very dry!!!!!! Suggested collection)

【 Kafka exception 】 Kafka common exception handling solution (continuously updated! Suggested collection)

【 Kafka operation and Maintenance 】 Partition from allocation, data migration, copy expansion (with video)

Kafka source 】 【 ReassignPartitionsCommand source code analysis (duplicates of scalability, data migration, and redistribution, copy migration across the path

[Kafka] Go to…. for more