Kafka online real environment combat and tuning advanced series

  • Kafka business environment combat – Kafka production environment planning
  • Kafka Business Environment in action – Kafka producer and consumer throughput test
  • Kafka business environment combat – Kafka Producer parameter setting and parameter tuning suggestions

This series of blogs summarizes and shares cases extracted from real business environments, and gives advice on tuning Kafka business applications and capacity planning for cluster environments. Please continue to follow this series of blogs. This set of Kafka tuning series copyright belongs to the author (Qin Kai Xin) all prohibited reprint, welcome to learn.

1. Kafka producer throughput test metrics

Kafka-producer-perf-test: a script provided by Kafka to test the performance of a producer. A script is used to calculate the average latency and throughput of a producer over a period of time.

1.1 kafka – producer – perf – test

In the kafka installation directory, run the following command. In a production environment, it makes sense to run the script as long as possible:

bin/kafka-producer-perf-test.sh –topic test –num-records 500000 –record-size 200 –througthput -1 –producer-props bootstrap.servers=bd-master:9092,bd-slave1=9092,bd-slave3=9092 acks=1

1.2 Analysis of test results is as follows:

500000 Records Sent,41963 Records/SEC (8.00 MB/ SEC), 2362.85ms/AVG latency,3513.00 ms Max Latency,2792ms 50h,3144ms 95th,3364 ms 99h,3503ms 99.9th

Kafka has an average throughput of 8.00 MB/ SEC, or about 64Mb/s of bandwidth, and sends an average of 41,963 messages per second. The average latency was 2362.85ms, the maximum latency was 3513.00ms, 95% messages took 3144ms, 99% messages took 3364ms, and 99.9% messages took 3503ms.

2. Kafka consumer throughput indicator description:

2.1 kafka – consumer – perfs

Bin /kafka-consumer-perfs-test.sh –broker-list bD-master :9092, BD-slave1 =9092, BD-slave3 =9092 –message-size 200 –messages 500000 –topic test

2.2 The following results are obtained:

2018-10-28 9:39:02 95.4188 92.2313 500271 484289 in this environment, a total of 95.4188MB messages were consumed in 1s, and the throughput was 92.2313MB/s, that is, 736Mb/s.

3 conclusion

Qin Kaixin was born in Shenzhen on October 28, 2018