This paper is participating in thePerformance optimization combat record”Topic essay activity

preface

Performance testing is exciting because it is one of the few skills in testing that requires deep technical knowledge;

But getting started is easier than digging deep. If you want to become a master in this technology, it is difficult to achieve without 5-6 years of technical accumulation;

The key is that the occurrence of performance problems is irregular, but there are certain routines in it, need to learn and summarize experience.

Here I also suggest: learning does not have to take your own company’s project, but to take other people’s open source project to practice, set their own performance problems, and then understand the nature of performance problems through the phenomenon, in order to achieve the purpose of performance optimization.

Environment to prepare

It’s not just environment setup, but script development, data construction, monitoring platforms, and more that are ready to perform performance test scenarios

  • Recommend the Jforum system set up in front, the script is also completed in front of the development, just need to observe and analyze the results according to the scene
The scene of browsing posts
  • Run scenario: Set 30 threads, no thought time, last 5 minutes…
  • Scenario Symptom: After a period of time, the TPS decreases sharply, the service does not respond, the log reports an error, and the SQL file does not slow. The server CPU is 100%, the average load spikes and falls, and the memory is normal
  • Analyze logs: %TOMCAT_HOMT%/logs/catalina.out
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd5f811d800 nid=0xc94 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd5f811b000 nid=0xc93 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd5f810c800 nid=0xc92 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd5f80dc000 nid=0xc90 in Object.wait() [0x00007fd5e8bfa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x00000000f81c9a98> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fd5f811d800 nid=0xc94 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fd5f811b000 nid=0xc93 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fd5f810c800 nid=0xc92 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd5f80dc000 nid=0xc90 in Object.wait() [0x00007fd5e8bfa000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x00000000f81c9a98> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Copy the code
Performance diagnosis

The top command is used to monitor CPU resources, find process PID that occupies too much CPU, and jSTACK to search for thread status. The BLOCKED thread appeared in the log analysis, and the reasons were as follows: broadband, middleware thread pool, database connection pool, etc. The possible cause is that the tomcat middleware thread pool is insufficient. New connections cannot be created because the thread pool is full.

Configuration optimization

Change the tomcat thread pool from 20 to 300

Repeated testing

Performance testing is an iterative process, testing until you can’t find a problem.

  • Run with a scenario and find a sharp drop in TPS after a period of time, then analyze the service log:
Exception in thread "http-bio-8080-exec-28" Exception in thread "http-bio-8080-exec-27" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
	at sun.nio.cs.StreamDecoder.<init>(StreamDecoder.java:251)
	at sun.nio.cs.StreamDecoder.<init>(StreamDecoder.java:231)
	at sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:69)
	at java.io.InputStreamReader.<init>(InputStreamReader.java:100)
	at freemarker.cache.FileTemplateLoader$4.run(FileTemplateLoader.java:210)
	at freemarker.cache.FileTemplateLoader$4.run(FileTemplateLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at net.jforum.util.legacy.clickstream.ClickstreamFilter.doFilter(ClickstreamFilter.java:59)
java.lang.OutOfMemoryError: GC overhead limit exceeded
Copy the code

See the OutOfMemoryError, make sure the previous guess is correct and optimize the service configuration

Performance analysis

There is a problem with the OOM memory, that is, the GC heap overhead is out of limit, it is possible that the application is constantly GC but too slow to be used by the next thread. There are, therefore, may be program memory allocation issues, even if add configuration: – XX: – UseGCOverheadLimit, will throw an exception: Java. Lang. OutOfMemoryError: Java heap space, can be explained as the heap memory overflow.

Heap memory Cause: Tomcat JVM parameters need to be adjusted to increase the heap memory Xms\Xmx

  • JVM parameter optimization

Add tomcat thread pool, adjust Tomcat JVM parameters 128M to 1g

Run the scene again

  • As a result, there was no abnormal situation, and 30 threads were supported to be continuously concurrent. TPS rose and reached a stable process, while RT did not increase and remained within 3S.