The main function of ydata-server is to receive upstream Kafka data, and then serialize, classify, and send it to Kafka. Deploy a Java 8-based version.
Before optimization
- Memory usage
- Garbage Collection
The pressure test lasted for more than 1 hour, and 4 Full GC occurred. The throughput looks ok.
- Configuration heap memory parameters:
Exceeding the container’s memory limit was indeed detected…
First optimization
- Optimization idea
All functions of ydata-server are simple consumption of data, serialization, etc., with no logic to use direct memory.
All the functions of ydata-server are performed in heap memory.
In the allocation of heap space, the proportion of new generation is 1/3, and that of old generation is 2/3. However, in the observation of garbage collection data, garbage collection does not happen many times in the old generation, and occupies very little memory.
It is obvious that the new generation memory, which is used frequently, only accounts for 1/3 of the heap memory, while the old generation memory, which accounts for 2/3 of the heap memory, has been idle for a long time, resulting in high memory utilization.
Therefore, the proportion of new generation memory can be appropriately increased. -xx :NewSize Specifies the memory size of the new generation.
When observing garbage collection data, it was also found that ydata-server had 4 Full GC. Although 4 Full GC is still a small number, the causes of the occurrence are worth in-depth analysis.
The main logic of ydata-server is to receive data from Kafka, serialize it, and so on. These operations are very fast, and when they are done, the associated objects should be recycled. In theory, ydata-server should only cause new generation garbage collection, not old generation garbage collection (Parallel garbage collector, old generation Full GC parallelization).
So what caused recycling in the old days?
The direct cause is the accumulation of old objects. (bs)
So what caused the accumulation of old objects?
The following figure shows the general allocation process of objects. It can be seen that when the memory of S area is insufficient to put the surviving objects in Eden area, the objects will be directly put into the old age
Looking at the previous garbage collection situation, the size of s-zone is only 3M, that is to say, if a batch of data about 3M survives in Eden Zone, it will directly enter the old era, resulting in the accumulation of the old era.
Ydata-server also provides a rest interface, and if the data returned by the interface is in the 3M range, it will probably go straight to the old age.
Check the interface of the front-end page, and find that there are some interfaces, the data returned is relatively large.
Resending the request, and monitoring the heap memory in real time, it turns out that memory in the old years has increased slightly. Many times of page query, resulting in the accumulation of old years, but in general page query is not frequent (still in the testing stage), so garbage collection is only a few times.
To do this, you can increase the size of the S-region of heap memory to make it fully functional. Parallel will automatically adjust the size of the S-region to achieve maximum system throughput, and the s-region will get smaller and smaller… This can be resolved by disabling automatic adjustment (-xx: -useadaptivesizePolicy) or using the CMS GC.
- To optimize the parameters
-Xmx2g -Xms2g -XX:NewSize=1g -XX:-UseAdaptiveSizePolicy
Copy the code- Memory footprint
- Garbage Collection
// Screenshots of the scene can not be found, only text description
There were still 3 Full GCS, but these 3 Full GCS had occurred before the manometry. The Full GC was never sent again during the manometry.
The new generation garbage collection is very frequent, with roughly one Young GC per second and each one taking around 130 milliseconds, meaning that JVM garbage collection throughput is around 87%, which is not very good.
New generation garbage collection is frequent, there is no way, the amount of data over so much, so, can only increase the memory size of the new generation, and the old age, memory stability, do not have to increase.
Second optimization
- Optimization idea
Garbage collection after startup:
As you can see, there are three Full GC runs, but the footprint of the older generation is very small. So, somewhere else should trigger the Full GC.
In addition to heap memory, Full GC is also triggered when the method area is insufficient. At startup, the method area memory is too small and is expanded, and during expansion, Full GC occurs.
Therefore, the 3 Full GCS at startup may be caused by the rising of the meta space water level.
You can specify the size of the MetaspaceSize to avoid this problem. For example, -xx :MetaspaceSize=256m. Parameters related to the permanent generation are set before Java8.
Since the -XMx2G memory was too small, leading to frequent Young GC, the memory was appropriately scaled up to achieve JVM throughput of around 95%. Of course, if memory size is a high priority, throughput can be sacrificed in exchange for a smaller memory footprint.
Parallel Old GC will cause Full GC when the Old GC is Full. To avoid Full GC, try to keep the memory of the Old GC at a low value. You can use the CMS GC, through its Majar GC mechanism, to keep the memory footprint of older generations at a low value.
- To optimize the parameters
-Xmx3g -Xms3g -XX:NewSize=2g -XX:+UseConcMarkSweepGC -XX:MetaspaceSize=256m
Copy the code- Memory footprint
The memory footprint is up slightly from the previous setting of -XMx2G, but still around 60% and very stable.
- The garbage collection
conclusion
1, need to clarify the purpose of tuning, such as throughput, memory footprint
2, the premise of tuning is monitoring, in the process of tuning, using Java own jstat command, K8S deployment of Prometheus. It also monitors memory, CPU, network traffic (estimated size of data received from Kafka), JVM garbage collection, and so on. Proficiency in monitoring tools is required.
3. You need to understand the default garbage collector of each Java version, its principle and characteristics, and understand the relevant parameters of the corresponding GC
4, need to have a certain understanding of business functions and code reading ability, if you do not know what the service is, it is impossible to start. If necessary, read the project source code for details of its implementation.