I. Description of the phenomenon

As the amount of service data expands rapidly, the search center is prone to alarms during peak hours. Until one day, users reported that searching for items was very slow. In view of this performance bottleneck, it was temporarily dealt with through rapid expansion, but it did not fundamentally solve the problem. In view of the problem of low search performance, a search performance investigation and optimization was carried out to solve the problem thoroughly and remove the performance constraint.

Two, problem investigation

Search for central hardware configuration information:

  • The ES data node configuration is as follows

  • Configuration: 32 vcpus, 128 GiB, 1TB, disk throughput IO 350MB/s

During the fault period, the number of requests increases rapidly, causing the DISK I/OS of ES to reach the bottleneck. CPU load is also high, which is an avalanche caused by peak service hours.

Then from the search traffic monitoring analysis, it is found that the long tail word request (keyword longer search request) more than in the past. According to the theoretical analysis, the long tail word request also has a certain degree of influence on the search performance.

Three, pressure test certificate

In order to verify the impact of long tail word request on performance, we decided to pressure the search system in the online production environment to evaluate the performance of the current search system.

The compression scheme and scripts are prepared by pressing directly against the DUBBO interface of the search center. The pressure test request is the download of the request body corresponding to the online environment at the time the problem will occur. During the evening business peak period, requests made during the day in the true line environment are used as pressure test samples. This also ensures the true validity of the pressure test request.

In the process of pressure measurement, the number of concurrent requests is gradually increased from 10 -> 20-> 30 for pressure measurement. In order to ensure the stability of the system, the pressure measurement time is controlled at about 10 minutes.

In the process of actual pressure measurement, it was found that:

  1. As the number of concurrent requests increases, the CPU of the ES machine is not high.

  2. However, the DISK I/OS of the machine where ES resides get higher and higher. At 20 concurrent requests, the threshold is close to.

As you continue to increase the number of concurrent requests, the queue in the thread pool increases and the number of requests piles up, resulting in higher CPU usage. The observed phenomena are consistent with the characteristics of the problems occurring on the line. Thus, the problem scenario is replicated and performance bottlenecks are identified.

After analysis and comparison, draw a conclusion. The results of this pressure test are consistent with the symptoms of an online problem: the performance bottleneck is in disk IO.

Then reduce the limit length of your search keywords. When searching for terms by keyword, the search term is broken down into small – grained terms (participles). The more word segmentation, the more times the ES document will be queried.

For example, the segmentation of “Xiaomi Intelligence” is “mi” and “intelligence”, and the ES document is queried twice with different segmentation, and finally the intersection is taken. As the length of the search statement increases, there will be more word segmentation, and the CPU will be under pressure when the intersection of the result set obtained by different word segmentation is taken, which will put pressure on the performance of the search system.

4. What more can be done about the situation

Considering the observed situation and the development of subsequent services and data volume, optimization is mainly carried out in the following three possible directions to improve system performance:

Hardware Resource optimization

The disks in the ES cluster are replaced with local SSDS and the configuration is changed.

  • The replaced ES data node:

  • Configuration: 16 Vcpus, 128 GiB, 2 * 1788 Gi, disk throughput IO 1 GB/s

Configuration changes:

  • On CPU: 32 cores -> 16 cores

  • On disk: 350 MB/s -> 1 GB/s

Why do you do that?

1. The data shows that the performance can be greatly improved after the replacement of low-cost SSDS.

Official disk performance data

Conversion based on the current configuration:

  • Before the replacement: Cloud disk: 750 MB/s, IOPS =51,800

  • Local SSD: read: 2GB/s, write: 1GB/s, IOPS= 300,000

IOPS (Input/Output Operations Per Second) is a measurement method used to test the performance of computer storage devices. It is the number of read/write Operations Per Second. Is one of the main indicators to measure disk performance.

2. Replace low-cost SSDS to reduce network consumption. Because the network consumption per read/write on a cloud disk is higher than that on the local site, switching to a local SSD can theoretically reduce the network consumption.

Additional configuration changes:

1. The CPU configuration is reduced from 32 to 16 cores. According to the pressure test results, CPU is not a performance bottleneck, and the search terms are not too long according to the normal habits of users. Moreover, the security department has made security interception to the search page, preventing the threat of web crawlers. The reduced cost on the CPU can be added to the disk configuration expansion.

2. Increased the disk throughput configuration to 1GB/s. Because the system bottleneck is in disk IO, disk upgrade is necessary. The following figure shows the reason for the high DISK I/O ratio. In the search phase, disk I/O operations are performed when the ES index is used to obtain the complete commodity information. The larger the commodity information in the index is, the larger the disk read size is. (This was optimized in step 2)

Data structure optimization

Elasticsearch relies heavily on file system caching to speed up searches. In general, you should make sure that at least half of the available memory goes into the file system cache so that Elasticsearch can keep indexed hot spots in physical memory.

In terms of data, a number of large node fields in ES were removed (the original business nodes occupied a large proportion). The original index size was 900+GB, but now the index size is 400+GB after the large node is removed. The removed data is stored in other ways and no longer occupies key indexes.

Here’s an example:

If the ES node of the company has 3 machines, each machine has 64G memory, the total memory of the ES cluster is 64 * 3 = 192G.

Each machine uses 32GB for ES’s own JVM, leaving 32GB per machine for file System cache. Therefore, the ES cluster gives file system cache 32 * 3=96 GB of memory.

At this time, if there is 1T index data on ES, then each machine needs to occupy about 300G+ of data. The file System cache on each machine has only 32GB of memory, which means that about 10% of the data can be stored in memory, and the rest is on disk. Then the search operation is performed, and most of the operation is read from the disk. The DISK IO is very high, so the performance is affected.

ES parameter optimization

Each shard in ES refresh every second, and each refresh generates a new Segment. Every search request must access every Segment, which means that the more segments there are, the slower the search request will be. ES has a background process that merges Segments into larger Segments.

Currently, the system has not verified whether the Segment size is the optimal threshold, so the impact on performance is verified by adjusting the Segment size during the period.

Initially adjust the Segment from 5G (default value)-> 10G -> 15G, use the same number of concurrent, pressure test samples for pressure test.

As can be seen from the results of several pressure tests above, the larger the Segment, the higher the TPS, and the better the performance.

Pressure test after optimization

Through the analysis of the search, the following optimizations were completed:

  1. Upgrade an ES cluster from an SSD cloud disk to a local SSD disk and tune the configuration

  2. Data structure optimization to remove large node fields in the index

  3. ES parameter Segment tuning

Then, the same scenario as the first time was used for pressure measurement. Then perform a comparison based on performance indicators and resource usage before and after the upgrade. It was found that the DISK IO of ES decreased significantly, and the CPU of ES increased significantly.

Main comparison results:

  1. IO: Because the amount of data is greatly reduced (about 60% this time), most scenarios go directly to OS-Page Cache, reducing the disk I/O usage by 99%.

  2. CPU: CPU usage increased from 20% to 40%;

  3. Memory: there is no basic difference, most of it is used by OS page cache;

Five, the summary

The performance tuning of the search business is ultimately the performance tuning of ES, which involves tuning everything. It includes not only the configuration of the server, but also the optimization of the data structure and the implementation of technology based on business requirements during the runtime. Its optimization depends on the deep theoretical understanding and application practice of ES. In the actual process, bias optimization should also be carried out according to the requirements of the system. Through the in-depth study of the system, continuous optimization, to ensure the stable operation of search business.

Recommended reading

Redis Bitmaps

Kubernetes Scheduler source code parsing and custom resource scheduling algorithm practice

First introduction to JVM (take you from a different perspective to understand JVM) mysql snapshot read principle implementation

, recruiting

Zhengcaiyun Technology team (Zero) is a passionate, creative and executive team based in picturesque Hangzhou. The team has more than 300 r&d partners, including “old” soldiers from Alibaba, Huawei and NetEase, as well as newcomers from Zhejiang University, University of Science and Technology of China, Hangzhou Electric And other universities. Team in the day-to-day business development, but also in cloud native, chain blocks, artificial intelligence, low code platform system, middleware, data, material, engineering platform, the performance experience, visualization technology areas such as exploration and practice, to promote and fell to the ground a series of internal technical products, continue to explore new frontiers of technology. In addition, the team is involved in community building, Currently, There are Google Flutter, SciKit-Learn, Apache Dubbo, Apache Rocketmq, Apache Pulsar, CNCF Dapr, Apache DolphinScheduler, and Alibaba Seata and many other contributors to the excellent open source community. If you want to change something that’s been bothering you, want to start bothering you. If you want to change, you’ve been told you need more ideas, but you don’t have a solution. If you want change, you have the power to make it happen, but you don’t need it. If you want to change what you want to accomplish, you need a team to support you, but you don’t have the position to lead people. If you want to change the original savvy is good, but there is always a layer of fuzzy window…… If you believe in the power of believing, believing that ordinary people can achieve extraordinary things, believing that you can meet a better version of yourself. If you want to be a part of the process of growing a technology team with deep business understanding, sound technology systems, technology value creation, and impact spillover as your business takes off, I think we should talk. Any time, waiting for you to write something and send it to [email protected]

Wechat official account

The article is published synchronously, the public number of political cloud technology team, welcome to pay attention to