Original: Taste of Little Sister (wechat official ID: XjjDog), welcome to share, please reserve the source.

As a very important member of NoSQL database, ES is used more and more widely. Although it has some defects in data timeliness due to index delay, its excellent design of large capacity and distributed makes it play a great role in real-time search field where timeliness is not particularly required.

According to application scenarios and applications, ES can be divided into two typical application modes: write and read. For ELKB, we need to pay extra attention to write optimization; For example, when synchronizing data from MySQL to ES wide tables, we need to pay extra attention to its read optimization.

Without further ado, let’s go straight to the optimization method. If you are not clear about some concepts of ES, please bookmark this article and read it slowly.

1. Write optimization

Logs are often written but rarely read, requiring high write speed. One of our clusters, for example, had 100 TERabytes of logs in a single cluster, with 10W logs written per second.

Data writing operations include Flush, refresh, and merge. By adjusting their behavior, you can make a trade-off between performance and data reliability.

1.1 Translog Asynchronization

First, ES needs to write a translog, which is similar to redolog in MySQL, to avoid data loss in the event of a power outage. ES defaults to flush once per request, but this is not necessary for logging, and you can make this process asynchronous with a flush interval of 60 seconds. The parameters are as follows:

curl-H"Content-Type: application/json"-XPUT'http://localhost:9200/_all/_settings? preserve_existing=true'-d'{ "index.translog.durability" : "async", "index.translog.flush_threshold_size" : "512mb", "index.translog.sync_interval" : "60s" }'Copy the code

This is arguably the most important optimization step and has the greatest impact on performance, but in extreme cases there is the possibility of losing some data. This is tolerable for logging systems.

1.2 Increasing the Refresh interval

In addition to writing translog, ES writes data to a buffer. But watch out! In this case, the contents of the buffer are not searchable and must be written to the segment, i.e. to the Lucence index. This is the refresh action, which defaults to 1 second. The data that you write, with a high probability of one second, will be searched.

This is why ES is not a real-time search system, it generally has a merging process from data writing to data reading, with a certain time difference.

The refresh interval can be modified with index.refresh_interval.

For logging systems, of course, it should be a bit larger. Xjjdog adjusts to 120s to reduce the frequency of these segments, thus reducing I/O pressure and writing speed.

curl-H"Content-Type: application/json"-XPUT'http://localhost:9200/_all/_settings? preserve_existing=true'-d'{ "index.refresh_interval" : "120s" }'Copy the code

1.3 the merge

Merge is actually lucene’s mechanism. It mainly merges small segments to generate larger segments to improve the speed of retrieval.

The reason is that the refresh process generates a large number of small segment files, and data deletion also generates space debris. So merge, in layman’s terms, is like a defragmentation process. There are vaccum processes that do the same thing with Postgresql and others.

Obviously, this sort of collation is a waste of BOTH I/O and CPU.

If your system merges frequently, it is a good idea to adjust the block size and frequency of merges.

2. Read optimization

2.1 Specifying a Route

If you write data into ES, it’s going to give you a discrete hidden ID, which shard it falls into, you don’t know. If you query data based on a query condition, it will take 6 queries if you set 6 shards. If we can know which fragment data is in when routing, the query speed will naturally increase, which requires us to manually specify routing rules when constructing data. Its actual operation rules are as follows:

shard = hash(routing) % number_of_primary_shards

Copy the code

For example, a query might look like this.

GET my-index-000001/_search
{
  "query": {
    "terms": {
      "_routing": [ "user1" ] 
    }
  }
}

Copy the code

Of course, if your query has a large number of dimensions and you have high requirements for data query speed, it is a good choice to store multiple copies of data according to routing.

2.2 Cold and hot separation of rollover

Rollover automatically transitions to a new index based on index size, number of documents, or lifetime. When rollover is triggered, a new index will be created, the write alias will be updated to point to the new index, and all subsequent updates will be written to the new index, such as IndexName-000001. This pattern.

As the name rollover suggests, it bears some resemblance to Java logs, such as Log4j’s RollingFileAppender.

When an index becomes very large, usually tens of GB, its query efficiency becomes very low and the cost of index reconstruction is also high. In fact, many of the indexed data have obvious patterns in the time dimension, and some cold data will be rarely used. At this point, a scrolling index would be a good idea.

Generally, rolling indexes can be combined with index templates to automatically create indexes according to certain conditions. Official ES documents have specific _rollover creation methods.

2.3 Replace TermQuery with BoolQuery

Bool queries now include four clauses, must, filter, should, and must_NOT. Bool queries are true and false comparisons, while TermQuery is an exact string comparison, so BoolQuery will naturally be faster than TermQuery if the requirements are similar.

2.4 Split the large query into component query

There are two obvious problems with some business queries that are complex and we have to splice a very large wide table into ES.

  1. The data of the wide table often need to be checked and assembled from other data sources, and the data update has great pressure on the source library or ES itself

  2. JSON query is complicated to write, the query efficiency is unknown, and the memory locked by a query is too high for further optimization

In fact, no matter in RDBMS or ES, wide tables are related to complex query statements, their lock time is long, and business is not flexible enough.

Strategies for dealing with this scenario typically shift complex data queries to the stitching of business code. For example, take a very long single query and break it into 100 smaller queries that loop through. All databases respond well to smaller query requests, and overall performance will be better than complex single queries.

This challenges our ES index modeling ability and coding ability. After all, at the ES level, unrelated indexes collectively provide a so-called data center interface to other services.

2.5 Increase the speed of the first index

The index data of many services comes from traditional databases such as MySQL. The first index is a full index followed by an incremental index. When necessary, index reconstruction will also be carried out. A large amount of data is poured into ES, resulting in the slow establishment of index speed.

To mitigate this, it is recommended that the number of copies be set to 1 when creating the index, i.e., there are no slave copies. Wait until all data is indexed, then increase the number of copies to normal levels.

This way, the data can be quickly indexed and copies copied slowly in the background.

3. General optimization

Of course, we can also make some general optimizations for ES. For example, if a thread pool is found to have significant bottlenecks using the monitor interface or trace tool, you need to resize the thread pool.

Specific optimization items are as follows.

3.1 Thread pool optimization

The new version optimizes the configuration of thread pools, eliminating the need to configure complex search, BULK, and INDEX thread pools. Size, thread_pool.write. Size, thread_pool.listener.size, thread_pool.analyze. Adjust the data exposed by the _CAT /thread_pool interface.

3.2 Physical separation of heat and cold

With the rollover interface above, we can implement index scrolling. But how do you store cold data on slower but cheaper nodes? How do I move some indexes over?

ES supports tagging nodes by adding attributes to the elasticSearch.yml file. Such as:

Node.attr. Temperature: hot // Node.attr. Temperature: coldCopy the code

After the node has hot and cold properties, the next step is to specify the hot and cold properties of the data to set and adjust the data distribution. ES provides index Shard Filtering for index migration.

First, you can set hot and cold properties on the index as well.

PUT hot_data_index/_settings
{
    "index.routing.allocation.require.temperature": "hot"
}

Copy the code

These indexes are automatically transferred to the cold unit. We can write some scheduled tasks to automate the transition process using data from the _cat interface.

3.2 Scattered I/O for Multiple Disks

In fact, you can configure multiple disks to spread I/O pressure, but data hotspots are concentrated on a single disk.

ES supports multiple disks on a single machine, so it scales more in terms of storage scale. You can mount multiple disks by configuring the path.data attribute in the configuration file.

path.data : /data1, /data2, /data3

Copy the code

It is worth noting that if you are expanding, you will need to reassign indexes in conjunction with reroute interfaces.

3.3 Reduce the size of a single record

Lucene’s index building process, which is very CPU intensive, can reduce the number of inverted indexes to reduce CPU consumption. The first optimization is to reduce the number of fields; The second optimization is to reduce the number of index fields. To do this, set the index property of the field that does not need to be searched to not_analyzed or no. As for _source and _all, they are not very useful in actual debugging and will not be described again.

End

ES is more and more widely used, from ELKB to APM, from NoSQL to search engine, ES is becoming more and more important in enterprises. This paper analyzes the optimization of ES write and read scenarios, and tries to help you speed up ES from principle to practice. Hopefully you’ll be more comfortable with ES.

In general, an ES cluster has high requirements on configuration, especially in scenarios such as APM, which even accounts for 1/3 or more resources of the PaaS platform. ES provides many configuration options. We can adjust the performance of ES according to application scenarios to better serve us.

Xjjdog is a public account that doesn’t allow programmers to get sidetracked. Focus on infrastructure and Linux. Ten years architecture, ten billion daily flow, and you discuss the world of high concurrency, give you a different taste. My personal wechat xjjdog0, welcome to add friends, further communication.

Recommended reading:

2. What does the album taste like

3. Bluetooth is like a dream 4. 5. Lost architect, leaving only a script 6. Bugs written by the architect are unusual

Taste of little sister

Not envy mandarin duck not envy fairy, a line of code half a day

317 original content