preface

  • Why are ES queries and aggregations so fast? How is the bottom layer implemented?
  • How is data stored in an ES cluster? How to achieve automatic distribution?
  • Why the master shard number of ES cannot be adjusted after it is set, but the replica shard number can be adjusted?
  • How to optimize index and query methods, effectively utilize cache and improve query efficiency?
  • What if smooth upgrade or expansion is guaranteed without continuous service?
  • How to optimize query efficiency?

After reading the Definitive Guide to Elasticsearch, all questions will be answered

I. Basic concepts

1. The shard

  • The smallest unit of work that holds a portion of the data in the index. Lucene is an instance of Lucene and is a complete search engine in its own right. But the application does not communicate directly with the shard.
  • You can think of it as a container, and the data migration between nodes is in shards
  • Master shard and deputy shard (a copy of the master shard)
  • When the index is created, the number of master shards is fixed, but the number of replica shards can be adjusted
  • By default, an index allocates 5 primary shards
  • If the node where the primary shard resides fails, the primary node is elected again and the secondary shard is upgraded to the primary shard
  • After the faulty node is restarted, data that has not been synchronized during the fault is synchronized

2. Document

  • The root object is serialized into a JSON object
  • _version is incremented by one every time a document is modified or deleted
  • Documents are not modifiable. Update is delete and then create a new one
  • Deleted documents are not immediately removed, only marked as deleted. And then we clean it backstage
  • Set your own version of the document: add the version_type=external parameter

3. Conflict resolution

  • Resolve conflicts by implementing optimistic locking with version numbers

4. Document metadata

  • _index Where the document is stored
  • The class of the object represented by the _type document (the 7.x version will remove the _type)
  • _ID Unique identifier of a document. Can be set manually or automatically generated (22 bits long)

5. Cluster architecture diagram

Renderings of two nodes, three primary shards and one sub-shard

Expand to three nodes into renderings

6. Cluster status

Cluster state is a data structure that exists in each client. Save the following information

  • Level set
  • Cluster nodes
  • Index and associated mapping, alias information, etc
  • The sharding of the index, and the assigned node

Cluster status -status

  • Green: All primary and secondary shards have been allocated
  • Yellow: All primary shards are allocated, but at least one replica shard is not allocated
  • Red: At least one primary shard (or all sub-shards) is missing

Working principle of cluster

1. How is data stored in distributed systems

  • Documents are routed to shards
  • The following algorithm is used to obtain the fragment number of the document 
    shard = hash(routing) % number_of_primary_shards
Copy the code
  • Routing is an arbitrary string. The default value is _id
  • The number of master shards cannot be changed, otherwise the previous route will fail and the document will not be found
  • Custom routes ensure that related documents are stored in the same shard

2. How do master and replicated shards interact?

  • Requests can be sent to any node
  • Each node is capable of handling arbitrary requests
  • Each node knows the node where any document is located (the saved cluster state) and forwards the request
  • It is best to loop each node for load balancing when sending requests

2.1 Write Operations (Creating, Deleting, and Indexing)

Sequential steps

  • The client sends requests (create, delete, index) to node1
  • The node uses the hash algorithm to get the shard number 0, because shard 0 is at node 3, and forwards the request to node 3
  • Node3 successfully saves data to the primary shard and, if successful, forwards requests to Node1 and Node2 to secondary nodes
  • If all replication nodes are successful, a successful reply is sent to requesting node 1, which then returns the reply to the client
Adjustable parameters
  • Replication: The default value is sync. The primary shard is returned only after receiving a successful response from the replication shard. Async indicates that the request is returned after successful execution in the primary shard and still forwards the request to the sub-shard without knowing whether it was successful or not
  • Consistency: When a master shard attempts to write, a quorum or half of the shards must be available. It can be one, all, or quorum (default). Quorum takes effect only when number_of_replicas is greater than 1 
    Int ((primary[always 1] + number_of_replicas) /2 + 1)Copy the code
  • Timeout: waiting time when fragments are insufficient. The default 1 min

2.2 read operation

  • The client sends a GET request to Node1
  • The node uses the hash algorithm to obtain that the main fragment to which the document belongs is 0
  • Node1, node2, node3
  • A sub-shard node is selected by some policy, such as node2
  • Node2 returns the document to node1
  • Node1 is then returned to the client

2.3 the update operation

Sequential steps

  • The client sends an update request to node1
  • The main fragment location is obtained by hashing algorithm, and the request is forwarded to Node3
  • Node3 retrieves the document, modifies the _source field to the JSON document, and then rebuilds the index. If another process modifies the document, it repeats this step as many times as set by retry_on_conflict and fails
  • If node3 is successfully updated, the entire new document (not the modification request) is sent to the replicated nodes of node1 and Node2 to rebuild the index. If both are successful, it is returned to Node1 and then back to the client

2.4 Multi-document mode

Mget multiple document read

  • The client sends an MGET request to Node1
  • Node1 builds a request for each shard and forwards it to the node where the shard resides
  • When all replies are received, Node1 combines the responses and returns them to the client
Bulk Multi-document write

  • The client sends a request to Node1
  • Node1 builds bulk requests for each shard and forwards them to those master shards
  • The master shard executes sequentially, and when each is completed, it is sent to the sub-shard
  • When all operations are complete, the collation response is returned to the client

3. How is the index established

3.1 Basic Concepts

  • Mapping: used for field confirmation. Each field matches the confirmed data type
  • Analysis: Word segmentation of the whole text to establish an inverted index
  • Inverted index: consists of a unique list of words in a document and their location within the document, designed for quick retrieval of results

3.2 Analysis

Process of analysis
  • Analysis is done by an analyzer
  • The analysis process begins by marking a paragraph of text as a single word (item).
  • Then standardize (for example, all lowercase) items to improve searchability
  • The details of the analysis can be viewed through the _analyze API
The components included in the analyzer

Es provides a number of ready-to-use components that can be used in custom combinations

  • Character filter: The string goes through this first for some filtering operations
  • Tokenizer: Splits text into words, such as Spaces, commas, etc. Chinese can be used with a special participle
  • Token filter: Modify words, such as lowercase, remove mood words, add synonyms
Built-in profiler
  • Standard profiler: This is used by default. Standard shard, removing most of the symbols and finally converting to lowercase
  • Whitespace analyzer: Split by whitespace, not converted to lowercase
  • Language analyzer: Analyzes the characteristics of a particular language
A query
  • Field query: accurate match, query will not be queried before the string analysis
  • Full-text query: The string to be queried is parsed with an analyzer before the query
Manually specifying the analyzer
  • When a string is added to ES, es will automatically use the standard parser to do word segmentation, but some characters may be common id, tag and other fields, do not need to do analysis, can be manually specified mapping
The order in which parsers are looked up when indexes are created
  • The analyzer for the specified field in the Mapping file
  • The _Analyzer field of the document itself
  • Default Analyzer for the type specified in the Mapping file
  • The global default Analyzer in the Mapping file
  • Node level default Analyzer
  • Standard analyzer
The order in which parsers are looked up when indexes are looked up
  • Query the analyzer in parameters
  • The analyzer for the specified field in the Mapping file
  • Analyzer of the specified type in the Mapping file
  • The global default Analyzer in the Mapping file
  • Node level default Analyzer
  • standard analyzer

3.3 mapping

role

Defines the field type, the data type of the field, and how it is handled by ES.

Supported field types
type Represents the data type
String string
Whole Number byte short integer long
Floating point float double
Boolean boolean
Date date

Es will automatically guess the field type if the new field is not mapped

What can be done with custom field mappings
  • Distinguish between full-text strings (word segmentation required) and exact strings (word segmentation not required)
  • Use language-specific profilers
  • Optimize partial matching fields
  • Specify a custom date format
Parameters included in the mapping
  • Properties: Lists mappings for each field that may be included
  • Metadata fields: _type, _id, _source
  • Dynamic: Determines the policy for adding fields (_source will always be saved)
    • Ture autoadd
    • False Ignore field
    • Strict throws an exception
  • Set items: for example, Analyzer
  • Other Settings
Customize the field mapping
  • The field parameter to map is type, and you rarely need to map any type other than string
  • The index field of the string field controls how the string is indexed.
    value meaning
    analyzed Word segmentation index
    not_analyzed Indiscriminate index
    no Don’t index
  • The string field selects Anlyzed as index when analyzer specifies the analyzer. For example: simple, English, whitespace
  • Updating a mapping can only add fields, but cannot modify fields that have already been added. Otherwise, an error may occur
Properties of document fields
  • type
  • index
  • analyzer
  • ip
  • geo_point
  • geo_shape
Metadata _source field
  • Purpose: Save the original JSON field
  • Why is it needed
    • Search results are fully documented
    • Without it, some update requests do not work
    • When updating a mapping file, you can directly obtain the content
    • Easier to detect errors
  • How to disable: enabled: false
  • Use: You can specify which columns to return with _source when searching
Metadata _all field
  • Use _all when the query does not know which field to specify. It can also be disabled. When _all is used, the values of all other fields are indexed as one large string
Dynamic template

The dynamic_templates setting dynamically matches different mappings by field name or type

  • The data type used by the match_mapping_type template
  • Name of the field used by the match template
  • Full path of fields used by path template (nested JSON)

Structured Query Language

1. The filter

An overview of the

Whether the fields of the document contain specific values, faster than queries, and cacheable results

In principle, full-text indexing or other relevance scoring should be used for query statements, otherwise filtering should be used.

Important filter statements

  • Term: exact match
  • Terms: Exact match of multiple conditions
  • Range: indicates range filtering
  • Exists: Indicates whether the field is contained
  • Missing: There is no field
  • Bool: Merge multiple query results
    • Must: and
    • Must_not: not
    • Shoud: the or

Filter order

  • Filtering order has a significant impact on performance
  • More detailed filtering criteria should come first to exclude more documents
  • Cached filters should precede those that are not cached (see caching section below).

2. The query

The paper

How well do the fields of each document match specific fields, are slower than filtering, and the results are not cacheable

Critical query statements

  • Math_all: Query all documents
  • Match: Standard query with full text and precision support

    When match specifies multiple values, the inner part of the word will be executed after multiple matches and bool queries. The default is OR. You can change the operator parameter to “and”.

  • Multi_match: Searches for multiple fields at the same time. Wildcard characters are supported
  • Bool: Same as bool filter, but more to calculate _score

3. Relevance sorting

The sorting way

  • _score: default sorting mode, default reverse order

  • Field sorting: _score does not need to be calculated, default positive order

  • Multi-level sort: Multiple fields can be specified. Sort by the first field, and sort by the second field

  • String parameter sort:

    Forced sorting of parsed fields consumes a lot of memory

Introduction to relevance

Similarity algorithm: TF/IDF (retrieval word frequency/reverse document frequency)
  • TF: Word frequency, the more times it appears in the current document, the greater the correlation
  • IDF: Inverts the frequency of documents, the number of documents as a percentage of the number of files in which the word appears. The greater the frequency of the word, the smaller the IDF
  • Due to performance issues, each shard only calculates the IDF within that shard, not all documents
  • The Boost parameter can be weighted

4. Implementation of distributed search

An overview of the

Search consists of querying multiple shards, merging the shard meta-information, and then retrieving the real data based on the meta-data.

Querying multiple indexes is exactly the same as querying one index, except that more shards are queried. When extending, you can create a new index and query both indexes, or alias indexes, instead of migrating old data to the new index

Query

  • The client sends search to node3 to create an empty priority queue from+size

  • The request is broadcast to each shard, and each shard performs the query locally and places it in a local priority queue with the size from+size

  • Each node returns the query results (ID and _score) to Node3, which sorts the results globally

    • Multiple requests poll all shard copies to balance load and improve system throughput
    • Multiple indexes work like single indexes, but with more sharding
    • Deep paging can cause the sorting process to be onerous, taking up a lot of CPU and broadband

Fetch

  • The coordination node identifies which documents need to be fetched and sends requests to the corresponding shard
  • Each shard loads the document, does relevant processing (such as highlighting), and returns it to the coordination node
  • The coordinating node returns the data to the client

Search options (optional)

  • Preference: Controls which shard or node is used to process the request
  • Timeout: Coordinates the result of how long a node waits before abandoning another node
  • Routing: Limiting which shards to search for is useful for large scale systems
  • Search_type: query_then_FETCH is the default search type
    • Count: When no result is needed, only quantity is needed
    • Query_and_fetch: Query and fetch
    • Dfs_query_and_fetch, dfs_query_then_fetch
    • Scan: used together with Scroll. It can retrieve large amounts of data efficiently. Disable sort implementation

Scan and scroll

scroll

Cursors, similar to traditional databases, search for index snapshots at the time of the query, and changes before the end of the query are not perceived

scan

No sorting, return as long as there is a result

The internal principle of sharding

1. Dynamic index update principle

1.1 Inverted indexing – Ensures that documents are searchable

1.2 The contents of an inverted index are immutable

1.3 Immutable while dynamically adding segments

When you query, all segments are queried in sequence and the results are aggregated so that new documents are added to the document with minimal cost

  • New documents are first written to the index cache of the memory area
  • The buffer contains the inverted index, segment name, and so on that the new segment contains
  • Buffer is submitted
  • A new segment is opened and the document can be indexed
  • The memory cache is cleared and waiting for new documents

1.4 Delete and Update

Because segments are immutable, the update and delete operations do not actually delete. Del files are added and segment files are created, and files marked as del are removed from the results before the query returns

1.5 Near-real-time search

Because brushing from buffer to disk is expensive. Es allows once a file is cached, segments can be set to open and files can be searched

1.6 the refresh

Each shard opens a new segment every second by default, so new changes take 1s to see. Refresh_interval can be set to reduce the frequency of refreshing

1.7 Persisting Changes

With the addition of the buffer, the transaction log (512 MB by default) is added to ensure that the data is fully persisted. Each time flush (every 30 minutes, or the transaction log is too large) is flushed to disk, segments are fully committed, clearing the transaction log

Combined section 1.8

By automatically refreshing segments every second, it doesn’t take long for segments to skyrocket. Each segment consumes computer resources, and each segment must be checked in turn for each query. The more segments, the slower the query. The es background merge segment solves this problem. Merging large segments consumes IO and CPU resources.

1.9 Optimize the API

Force merge segments. This works well for indexes where the data is not changing. Do not use indexes where the data is still growing dynamically.

2. The cache

An overview of the

  • Caching is for filtering queries
  • The core is a set of bytes that holds which documents qualify for filtering
  • The cached byte set is updated incrementally
  • Each filter is cached independently and reusable
  • Most branch and leaf filters (such as TERM) are cached, while composite filters (such as bool) are not

The condition that cannot be cached

  • Script filters, scripts are opaque to ES
  • Geo filters are less likely to be reused
  • Date ranges accurate to milliseconds are not cached, integers are cached

Suggestions for filtering time ranges

  • For the query whose time is accurate to millisecond, the filter criteria can be split into date and date and time. The former filter criteria will be cached and the order will not change

5. Full text retrieval

1. Full-text retrieval includes two aspects

  • Relevance: TF/IDF, geographic proximity, fuzzy similarity, or other algorithms
  • Analysis: word segmentation, create inverted index

2. Full-text query classification

  • Low-level query: term query. There is no analysis phase, and specific phrases are precisely matched
  • Full text search: match, query_string and other queries. There is an analysis phase.
    • Date, INTEGER Specifies the exact query type
    • String type of NOT_analyzed: Analyzes query terms (such as turning to lowercase) and performs a single phrase query
    • Analyzed string type: queries are parsed to generate a list of phrases. Query and merge the query results

6. The aggregation

1. Basic concepts

Barrels (buckets)

A collection of documents that meet certain criteria. Similar to group by in SQL

Metrics

Perform statistical calculations on the documents in the bucket. Similar to SQL in the count, sum, Max and other statistical methods

2. Approximate polymerization

2.1 an overview of the

  • Distributed algorithm three factor model can only choose to meet two items at the same time: accurate, real-time, big data
  • Ea chooses big data and real time. Provides accurate, but not 100% accurate, results at the expense of small estimation errors in exchange for efficient execution and minimal memory consumption
  • Two approximation algorithms: Cardinality and Percentiles

2.2 Cardinality measurement

  • SQL – like distinct
  • It’s an approximation algorithm. Based on HyperLogLot++ (HLL) algorithm. HLL first hashes the input, and then estimates the cardinality according to the bits in the hash operation. HLL paper
  • Algorithm features
    • Configurable precision: parameter precision_threshold (higher precision = more memory)
    • Precision_threshold The precision_threshold ranges from 0 to 4000, and the memory used for data structures is precision_threshold * 8
    • When the value is set to 100, the data volume error of the million level can be maintained within 5%
    • Small data sets are very accurate
    • Fixed amount of memory that can be configured for use
  • Optimization: Hash values are computed in advance, but performance bottlenecks shift from aggregation to indexing (indexes must be rebuilt and hash fields added), depending on the business scenario.

2.3 Percentiles Measurement

  • Represents observed values performed at a specific percentage, usually used to find anomalies.
  • It’s also an approximation algorithm. Use the TDigest algorithm
  • Algorithm features
    • In extreme percentages, the numbers are more accurate. Like 1% or 99%. This is determined by the data structure.
    • Small data sets are very accurate

3. significant_terms

  • Sigterms is different from other aggregations and is used to find indicators of medical abnormalities in data sets
  • Use statistical data and compare normal data to find indicators that may have abnormal frequency

4. Aggregated data structures

4.1 Doc Values

  • Aggregate, sort using Doc Values data structure

  • Map documents to the terms they contain

  • Generated at the same time as the index and inverted index. Segment-based and immutable.

  • Doc values data is stored on disk, not managed by the JVM.

  • Use column storage, data neatly arranged, easy to compress

  • The analyzed field is not supported

  • Doc values are turned on by default for all fields except the analyzed string. Disable doc values if you will never aggregate and sort certain fields. Saves disk space and index speed

4.2 Fielddata

  • Anaylzed strings that support aggregation using a data structure called Fielddata, which is stored in the memory heap. Older versions used Fielddata without Doc values
  • The anaylzed process consumes a lot of memory and generates a lot of tokens, which is not aggregation friendly
  • Fieldata remains in memory until it is expelled or the node crashes. Notice how big it is.
  • Dielddata does not exist when the index is being built; it is created at query time
  • Indices. Fielddata. Cache. Size: percentage or actual size. Controls how much heap space is allocated to FieldData. The analysis fields are loaded into Fielddata each time the query is aggregated, and if the Fielddata size in the query results exceeds the specified size, the other values are reclaimed to make space.
  • If there is not enough space to keep FieldData in memory, Elasticsearch reloads data from disk and reclaims other data to get more space. The memory reclamation mechanism causes heavy disk I/O and creates a lot of garbage in memory that must be reclaimed at a later date.
  • Monitor fildData: GET /_stats/ fieldData? fields=*

5. Aggregation optimization

  • Nested aggregated queries for large amounts of data are extremely inefficient. The default query mode is depth-first.
  • Breadth-first can be used to handle cases where the number of aggregations is much smaller than the total number of groups. The parameter is collect_mode: breadth_first

7. Geographical location

1. Set the field type to Location

Geocoordinate points cannot be detected by dynamic mapping fields. The corresponding field type (type parameter) must be explicitly declared as geo_point

2. Geo_point format

  • The value is “40.715, -74.011”. The dimension is in the first place and the precision is in the second place
  • Array: [40.715, -74.011], dimension in front, precision in back
  • Object: {“lat”: 40.715, “lon”: -74.011}

3. Filtering method

  • Geo_bounding_box :: Coordinates that fall on the specified rectangular box
  • Geo_distance :: Point within a given distance
  • Geo_distance_range :: Points in the distance range
  • Geo_polygon :: Point that falls on a polygon

4. Use caution

  • Geolocation filters are expensive to use because they load the geolocation information of all documents into memory and then compute. Use with caution or at the end of the filter
  • By default, the BOOL filter will filter the geographic information last
  • It is not cached by default
  • Each latitude and longitude combination needs 16 of its own memory, can be set to compress the format, reduce precision, reduce memory
  • Reasonable precision Settings: geohash_prefix and geohash_precision parameters. This is combined with a GeoHash filter for efficient queries

5. geohash

  • Divide the world into 4*8=32 cells, each cell identified by a letter or number. These units were then broken down into 32 smaller units, repeated over and over again
  • The longer the length, the higher the accuracy
  • Geohashes with the same prefix are closer together, and the longer the common prefix, the closer they are
  • It is also possible that positions that happen to be right next to each other have completely different Geohashes

6. Geographic aggregation

  • Geo_distance aggregation: Groups documents in a circle centered around a specified center store
  • Geohash_grid grid aggregation: Group documents by geohash units for rendering on a map
  • Geo_bounds: A rectangular box containing coordinate points

7. Geo_shape

  • Geographic shapes are drawn by geoHash units

Data modeling

1. Association

Flat storage is used to process association relationships and data is redundant to the same index, improving query efficiency

2. Nested objects

design

Internal storage

When a JSON file contains an array, the internal storage will be flattened, causing the logical relationship to be lost. The nested relationship should be changed instead of the default object. Nested objects are internally indexed as separate hidden documents

The query

Use special NESTED queries or nested filtering

The sorting

3. Father-son relationship

The principle of

  • Nested is similar to nested, except that nested is stored in the same document, whereas parent-child relationships are completely different documents
  • Parent and child documents need to be stored in the same shard
  • Parent-child mappings are stored in the doc-values data structure and are entirely in memory
  • Suitable for the case of fewer parent documents and more child documents

advantage

  • When the parent document is updated, the child document index is not updated
  • Creating, deleting, and modifying child documents does not affect the parent document and other documents

disadvantage

  • Queries are 5-10 times slower than nested types
  • Not suitable for multiple parent documents

Design parent-child relationship

  • Specifies that one document type is the parent of another document type
  • When a parent document is created, it is no different from a regular document
  • When creating child documents, you must specify the parent document ID with parent. Create associations and ensure that they are assigned to the same shard (hash using the parent document ID)
  • Use parent-child relationships sparingly, only when there are fewer parent documents

4. Capacity expansion design

Expansion ideas

  • Start by looking to see if there are inefficient queries that can be optimized
  • Whether there is insufficient memory
  • Whether swap is enabled
  • If an index has been created, the number of fragments cannot be changed. You can migrate the old data to the new index by re-indexing the index
  • The search performance depends on the response time of the slowest node
  • Because a single index is the same as multiple indexes, you can configure multiple indexes to expand capacity

Shard Number setting

  • Estimate the total amount of data based on existing data volume and periodic growth
  • Based on the existing hardware information, set a single shard with 0 copies to find the maximum number of documents that a single shard can support under the current hardware conditions
  • The total number of shards/the maximum number of individual shards can be roughly estimated

Optimization of time-based data flow scenarios

  • The index is segmented by time
  • Old data will not be changed, using the Optimize API for segment merging.
  • Most indexes will have about 50-150 segments, even if they hold billions of terabytes of documents. An excessive number of segments indicates a problem with the merge (for example, merging not fast enough to create segments)
  • However, segment merges consume all the I/O resources on your nodes, potentially making the cluster unresponsive. If you want to execute on an indexoptimizeYou need to move the index to a secure node before executing it.
  • In order not to affect the normal index, period of merger background limit disk read and write speed is 20 MB/s, and can be adjusted according to actual condition, such as SSD disk, parameters for indices. The store. The throttle. Max_bytes_per_sec. Even if there is no query, set it to None, i.e., no limit, and change back after merging.
  • Also, optimizing an index that is still writing data can be a bad idea because it consumes a lot of I/O on nodes and hits existing indexes
  • We can temporarily remove the replica shard, optimize it, and then restore the replica shard
  • Before removing the copy, the data can be backed up through the Snapshot Restore API
  • For older data that will not be used, close the index. After being shut down, no other resources are occupied except disks. Flush ->close
  • Data archiving: Snapshot Restore API stores data to HDFS or elsewhere

User-based data flow scenarios

  • Specify route: Ensure that the same type of data is distributed to the same fragment. Route parameters are also passed in the query to ensure that only specific fragments are queried, resulting in performance loss caused by multi-fragment query
  • Use an alias to specify a specific name for a specific route value and filter. To achieve the effect of multiple names sharing an index. It looks like multiple indexes.
  • When the amount of data in a fragment increases to the point that an index needs to be created separately, the _alias operation is used to specify the remove and add parameters of the action to implement smooth migration.

Ix. Management and monitoring

1. Set important parameters

  • cluster.name
  • node.name
  • path.data
  • path.logs
  • path.plugins
  • Discovery.zen.minum_master_nodes: Minimum number of primary nodes to prevent brain splitting (multiple primary nodes)
  • Discover. Zen. Ping. Unicast. Hosts: cluster unicast list
  • Gateway. recover_after_nodes Specifies the minimum number of nodes to make the cluster available
  • Gateway. expected_node How many nodes are expected in the cluster
  • Gateway. recover_FATer_time How long to wait for data recovery
  • Logger. discovery Log level
  • Index. Search. Slowlog. Threshold. Query. Warn: “10 s” the output of the query is slow and 10 s warn log
  • Index. Search. Slowlog. Threshold. The fetch. The debug: “500 ms” slow query and the output of 500 ms debug log
  • Index. The indexing. Slowlog. Threshold. The index. The info: “5 s slow query log output info with 5 s
  • Index. unassign.node_left. delayed_timeout Changes the fragment time delay
  • Cluster. Routing. Allocation. The enable “:” none “fragmentation distribution is prohibited

2. Do not modify the configuration

  • Do not change the garbage collector, default to CMS. Don’t switch to a newer G1
  • Number of threads, default is CPU cores. IO operations are performed by Lucene threads, not es.

3. Heap memory configuration

  • The default value is 1 GB. The actual production environment must be changed
  • Ensuring that Xms, like Xmx, prevents the heap size from changing at runtime, which can be very resource-intensive
  • Do not fragment more than half of native memory. Lucene itself requires memory and cache.
  • You can reduce heap memory if you do not need to aggregate the segmentation. The smaller the heap memory, the better the performance of Elasticsearch (faster GC) and Lucene (more memory for caching).
  • Do not have more than 32GB of memory. As each object’s pointer gets longer, you use more CPU memory bandwidth, which means you actually lose more memory. If you want to keep it safe and reliable, setting the heap to 31 GB is a safe choice
  • If the memory is large, you can consider allocating multiple ES instances to a machine, but the total heap memory should not exceed half. At the same time cluster configuration. The routing. Allocation. Same_shard. Host: true. Prevents the same shard (master and slave) from being on the same machine
  • Set bootstrap.mlockall: true to lock memory and not take place memory Considerations

4. Operation, maintenance and optimization

  • Log files are stored in the logs file in the installation directory by default. “Logger. discovery” : “DEBUG” can set the log level
  • You can set the output of slow query logs
  • If you don’t want real-time accuracy, change the index.refresh_interval to 30s, set it to -1 for bulk pours, and set it back when the pour is complete
  • Number_of_replicas is set to 0 for large volume replicas, closing the replicas to improve efficiency
  • Try to use ids automatically generated by ES to avoid inefficient version lookups. If you use your own ID, use one with good compression performance and avoid using too random ids
  • Delayed sharding: Prevents massive data migration problems caused by nodes dropping and then restarting. Because the data on the downed node may be completely deleted due to the failure and then copied again. Parameters for the index. The unassigned. Node_left delayed_timeout

5. Roll the restart

  • Upgrade or maintain each node one by one without stopping the clustering function
  • Stop indexing new data first
  • Fragment allocation is prohibited. cluster.routing.allocation.enable” : “none”
  • Shut down a node and perform upgrade maintenance
  • Start the node and wait to join the cluster
  • Restart fragment allocation. cluster.routing.allocation.enable” : “all”
  • Repeat the previous steps for the other nodes
  • Restore index update data