What is the Elasticsearch
Elasticsearch is an open source distributed document storage and search engine that can store and retrieve data structures in near real time and relies heavily on Apache Lucence, a full-text search engine written in Java.
Elasticsearch presents data as structured JSON documents and provides full-text search for languages such as PHP, Python, and Ruby through RESTful apis and Web clients. The Elasticsearch service is elastic because it is easy to scale horizontally — just add more nodes to distribute the load. It is now used by many businesses to dynamically store, search and analyze large amounts of data, including Wikipedia, eBay, Github and Datadog.
Way to work
Before getting into performance metrics, let’s take a look at how Elasticsearch works. In Elasticsearch, a cluster consists of one or more nodes, as follows
Each node is a single instance of Elasticsearch whose ElasticSearch.yml profile specifies which cluster (cluster.name) it belongs to and which type of node it is. Any properties set in the configuration file, including the cluster name, can also be specified by command line arguments. The cluster in the figure above consists of a dedicated master node and five data nodes.
The three most common types of Elasticsearch nodes are:
- Eligible primary node: By default, each node can serve as the primary node. Each cluster automatically selects a master node from among all the master nodes. If the current primary node fails (such as power failure, hardware failure, or insufficient memory), a primary node is selected from the eligible nodes. The master node coordinates cluster tasks, such as distributing shards across nodes, and creating and dropping indexes. Eligible primary nodes can also act as data nodes. However, in larger clusters, it is common to start a dedicated primary node that does not store any data (as set in the configuration file)
node.dataforfalse) to improve reliability. In high-utilization environments, separating the master node from the data node helps ensure that there are always enough resources allocated for tasks that only the master node can handle. - Data nodes: By default, each node is a data node that stores data in a sharded form and performs indexing, searching, and aggregating operations. In larger clusters, you can set this in the configuration file
node.master: falseTo create dedicated data nodes to ensure that these nodes have sufficient resources to handle data-related requests without the additional workload of cluster-related administrative tasks. - Client node: If will
node.masterandnode.dataSet tofalse, the node becomes a customer node and acts as a load balancer to help route indexing and search requests. The client node can take on a portion of the search effort so that the data node and the master node can focus on the core task. Depending on usage, the client node may not be necessary because the data node can handle the request routing itself. However, if the search | index of workload is enough big, can use to help the client node road has a request.
Storage of data
In Elasticsearch, related data is usually stored in the same index, which can be considered the equivalent of configuration logic wrapping. Each index contains a set of related documents in JSON format. Elasticsearch’s full text index uses Lucence’s inverted index. When indexing a document, Elasticsearch automatically creates an inverted index for each field. An inverted index maps fields to the document that contains them.
Indexes are stored in master shards (one or more) and replica shards (zero or more), each of which is a complete instance of Lucence and can be used as a mini-search engine.
When creating an index, you can specify the number of master shards and the number of copies per master shard. The default value is five master shards per index and one copy per master shard. The number of primary shards cannot be changed after an index is created, so be careful when choosing the number or you may need to re-index later. The number of replicas can be updated later as required. To prevent data loss, the master node ensures that each replica shard is not assigned to the same node as the master shard.
Key indicators
Elasticsearch provides a number of metrics to help you detect signs of failure and take action when you encounter problems such as unreliable nodes, low memory, and long GC times. Some key indicators to monitor are:
- Search and index performance
- Memory and garbage collection
- Host and Network
- Cluster health and node availability
- Resource saturation and errors
The metrics listed above are available through Elasticsearch’s API and tools like Elastic’s Marvel.
Searching performance Indicators
Search requests and index requests are the two main request types in Elasticsearch. In some ways, these requests are similar to read and write requests in a traditional database system. Elasticsearch provides metrics that correspond to the two main stages of the search process (query and extract). The path from start to end of a search request is as follows
- The client sends a request to node 2
- Node 2 (coordination node) sends the query to a copy of each shard in the index (master or shard copy)
- Each shard performs the query locally and passes the results to node 2. Node 2 sorts and compiles these results into a global priority queue.
- Node 2 finds the document to extract and issues multiple GET requests to the relevant shard
- Each shard loads the document and returns it to node 2
- Node 2 returns the result to the client
When Elasticsearch is primarily used for search, it is necessary to monitor query latency and take action when the threshold is exceeded. It is important to monitor relevant metrics for queries and extracts to help determine search performance over time. For example, peak and long-term growth trends in query requests can be tracked in preparation for optimized configurations for better performance and reliability.
| Indicator description | Indicators of | The index type |
|---|---|---|
| The total number of query | indices.search.query_total | throughput |
| Query the total | indices.search.query_time_in_millis | performance |
| Number of queries currently being processed | indices.search.query_current | throughput |
| The total number of the fetch | indices.search.fetch_total | throughput |
| Fetch the total | indices.search.fetch_time_in_millis | performance |
| Number of fetches currently being processed | indices.search_fetch_current | throughput |
The key indicators
Query load: Monitoring the number of queries currently in progress gives you an idea of the number of requests being processed by the cluster at any given moment. Consider issuing warnings for unusual spikes and dips.
Query latency: Although Elastisearch does not explicitly provide this metric, it is possible to use existing metrics to estimate this value by periodically dividing the total number of queries by the total elapsed time.
Fetch delay: The second phase of the search (Fetch phase) is usually less time-consuming than the Query phase. If this value keeps increasing, it could mean that the disk is slow or that too many results are being requested.
Index performance Indicators
Index requests are similar to write requests in traditional database systems. If the Elasticsearch cluster is primarily used for indexes, monitoring index performance is essential. Before we talk about monitoring metrics, let’s look at how Elasticsearch handles indexes. When new information is added to the index or existing information is deleted, each shard in the index is updated in two steps: refresh and Flush.
refresh
Documents newly added to the index are not immediately available for search; they are written to the memory buffer and wait for the next index refresh, which by default happens every second. Refresh creates a new memory segment from the previous memory buffer so that the contents can be searched, and then clears the cache as follows
Period of
The sharding of an index consists of multiple segments. The core data structure of Lucene is a change set of segments that are actually indexed. Each segment uses files, memory, and CPU. To make efficient use of these resources, these segments are created on each refresh and then merged.
Segments are miniature inverted indexes that map words to the documents that contain them. Each time an index is searched, either the master shard or the replica shard is searched, and each fragment is searched within the shard.
Segments are immutable, so updating the document will:
- The information is written to a new segment during refresh
- Mark old information for deletion
When an expired segment is merged with another segment, the old information is eventually deleted.
flush
As new index documents are added to the memory buffer, they are also added to the shard translog, which is used to log persistent operations. Every 30 minutes or when the Translog size reaches its maximum (512MB by default), a Flush operation is triggered. During Flush, any documents in the memory buffer are flushed (stored in a new segment), all segments in memory are committed to disk, and the Translog is emptied.
Translog helps prevent data loss when a node fails. You can use Translog to help recover operations that might have been lost between flusher. Logs are submitted to disk every 5 seconds; Or after a successful index, delete, update, or batch request, the log is committed to disk. The process is as follows
Elasticsearch provides a number of metrics for indexing
| Indicator description | Indicators of | The index type |
|---|---|---|
| The total number of documents indexed | indices.indexing.index_total | throughput |
| Total time spent indexing documents | indices.indexing.index_time_in_millis | performance |
| The number of documents currently indexed | indices.indexing.index_current | throughput |
| Total number of index refresh | indices.refresh.total | throughput |
| Refresh the total | indices.refresh.total_time_in_millis | throughput |
| Index flush to total number of disks | indices.flush.total | throughput |
| Flush the total | indices.flush.total_time_in_millis | throughput |
The key indicators
Index latency: Elasticsearch does not provide this metric directly, but it can be calculated using index_total and index_TIME_in_millis. If note latency increases, it may be because too many documents are indexed at once (Elastisearch recommends bulk index sizes of 5MB-15MB).
If you plan to index a large number of documents and no new information is required for immediate search, you can optimize index performance rather than search performance by reducing the refresh frequency until indexing is complete.
Flush delay: Because data is not persisted to disk until Flush is successfully completed, it is useful to track Flush delay and take action when performance starts to sink. If this indicator increases steadily, it may indicate that the disk speed is slow. This problem can escalate and ultimately prevent new documents from being added to the index. In this case, you can try to lower index.translog.flush_threshold_size, which determines the size that the translog size can reach before the flush is triggered. If Elasticsearch writes are heavy, consider using iostat to focus on disk I/O.
Memory and garbage collection
Memory is one of the key metrics to monitor. Elasticsearch and Lucene make use of all available memory on the node in two ways: JVM heap and file system cache. Elasticsearch runs in the Java Virtual Machine (JVM), which means that the duration and frequency of JVM garbage collection also needs to be monitored.
The JVM heap
Using Elasticsearch requires setting the appropriate JVM heap size. In general, the rule of thumb for Elasticsearch is to allocate less than 50% of the available RAM to the JVM heap and never go above 32GB. The smaller the heap allocated to Elasticsearch, the more memory Lucene has available, and Lucene relies heavily on the file system cache to respond quickly to requests; However, you should not set it too small, because you may experience low memory or throughput degradation due to frequent GC. This blog post describes it in detail.
The default JVM heap size of Elasticsearch is 1 GB, which is too small in most scenarios. You can export the desired heap size as an environment variable and restart Elasticsearch.
export ES_HEAP_SIZE=10g
Copy the codeThe garbage collection
Elasticsearch relies on the garbage collection (GC) process to free heap memory. Be aware of the frequency and duration of GC. Making the heap too large leads to long garbage collection times; These excessive pauses are dangerous and cause the other nodes in the cluster to think that the node is unresponsive.
| Indicator description | Indicators of | The index type |
|---|---|---|
| Total number of garbage collected by young generation | jvm.gc.collectors.young.collection_count | |
| Young generation garbage collection time | jvm.gc.collectors.young.collection_time_in_millis | |
| The total amount of garbage collected by the older generation | jvm.gc.collectors.old.collection_count | |
| Garbage collection time of the elderly generation | jvm.gc.collectors.old.collection_time_in_millis | |
| Current JVM heap ratio | jvm.mem.heap_used_percent | |
| The amount of COMMITTED JVM heap | jvm.mem.heap_committed_in_bytes |
The key indicators
JVM heap in use: Elasticsearch is set to start garbage collection when THE JVM heap utilization reaches 75%. It might be useful to monitor which nodes exhibit high heap utilization and set alerts to see if any of them are consistently using more than 85% of the heap memory: this indicates that garbage collection is not keeping pace with garbage creation. You can solve this problem by increasing the heap size or expanding the cluster by adding more nodes.
Used heap vs. committed heap: Heap memory in use usually follows a sawtooth pattern, rising when garbage piles up and falling when garbage is collected. When the ratio of used to committed heaps increases, it means that the rate of garbage collection is not keeping pace with the rate of object creation, which can result in slower garbage collection times and ultimately OutOfMemoryErrors.
GC duration and frequency: Collection Both young and old generation garbage collections go through a “world stop” phase, as the JVM stops executing programs to collect. During this time, the node cannot complete any tasks. The master node will check the other nodes every 30 seconds. If the garbage collection time of any node exceeds 30 seconds, the master node will consider the node to have died.
Memory utilization: As mentioned above, Elasticsearch makes full use of any RAM not already allocated to the JVM heap and relies on the operating system’s file system cache to handle requests quickly and reliably. There are many variables that determine whether Elasticsearch is successfully read from the file system cache. If the segment file was recently written to disk by Elasticsearch, it is already in the cache; However, if the node has been shut down and restarted, it is likely that the first time the segment is queried, the information will have to be read from disk. This is one of the important reasons why you need to ensure that the cluster remains stable and that the nodes do not crash.
The host indicators
I/O: Elasticsearch does a lot of writing and reading to the disk while creating, querying, and merging segments. Elasticsearch recommends using SSDS to improve performance for a large number of clusters with nodes that continue to experience a lot of I/O activity.
CPU usage: It is useful to visualize CPU usage. Increased CPU utilization is usually caused by a high number of search and index requests.
Network outgoing/incoming bytes: Communication between nodes is key to balancing the cluster. In addition to Elasticsearch providing transport metrics for cluster communication, you can also view the byte rates your network card is sending and receiving.
Number of open files: File descriptors are used for node-to-node communication, client connections, and file operations. If this number reaches the maximum capacity of the system, new connections and file operations cannot be performed until the old connection is closed.
Cluster status and node availability
| Indicator description | Indicators of | The index type |
|---|---|---|
| State of the cluster | cluster.health.status | |
| Number of nodes | cluster.health.number_of_nodes | availability |
| Number of fragments in initialization | cluster.health.initializing_shards | availability |
| Number of unallocated fragments | cluster.health.unassigned_shards | availability |
Cluster status: If the cluster status is yellow, at least one replica fragment is unallocated or lost. Search results are still intact, but if more shards disappear, data may be lost.
Red cluster status means that at least one master shard is missing and data is being lost, which means the search will return partial results.
Initialized and unallocated Shards: When an index is first created or a node is restarted and its host node tries to assign shards to the node, the shard is temporarily in the “initialized” state until it transitions to the “started” or “unallocated” state. If a fragment remains in the initialized or unallocated state for too long, it may indicate that the cluster is unstable.
conclusion
In this article, we cover some of the most important areas of Elasticsearch so that you can monitor it as you scale and extend your cluster.
When monitoring Elasticsearch indicators and node-level system indicators, you can select the most important indicators based on specific usage scenarios.