Shard-level slow search logging allows slow searches (query and fetch phases) to be logged to a dedicated log file. Logging is an integral part of any application. For a distributed solution like Elasticsearch, which has to handle a large number of requests, logging is inevitable and critical.

As the name implies, slow logging is used to record slow requests, whether search or index requests. We can set a “slow” threshold so that only those requests above that threshold are logged.

For Elasticsearch, slow logging is important because:

  • They help determine whether the application that should communicate with Elasticsearch is doing so.
  • We can examine queries that affect Elasticsearch and verify.
  • Logging can help maintain the cluster by providing important information about index/cluster health.

Now, based on your theory, let’s introduce the main types of slow logging available and their use cases.

 

Slow Log Classification

There are two main types of slow logs in Elasticsearch: search slow logs and index slow logs. Let’s discuss it.

Search Slow Logs

Slow search logs record slow searches. The slowness threshold depends on the application and its Elasticsearch implementation details. Each application can have different thresholds.

Searching in Elasticsearch is divided into two stages:

  1. Query phase – During the query phase, Elasticsearch collects the document ID of the related results. When this phase is complete, only the ids of the documents that matched the search are returned, and no additional information, such as fields or their values, appears.
  2. Fetch phase – During the fetch phase, the document ID from the query phase is used to fetch the actual document, so that the search request can be said to be complete.

Search slow logs show the split time of the query and the query acquisition phase. As a result, we have a complete picture of how long it took to complete the query and fetch phases, and we can examine the entire query itself.

 

Index Slow Logs

Indexing slow logs record the indexing process. After the document is indexed in Elasticsearch, the slow index log records the requested records, which take a long time to complete. Here, too, the time window can be adjusted in the index log configuration Settings.

By default, Elasticsearch logs the first 1000 lines of a document to a log file when enabled. This can be changed to NULL or the entire document can be logged, depending on how we configure the Settings.

In the next section, let’s look at how to configure logging and examine the two types of slow logging discussed above.

 

Index slow logging Settings

First, create the test index for which you want to configure indexed slow logs.

PUT testindex-slowlogs
Copy the code

Now configure slowlogs for index “testindex-slowlogs” as follows:

PUT testindex-slowlogs/_settings
{
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.threshold.index.info": "5s",
  "index.indexing.slowlog.threshold.index.debug": "2s",
  "index.indexing.slowlog.threshold.index.trace": "500ms",
  "index.indexing.slowlog.level": "info",
  "index.indexing.slowlog.source": "1000"
}
Copy the code

We can update these Settings with _settings. By default, they are disabled (set to -1). Levels (WARN, INFO, DEBUG, trace) allow you to control at which logging levels logs will be logged. Not all requirements are configured (for example, only a WARN threshold can be set). Several levels have the benefit of being able to quickly “grep” against specific thresholds for violations. By default, Elasticsearch will log the first 1000 characters of _source in the slow log. You can use the index. The indexing. Slowlog. Source changes. Setting it to false or 0 skips logging of the source entirely, and setting it to true logs the entire source regardless of size. By default, the original _source is reformatted to ensure that it fits a single log line. If it is very important to retain the original document format, you can through the index. The indexing. Slowlog. Reformat set to false to close to reformat, this will lead to the source according to the “as-is” records, and may span multiple log line.

For testing purposes, we set the above thresholds to 0 and the default values. We use _settings to fix our configuration:

PUT testindex-slowlogs/_settings
{
  "index.indexing.slowlog.threshold.index.warn": "0ms",
  "index.indexing.slowlog.threshold.index.info": "0ms",
  "index.indexing.slowlog.threshold.index.debug": "0ms",
  "index.indexing.slowlog.threshold.index.trace": "0ms",
  "index.indexing.slowlog.level": "trace",
  "index.indexing.slowlog.source": "1000"
}
Copy the code

So if any of these thresholds are greater than zero, then you’re going to generate an Index slow log. We entered the following document in Kibana:

POST testindex-slowlogs/_doc
{
  "price": 9925,
  "name": "Nariko"
}
Copy the code

Elasticsearch_index_indexing_slowlog.log: elasticSearch_index_indexing_slowlog.log:

Open the content of elasticSearch_index_indexing_slowlog. log and you can see:

Here we can see all the index slow log records. Because we set the threshold to 0, any one of these operations triggers the corresponding indexed slow log operation. In actual applications, you can set thresholds according to actual requirements.

These logs provide the following information:

  • time stamp
  • log level
  • type of log
  • node name
  • index name
  • time taken in micro/milli seconds
  • index type
  • document id
  • "_source" field

Using this information, we can understand the progress of the indexing operation and can detect/debug any exceptions (if any).

 

Search Slow Logging

Similar to slow index logs, search slow logs are applied to specific indexes. An example configuration for logging all search requests is as follows:

PUT testindex-slowlogs/_settings
{
  "index.search.slowlog.threshold.query.warn": "0ms",
  "index.search.slowlog.threshold.query.info": "0ms",
  "index.search.slowlog.threshold.query.debug": "0ms",
  "index.search.slowlog.threshold.query.trace": "0ms",
  "index.search.slowlog.threshold.fetch.warn": "0ms",
  "index.search.slowlog.threshold.fetch.info": "0ms",
  "index.search.slowlog.threshold.fetch.debug": "0ms",
  "index.search.slowlog.threshold.fetch.trace": "0ms",
  "index.search.slowlog.level": "info"
}
Copy the code

Because all parameters of the above Settings are 0ms, all search requests are logged. The log file name is “ElasticSearch_index_search_slowlog.log” in the log directory.

Run the following query on the index:

GET testindex-slowlogs/_search
{
  "query": {
    "match": {
      "name": "Nariko"
    }
  },
  "sort": [
    {
      "price": {
        "order": "desc"
      }
    }
  ]
}
Copy the code

After this query is successfully executed, the search slow log file above is opened. You’ll find logs that look something like this:

You can look at the file elasticSearch_index_search_slowlog.log. Since we set the log threshold to 0, each search produces a corresponding log record. In actual use, we can set the corresponding threshold according to our own situation. Check out elasticSearch_index_search_slowlog. log:

As you can see, two logs were generated above for a single search. This is because the search is performed by shard and there are 2 shards in the index under the default configuration. When we performed the search, the request was passed to each of the two shards and two logs were generated.

In these logs, we can view details such as search type, node, and shard number information with detailed query.

conclusion

In this tutorial, we discussed the importance of slow logging for Elasticsearch. We covered indexing and searching slow logs, as well as configuration examples and log samples. See “Elasticsearch: Monitoring Elasticsearch and Kibana” if you know how to import logs into Elasticsearch and analyze them first.

 

Reference:

【 1 】 qbox. IO/blog/missile – l…