Kibana and Elasticsearch versions covered in this article: 7.4.1

Discover page is often teased by colleagues for its slow opening and timeout search. After looking at some es performance optimization articles and ruling out server performance versus ES configuration issues, I pointed the finger at Kibana itself.

I made a request on the Discover page with the following conditions:

  • Request time: 2020.11.15 00:00:00.000-23:59:59.999
  • Number of files requested: 500 (default)
  • No other conditions were added.

In Response (in inspect function of Discover, request and Response corresponding to this search can be found), it can be seen that the number of files on this day is 1.35 million, and ES takes 10 seconds + to process the request. I asked a few more times, and the time was about 10 seconds. After analyzing it, I found several factors contributing to the influence:

  • Shard, Index, and Index Pattern
  • Date Histogram
  • Time Zone
  • Highlight

In the following articles, I will analyze one by one and propose solutions.

Request

{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30m",
        "time_zone": "Asia/Shanghai",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2020-11-15T00:00:00.000Z",
              "lte": "2020-11-15T23:59:59.999Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}
Copy the code

Response

{
  "took": 10514,
  "timed_out": false,
  "_shards": {
    "total": 7,
    "successful": 7,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1354407,
Copy the code

Shard, Index, and Index Pattern

In Response, you can see that the number of shards requested is 7. This is because in the index pattern I selected, there are seven unfrozen indexes, and each index contains only one shard.

GET /_cat/indices/stat-*? V&s =index&h=index, STH index STH stat-2020.11.02 true stat-2020.11.03 true stat-2020.11.04 true stat-2020.11.05 true Stat -2020.11.06 true stat-2020.11.07 true stat-2020.11.08 true stat-2020.11.09 true stat-2020.11.10 false Stat -2020.11.11 false stat-2020.11.12 false stat-2020.11.13 false stat-2020.11.14 false stat-2020.11.15 false The stat - 2020.11.16 falseCopy the code

(column STH stands for search.throttled, false is unfrozen index)

So the Discover page searches the entire index pattern without filtering.

GET stat-*/_search
Copy the code

However, if I specify a specific index to search for

GET the stat - 2020.11.15 / _searchCopy the code

The result will be reduced by about two seconds.

{
  "took" : 8125,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
Copy the code

Therefore, in view of this problem, we can choose the following two schemes to improve the search speed:

  1. Set a more specific pattern for your usual index
  2. Freeze infrequently used indexes

Date Histogram

A bar chart at the top of the Discover page shows the aggregated analysis results of Date Histogram over the search period. This is the part that corresponds to the request body

"aggs": {
  "2": {
    "date_histogram": {
      "field": "@timestamp",
      "fixed_interval": "30m",
      "time_zone": "Asia/Shanghai",
      "min_doc_count": 1
    }
  }
}
Copy the code

If we take away the aggregation analysis, our request times are almost always under two seconds.

If the full request is made using the Profile API, the initial query takes just over 200 milliseconds, whereas aggregation takes over 5 seconds.

"profile" : {
    "shards" : [
      {
        "searches" : [
          {
            "query" : [
              {
                "type" : "BooleanQuery",
                "description" : "#*:* #ConstantScore(DocValuesFieldExistsQuery [field=@timestamp])",
                "time_in_nanos" : 235074290,
                ......
              }
            ],
            ......
          }
        ],
        "aggregations" : [
          {
            "type" : "DateHistogramAggregator",
            "description" : "2",
            "time_in_nanos" : 5519704113,
            ......
          }
        ]
      }
    ]
  }
Copy the code

Date Histogram is useful in some scenarios, such as when we want to view time patterns or find specific periods of time. But this is not useful in certain situations, such as when we want to retrieve information that is less frequently present. So it would be nice if Date Histogram were optional.

There is a demand for this on Github of Kibana, but the issue mentioned in 2016 is still in the open state.

In the case that aggregation analysis cannot be avoided, we have to choose a longer interval in the Discover page to conduct aggregation analysis. However, in the case that the number of buckets is small, the strength of optimization is very small.

Time Zone

When searching for relevant issues on Github, I found that many of them mentioned that setting the Time zone would affect the performance of date Histogram. After deleting the time_zone condition in agGS, the search speed is reduced to 4 seconds. The time difference between not setting a Time zone and setting a Time zone is double.

As you can see from the Github Kibana#18853 summary, this can be optimized if we set up a time zone with a fixed time difference, such as Etc/ gmt-8. The reason is related to aggregation in ElasticSearch. Since many countries distinguish winter time and daylight saving time, es converts time doc by DOC when conducting timestamp related aggregation, which naturally increases the time consumed by aggregation analysis. However, if we choose a time zone with a fixed time difference, ES only needs to uniformly shift the results according to the time difference after the aggregation analysis.

Es in Java. Time. ZoneId. GetRules () isFixedOffset () method to determine whether a time zone conversion have winter daylight saving time. The last conversion time is 1991-09-15T00:00+09:00 to +08:00. The last conversion time is 1991-09-15T00:00+09:00 to +08:00. I looked it up and found out that China did have winter time and daylight saving time for six years from 1986 to 1991. See the encyclopedia for details.

ZoneId tz = ZoneId.of( "Asia/Shanghai" ); System.out.println(tz.getRules().isFixedOffset()); // false Instant now = Instant.now(); System.out.println(now); / / 2020-11-17 T10: soul. 989 z System. Out. The println (tz) getRules () previousTransition (now)); // Transition[Overlap at 1991-09-15T00:00+09:00 to +08:00]Copy the code

If you want to avoid timezone problems, set your timezone to Etc/ gmt-8.

ZoneId tz = ZoneId.of( "Etc/GMT-8" );
System.out.println(tz.getRules().isFixedOffset());	// true
Copy the code

Highlight

In addition to the above, hightlight can be turned off in kibana configuration, which will also greatly reduce the search time. Highlight was turned off early in the day, so it wasn’t included in the previous request.

conclusion

A few ways to optimize the search speed of the Discover page:

  1. Set a more specific pattern for your usual index
  2. Freeze infrequently used indexes
  3. On the Discover page, select a longer interval for aggregate analysis.
  4. In the Kibana configuration, change the time zone to Etc/GMT-8 (depending on your current time difference).
  5. Turn off highlight