Elasticsearch Index lifecycle management

preface

Elasticsearch has the following typical features when storing system logs in the ELK architecture:

• It’s a huge amount of data
• Access new data frequently, and its value diminishes over time

The number of indexes created by Elasticsearch increases with the increase of data volume. Therefore, you need to maintain and manage indexes or even delete indexes. Otherwise, the increased data volume will waste disk and memory space and seriously affect Elasticsearch performance.

For Elasticsearch, Index Lifecycle Management(ILM) has been introduced in version 6.6 and will be available in version 6.7.

This example uses the following environment:

• Elasticsearch version 7.2.0
• Kibana version 7.2.0
• Number of Elasticsearch nodes: 2

An overview,

Index lifecycle management ILM is a life-cycle management process for Elasticsearch to set, create, open, close, and delete indexes.

1.1 Phase Introduction

The index life cycle is divided into four phases: hot, warm, Cold, and DELETE.

Hot stage is mainly responsible for rolling update of index, warm, cold, delete stage is mainly responsible for further processing of rolling update data.

Detailed introduction

Add the life cycle management mode

• Add lifecycle management policies through index templates

  Apply the policy to the index covered by the entire alias

• Add a lifecycle management policy for a single index

  Only the current index can be overwritten. Newly scrolled indexes are not affected by the policy.

Phase action

1.2 common Action

Configure life cycle management

2.1 requirements

The index policies are as follows:

• The initial creation index contains a master shard and a replica shard
• The newly created index is in the HOT phase by default
• When the number of documents reaches five, the old index moves from the hot phase to the warm phase
• During the Warm phase, index replica shards are deleted, leaving only the master shard
• After 10 seconds, the index moves from the warm phase to the DELETE phase, and after 20 seconds, the index is dropped.

Step 2.2

• Step 1: Configure lifecycle detection time.
• Step 2: Create an index policy.
• Step 3: Create an index template that specifies the index policy to use.
• Step 4: Create an index that matches the above index template.
• Step 5: Write data to the index so that the index triggers a rolling update policy.
• Step 6: View the stage of the index

2.3 implementation

(1) Configure the lifecycle detection time

PUT /_cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1s"}}Copy the code

The default is 10 minutes, but changed to 1 second to test the effect. You do not need to change this if you are in production.

(2) Create an index policy

# / _ILm /policy is a fixed format. Leefs_ilm_policy is the name of the index policy. PUT / _ILm /policy/ leefS_ILM_policy {# policy: Configure the policy"policy": {# phases: Phase configuration"phases": {
      "hot": {
        "actions": {# rollover: rolling updates"rollover": {# max_docs: maximum number of documents5Perform operations"max_docs": "5"}}},"warm": {# min_age: indicates the minimum dwell time of this phase"min_age": "10s"."actions"{# allocate: Specifies the number of copies of an index"allocate": {# number_of_replicas: Sets the number of index replicas"number_of_replicas": 0}}},"delete": {
        "min_age": "20s"."actions": {# delete: deletes the index. Without this method, the index will not be deleted even in the delete stage"delete": {}}}}}}Copy the code

instructions

Not every phase is necessary when creating an index policy, and except for the HOT phase, all phases can be omitted as required.

All methods in actions, including rollover, can be omitted as required.

(3) Create an index template and specify the index policy to use

PUT _template/ leefS_ilM_template {# matching index name to the template"leefs_logs-"At the beginning"index_patterns": ["leefs_logs-*"]."settings": {# number_of_shard: Sets the number of primary shards"number_of_shards": 1, # number_of_replicas: Sets the number of replicas"number_of_replicas": 1, # specify the index policy that the template matches by the index policy name"index.lifecycle.name": "leefs_ilm_policy", # index rollover to switch the index alias to leefs_logs"index.lifecycle.rollover_alias": "leefs_logs"}}Copy the code

(4) Create an index that matches the above index template

DELETE leefs_logs* # create first index PUT leefs_logs- 000001.
{
  "aliases": {
    // Set the index alias to leefs_logs
    "leefs_logs": {
      // Allow indexes to be written to data
      "is_write_index":true}}}Copy the code

(5) Write data to the index to trigger the rolling update policy

POST leefs_logs/_doc? refresh {"name":"llc"
}
Copy the code

Execute the above name five times to trigger the rolling update policy.

(6) Check the stage of the index

GET leefs_logs-*/_ilm/explainCopy the code

The query results

GET _alias/Copy the code

Third, update the strategy

1. If no index applies this policy, then we can update the policy directly.
2. If the policy is applied to an index, the current phase will not be synchronized. When the current phase ends, the index will enter the next phase of the new policy.
3. If a policy is changed, the current phase does not change. After the current phase ends, the new policy will manage the next life cycle.

Start and stop index lifecycle management

ILM is enabled by default.

(1) View the current running status of ILM

GET  _ilm/status
Copy the code

The execution result

{
  "operation_mode" : "RUNNING"
}
Copy the code

ILM operation mode

(2) Stop ILM

POST  _ilm/stop
Copy the code

When it stops, all other policy measures will stop. This will be reflected in the status API

{
    "operation_mode": "STOPPING"
}
Copy the code

The ILM service then asynchronously runs all policies to a point where it is safe to stop. When ILM confirms that it is safe, it moves to the STOPPED mode

{
    "operation_mode": "STOPPED"
}
Copy the code

(3) Start ILM

POST  _ilm/start
Copy the code

Query the status after the startup

{
  "operation_mode": "RUNNING"
}
Copy the code

Attached is a reference article link:

Developer.aliyun.com/article/780…

Help.aliyun.com/document_de…