Introduce the role of the index lifecycle

In the daily management of Elasticsearch, there are a lot of, such as system log application scenario for the behavior data, etc. That is a large amount of data is the feature of the scene, and the number of growth index as time also will continue to grow, however, these scenarios are basically only recently for a period of time the data have the use value or (hot) are often use, Historical data is not useful or is rarely used (cold data), so you need to manage the index carefully or even delete it. Otherwise, as the data volume increases, the performance of Elasticsearch will be severely affected, wasting disk space and memory.

New to Elastic Stack 6.6 is Index Lifecycle Management, which supports Index Lifecycle Management and a UI for policy configuration on Kibana.

Common phases of the index life cycle

  • Hot: The index also has a lot of read and write operations.
  • Warm: Indexes do not have write operations and need to be queried.
  • Cold: There is no write operation and few reads.
  • Delete: The index is no longer needed and can be safely deleted.

Note: The above is a common definition of the index lifecycle phase, and the specific policies can be defined based on the actual business situation.

Deploy the Elasticsearch cluster

Deploy a Elasticsearch cluster consisting of 2 Hot, 2 warm, and 2 Cold nodes, and deploy Kibana and Cerebro for debugging and observation. Docker-comemage. yaml file is as follows:

Version: '2.2' Services: cerebro: image: lmenezes/cerebro:0.8.3 container_name: hwc_cerebro ports: - "9000:9000" command: - -Dhosts.0.host=http://elasticsearch:9200 networks: - hwc_es7net kibana: image: Docker. Elastic. Co/kibana/kibana: 7.1.0 container_name: hwc_kibana7 environment: #- I18N_LOCALE=zh-CN - XPACK_GRAPH_ENABLED=true - TIMELION_ENABLED=true - XPACK_MONITORING_COLLECTION_ENABLED="true" ports: - "5601:5601" networks: - hwc_es7net elasticsearch: image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_hot1 environment: - cluster.name=cr7-hwc - node.name=es7_hot1 - node.attr.box_type=hot - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_hot1:/usr/share/elasticsearch/data ports: - 9200:9200 networks: - hwc_es7net elasticsearch2: Image: docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_warm1 environment: - cluster.name=cr7-hwc - node.name=es7_warm1 - node.attr.box_type=warm - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_warm1:/usr/share/elasticsearch/data networks: - hwc_es7net elasticsearch3: image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_cold1 environment: - cluster.name=cr7-hwc - node.name=es7_cold1 - node.attr.box_type=cold - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_cold1:/usr/share/elasticsearch/data networks: - hwc_es7net elasticsearch4: image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_hot2 environment: - cluster.name=cr7-hwc - node.name=es7_hot2 - node.attr.box_type=hot - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_hot2:/usr/share/elasticsearch/data networks: - hwc_es7net elasticsearch5: image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_warm2 environment: - cluster.name=cr7-hwc - node.name=es7_warm2 - node.attr.box_type=warm - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_warm2:/usr/share/elasticsearch/data networks: - hwc_es7net elasticsearch6: image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.1.0 container_name: es7_cold2 environment: - cluster.name=cr7-hwc - node.name=es7_cold2 - node.attr.box_type=cold - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - discovery.seed_hosts=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 - cluster.initial_master_nodes=es7_hot1,es7_warm1,es7_cold1,es7_hot2,es7_warm2,es7_cold2 ulimits: memlock: soft: -1 hard: -1 volumes: - hwc_es7data_cold2:/usr/share/elasticsearch/data networks: - hwc_es7net volumes: hwc_es7data_hot1: driver: local hwc_es7data_warm1: driver: local hwc_es7data_cold1: driver: local hwc_es7data_hot2: driver: local hwc_es7data_warm2: driver: local hwc_es7data_cold2: driver: local networks: hwc_es7net: driver: bridgeCopy the code

Run the docker-compose up -d command to start the Elasticsearch cluster. (Docker and Docker-compose have been installed)

Example Change the IML refresh time

Set ILM refresh time to 1 second for experimental demonstration (default is 10 minutes) :

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval":"1s"
  }
}
Copy the code

Set Index Lifecycle Policies

Set the hot/warm/cold and DELETE phases:

  • Hot: rollover after more than 5 documents.

  • Warm: After 20 seconds, the index enters the warm phase and is set to read-only.

  • Cold: After 40s, it enters the warm phase, and shrinks the copies from 1 to 0.

  • Delete: After 60 seconds, the system enters the delete phase and deletes the index.

    PUT /_ilm/policy/log_ilm_policy { “policy”: { “phases”: { “hot”: { “actions”: { “rollover”: { “max_docs”: 5 } } }, “warm”: { “min_age”: “20s”, “actions”: { “allocate”: { “include”: { “box_type”: “warm” } }, “readonly”: {} } }, “cold”: { “min_age”: “40s”, “actions”: { “allocate”: { “include”: { “box_type”: “cold” }, “number_of_replicas”: 0 } } }, “delete”: { “min_age”: “60s”, “actions”: { “delete”: {} } } } } }

Create the first index

The index is allocated to the hot node, the log_ILM_policy IML policy is invoked, the rollover alias is set to ILm_alias, the master fragment is set to 1, and the copy fragment is set to 1. Setting “is_write_index”: true For rollover, alias will contain all Rollover files.

PUT ilm_index-000001
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "log_ilm_policy",
    "index.lifecycle.rollover_alias": "ilm_alias",
    "index.routing.allocation.include.box_type":"hot"
  },
  "aliases": {
    "ilm_alias": {
      "is_write_index": true
    }
  }
}
Copy the code

Create a document

Perform five consecutive POSTS to create five documents.

POST ilm_alias/_doc
{
  "name":"cr7",
  "age": 15
}
Copy the code

To observe the

Initially, you can see that there are five doc’s in total.

When iml_index-000001 reaches 5 doc, a rollover operation is performed and a new index iml_index-000002 is created.

After 20 seconds, move the imL_index-000001 index to the warm node.

After 40 seconds, move index IML_index-000001 to cold node and reduce the number of copies from 1 to 0.

After 60 seconds, index IMl_index-000001 is deleted.

If you try to write or modify data to IML_INDEx-000001 during the warm and Cold phases, you will receive the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/8/index write (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/8/index write (api)];"
  },
  "status": 403
}
Copy the code