An overview of the

This article is about translating the Policy Phases and Actions module of ElasticSearch lifecycle management (Policy Phases and Actions).

Note: Based on version 6.7

There are four phases in the index life cycle, in order of execution.

The name of the describe
hot The index is actively writing
warm Indexes are usually not written, but are still queried
cold Indexes are no longer updated and queries are rare. Information still needs to be searched, but it doesn’t matter if those queries are slow.
delete Indexes are no longer needed and can be safely deleted

Each of these phases is called a phase. The policy does not require each stage to be configured for the index. For example, one policy might define just the hot and delete phases, while another might define all four phases.

Timing

Index entry to a phase is based on the min_age parameter set for that phase. The index enters phase MIN_age only when the age of the index is greater than the min_age set in this phase. Configure parameters using a duration format (see Time units).

Min_age If not specified, the default value for each phase is 0 seconds.

The sample

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1d"."actions": {
          "allocate": {
            "number_of_replicas": 1}}},"delete": {
        "min_age": "30d"."actions": {
          "delete": {}}}}}}Copy the code

The above example policy moves the index to the warm phase after a day. Until then, the index is in wait state. After entering the warm phase, it will wait 30 days before entering the delete phase and dropping the index.

Min_age is usually the elapsed time since the time the index was created. If the index is flipped, min_age is the time that has elapsed since the index was flipped. The goal here is to perform the following phases and operations relative to when the data was last written to the rolling index.

Before index Lifecycle management checks min_age and transitions to the next phase, the previous phase must be completed.

Phase Execution

The current phase definition of the executing index policy is stored in the metadata of the index. Phases and their actions are compiled into a series of discrete steps that are executed sequentially. Because some ILM operations are more complex and involve multiple operations against indexes, each of these operations is done separately in a unit called a “step.” The explain lifecycle API exposes this information to us to see which steps our index is either performing next or currently performing.

Actions

The following list shows the configurable actions for each stage.

The order in which configured actions are performed in each phase is automatically determined by ILM and cannot be changed by changing the policy definition.

Hot stage

  • Set Priority
  • Rollover
  • Unfollow

Warm stage

  • Set Priority
  • Allocate
  • Read Only (read-only)
  • Force Merge
  • Shrink
  • Unfollow

Cold phase

  • Set Priority
  • Allocate
  • Freeze
  • Unfollow

Delete phase (Delete)

  • Delete

Allocate

Phase allowed: warm, cold

The Allocate operation allows you to specify which nodes are allowed to host shards of an index and set the number of copies. Behind the scenes, it is modifying index Settings for slice filtering and/or copy counting. It is optional to configure allocation rules when updating the number of copies. When configuring allocation rules, setting the number of copies is optional. Although this can be treated as two separate index setting updates, both can be configured at once.

Configuration options

Name Required Default Description
number_of_replicas no – The number of replicas to assign to the index
include no – assigns an index to nodes having at least one of the attributes
exclude no – assigns an index to nodes having none of the attributes
require no – assigns an index to nodes having all of the attributes

If number_of_replicas is not configured, then at least one of include, exclude, and require is required. Empty Allocate Actions that are not configured are invalid.

Example: Change a copy

In this example, the number of copies of the index has changed to 2, and the allocation rule has not changed.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "allocate" : {
            "number_of_replicas": 2}}}}}}Copy the code

Delete

Phase allows: DELETE

That’s what a delete does, it drops the index.

There are no options associated with this operation.

The sample
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "delete": {
        "actions": {
          "delete": {}}}}}}Copy the code

Force-Merge

Phase allows: DELETE

When you do this, the index becomes read-only. Please refer to the

The Force Merge operation merges indexes into a maximum number of segments.

Configuration options

The name of the Whether must The default describe
max_num_segments is – Number of segments to merge. To merge indexes completely, set it to1
# # # # # sample
“`bash
PUT _ilm/policy/my_policy
{
“policy”: { 
"phases": {
  "warm": {
    "actions": {
      "forcemerge" : {
        "max_num_segments": 1
      }
    }
  }
}
Copy the code

}}

#### Freeze allows: Cold This action is performed by calling [Freeze Index] API] API (freeze) (https://www.elastic.co/guide/en/elasticsearch/reference/6.7/freeze-index-api.html) (https://www.elastic . Co/guide/en/elasticsearch/reference / 6.7 / frozen - indices. HTML) index > freezing index will close index and at the same API calls to open it. > This causes the primary to not be allocated in a short time > causing the cluster to turn red until the primary color is allocated again. > This restriction may be removed in the future. # # # # # sample ` ` ` bash PUT _ilm/policy/my_policy {" policy ": {" phases" : {" cold ": {" actions" : {" freeze ": {}}}}}}Copy the code

Read-Only

Phase allowed: warm

This action sets the index to read-only (see: index.blocks. Write)

There are no options associated with this operation.

The sample
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "readonly": {}}}}}}Copy the code

Rollover

Phase allowed: HOT

  • Index format must match pattern *^. * – \ d + $*, for example (logs-000001).
  • Managed indexes must be setindex.lifecycle.rollover_aliasRollover for aliases, and indexes must also be written under aliases.

For example, if the index you want to manage has the alias my_data. The managed index “my_index” must be a write index of the alias. For more information, read Write Index Alias Behavior.

PUT my_index
{
  "settings": {
    "index.lifecycle.name": "my_policy"."index.lifecycle.rollover_alias": "my_data"
  },
  "aliases": {
    "my_data": {
      "is_write_index": true}}}Copy the code

When an existing index meets one of the rolling update criteria, the Rollover Action transfers the alias to the new index.

Configuration options

Name Required Default Description
max_size no – Maximum primary shard index storage size. Please see theByte unitsTo format
max_docs no – The maximum number of documents to contain in the index before scrolling.
max_age no – The maximum time in which the index was created. Please refer to theUnit of timeTo format

At least one max_size, max_docs, max_age, or any combination of the three is required to specify.

Example: scroll when index is too large

This example scrolls the index at a minimum of 100 gigabytes.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_size": "100GB"
          }
        }
      }
    }
  }
}
Copy the code
Example: scroll when index contains too many documents

This example flips the index when it contains at least 100 million documents.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_docs": 100000000}}}}}}Copy the code
Example: Scroll when index is too old

This example created the index at least 7 days ago.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "7d"
          }
        }
      }
    }
  }
}
Copy the code
Example: rollover when an index is too old or too large

This example rolls up the index when it was created at least 7 days ago or at least 100 gigabytes. In this case, the index is flipped when any of the conditions are met.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_age": "7d"."max_size": "100GB"
          }
        }
      }
    }
  }
}
Copy the code

Set-Priority

The phases can be hot, warm, or cold.

Once the policy enters the Hot, warm, or cold phase, this action sets the index priority for the index. Indexes with a higher priority are restored before indexes with a lower priority after a node is restarted. Generally, the hot phase should have the highest index and the cold phase should have the lowest index. For example, set the hot phase to 100, warm phase to 50, and Cold phase to 0. The default priority of indicators that are not set to this value is 1.

Configuration options

The name of the Need to be The default describe
priority is – Priority of the index. Must be 0 or greater. This value can also be set to NULL to remove the priority.
The sample
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "set_priority" : {
            "priority": 50}}}}}}Copy the code

Shrink

When you run this operation, the index becomes read-only (see: index.blocks. Write)

This reduces the existing index to a new index with fewer master shards. It calls the Shrink API to Shrink the index. Since it is a prerequisite to assign all primary shards of an index to a node, this operation will first assign the primary shard to a valid node. After shrinking, it swaps aliases that point to the original index into the new shrinking index. The new index will also have a new name: “shrink-“. Therefore, if the original index was called “logs,” the new index will be named “shrink-logs.”

Configuration options

The name of the Need to be The default describe
number_of_shards is – Number of shards to shrink. Must be a factor of the number of fragments in the source index.
The sample
PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "warm": {
        "actions": {
          "shrink" : {
            "number_of_shards": 1}}}}}}Copy the code

Unfollow

This operation can be used explicitly, as shown below, but it also runs before the Rollover and Shrink operations, as described in the documentation for these operations.

The unfollow operation does not have any options, and if a non-follower index is encountered, the unfollow operation leaves the index unchanged and allows the next operation to operate on it.

PUT _ilm/policy/my_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "unfollow": {}}}}}}Copy the code

Full Policy

With all of these actions, we can support complex administrative policies for our indexes. This policy will define an index that will start in the hot phase, rolling every 50 GB or 7 days. After 30 days, it enters a warm phase and increases the number of copies to 2, forcing consolidation and contraction. After 60 days, it enters the cold phase and is assigned to the “cold” node. After 90 days, the index is deleted.

PUT _ilm/policy/full_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d"."max_size": "50G"}}},"warm": {
        "min_age": "30d"."actions": {
          "forcemerge": {
            "max_num_segments": 1}."shrink": {
            "number_of_shards": 1}."allocate": {
            "number_of_replicas": 2}}},"cold": {
        "min_age": "60d"."actions": {
          "allocate": {
            "require": {
              "type": "cold"}}}},"delete": {
        "min_age": "90d"."actions": {
          "delete": {}}}}}}Copy the code

reference

  • Official Manual: Policy Phases and Actions
  • Policy-definitions. Asciidoc

conclusion

Welcome to wechat public account “Code zonE”, focusing on sharing Java, cloud computing related content, including SpringBoot, SpringCloud, microservices, Docker, Kubernetes, Python and other related technology dry goods, looking forward to meeting you!