Elkstack is used as a log analysis tool to collect Nginx access logs, project logs, heartbeat detection logs, and server measurement logs. A large number of indexes are generated every day, occupying disk space. Delete expired data to release disk space.

Delete using _delete_by_query

Delete By Query API

Curl -u Username: password -h'Content-Type:application/json' -d'{ "query": { "range": { "@timestamp": { "lt": "now-7d", "format": "epoch_millis" } } } } ' -XPOST "Http://127.0.0.1:9200/ * - * / _delete_by_query? pretty"Copy the code

explain

-u is in the format of userName:password, and Basic Auth is used for login. -u is not required if ElasticSearch does not use x-pack for secure login

-h indicates that the document type is in JSON format

-XPOST specifies the POST mode

-d specifies the body content

{
    "query": {
        "range": { / / range
            "@timestamp": {// Time field
                "lt": "now-7d"./ / lt is less than (<), lte is less than or equal to (< =), gt is greater than (>), gte is greater than or equal to (> =), now - 7 d is the current time minus 7 days
                "format": "epoch_millis"}}}}Copy the code

Time to delete

$ crontab -e

* 0 * * * /usr/bin/curl -u username:password  -H'Content-Type:application/json' -d'{"query":{"range":{"@timestamp":{"lt":"now-7d","format":"epoch_millis"}}}}' -XPOST "Http://127.0.0.1:9200/ * - * / _delete_by_query? pretty" > /tmp/elk_clean.txtCopy the code

Invalid indexes that are more than 7 days old are deleted at 0 o ‘clock every day

Advantages:

  • No reliance on third-party plug-ins or code

  • Easy to understand

  • The index name does not need to be specified. The * wildcard character can be deleted

Disadvantages:

  • Low efficiency

Use the sh script to delete

Removing old indices in elasticsearch# answers-39746705

#! /bin/bash
searchIndex=logstash-monitor
elastic_url=logging.core.k94.kvk.nl
elastic_port=9200

date2stamp () {
    date --utc --date "The $1" +%s
}

dateDiff() {case The $1 in
        -s)   sec=1;      shift;;
        -m)   sec=60;     shift;;
        -h)   sec=3600;   shift;;
        -d)   sec=86400;  shift;;
        *)    sec=86400;;
    esac
    dte1=$(date2stamp The $1)
    dte2=$(date2stamp $2)
    diffSec=$((dte2-dte1))
    if ((diffSec < 0)); then abs=-1; else abs=1; fi
    echo $((diffSec/sec*abs))
}

for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices? v" |     grep -E " ${searchIndex}- 20 [0-9] [0-9] \. [0, 1] [0-9] \. [0, 3] [0-9]" | awk '{ print $3 }'); do
  date=$(echo ${index: -10} | sed 's/\./-/g')
  cond=$(date +%Y-%m-%d)
  diff=$(dateDiff -d $date $cond)
  echo -n "${index} (${diff})"
  if [ $diff -gt1];then
    echo " / DELETE"
    # curl -XDELETE "${elastic_url}:${elastic_port}/${index}? pretty"
  else
    echo ""
  fi
doneCopy the code

The _CAT/indicesAPI is used.

Use the curator

Support windowszip,msi, and linuxapt,yum

Curator Reference github-curator

The installation

The installation

configuration

Reference stackoverflow.com/questions/3…

1. The config file

---
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
client:
  hosts:
    * 127.0. 01.
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth: username:password
  timeout:
  master_only: True

logging:
  loglevel: INFO
  logfile:
  logformat: default
  #blacklist: ['elasticsearch', 'urllib3']Copy the code

2. The action file

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    * filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    * filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:Copy the code

Here is to use the index – ‘% d % % y. m.’ match, if it is according to the index creation date to delete, source: creation_date see www.elastic.co/guide/en/el…

3. Run

curator --config /path/config_file.yml /path/action_file.ymlCopy the code

Don’t forget to add the scheduled task crontab -e

I original, reproduced please declare

Blog the nuggets