A, requirements,

Use FileBeat to collect system logs into ElasticSearch.

Read log files in the system to eliminate unnecessary data.
Multi-line log handling.
Sensitive information (such as passwords) in filebeat.yml needs to be placed infilebeat keystoreIn the.
Use custom index templates.
The collected logs are deleted.
Ingest Node pipeline using ES to process data (adding fields, deleting fields, changing data types, etc.)

Second, the implementation

1. Preparation of filebeat.yml configuration file

filebeat.inputs:
- type: log
  # Enable/disable
  enabled: true
  encoding: "utf-8"
  # Collect logs from that path. If there are multiple inputs, it is best not to collect logs from that path twice, otherwise there will be problems
  Log paths can be written as wildcards
  paths:
    - "/Users/huan/soft/elastic-stack/filebeat/filebeat/springboot-admin.log"
  # If the word DEBUG appears in the log, the log is excluded
  exclude_lines:
    - "DEBUG"
  Add a custom field
  fields:
    "application-servic-name": "admin"
  Fields in # fields are not placed at the root level; true means they are placed at the root level
  fields_under_root: false
  Add a custom tag
  tags:
    - "application-admin"
  # Multi-line log handling, such as the exception stack in Java
  multiline:
    # regular expressions
    pattern: "^ \ \ [+"
    # Indicates whether to enable regular matching. The value is true or false
    negate: true
    # put the line that does not match the re after or before the line that matches the re.
    match: after
    If the last multi-line log was received, the last multi-line log is considered to be finished
    timeout: 2s
  [ignes node] [ignes node] [ignes node] [ignes node] [ignes node] [ignes node] [ignes node] [ignes node] [ignes node]
  pipeline: pipeline-filebeat-springboot-admin
  
Configure the name of the index template and the format of the index schema
setup.template.enabled: false
setup.template.name: "template-springboot-admin"
setup.template.pattern: "springboot-admin-*"

Index life cycle, need to disable, otherwise you may not be able to use the custom index name
setup.ilm.enabled: false

Fingerprint if our data does not have a unique primary key otherwise we can use add_id
processors:
  # fingerprint prevents the same piece of data from existing more than once in es of output. (In order to demonstrate the use of the message field as a fingerprint, the actual situation should be different depending on the business is not used)
  - fingerprint:
      fields: ["message"]
      ignore_missing: false
      target_field: "@metadata._id"
      method: "sha256"

Output to es
output.elasticsearch:
  # es address
  hosts: 
    - "http://localhost:9200"
    - "http://localhost:9201"
    - "http://localhost:9202"
  username: "elastic"
  password: "123456"
  # output to the index, because our local custom of the index name, so you need to setup below. The template. [name | pattern] configuration
  index: "springboot-admin-%{[agent.version]}-%{+yyyy.MM.dd}"
  # Enable/disable
  enabled: true
Copy the code

⚠️ : 1. The index life cycle must be disabled; otherwise, the user-defined index name may fail to be used. 2. It is estimated that there is a bug in fileBeat (7.12.0) version. Pipeline needs to be written in input stage, but writing in output stage does not take effect.

2. Create a custom index template

PUT /_template/template-springboot-admin {# Any index that starts with springboot-admin- will be matched at index creation time."index_patterns": ["springboot-admin-*"], # an index may match multiple index templates. Use order to control the order"order": 0."mappings": {
    "properties": {
      "createTime": {"type": "date"."format": ["yyyy-MM-dd HH:mm:ss.SSS"]}}}}Copy the code

Set createTime to date for simple demonstration.

3. Encrypt the password connected to es user

You can see the following information

output.elasticsearch:
  username: "elastic"
  password: "123456"
Copy the code

The username is in clear text, this is not secure, we use the FileBeat keystore to store the password.

1. Create keystore

./filebeat keystore create
Copy the code

2. Add an ES_PASSWORD key

./filebeat keystore add ES_PASSWORD
Copy the code

At the next prompt, enter your password. ES_PASSWORD is user-defined and will be used in modifying es Output in the filebeat.yml configuration file.

3. List how many keys there are in the keystore

./filebeat keystore list
Copy the code

4. Delete a key in the keystore

./filebeat keystore remove KEY(e.g. : ES_PASSWORD)Copy the code

5. Change the es password in filebeat.yml

4. Use es ingest node pipeline to process data

Ingest Pipeline allows us to perform operations such as common transformations on data before indexing it. ** For example, ** can convert data types, delete fields, add fields and other operations.

PUT _ingest/pipeline/pipeline-filebeat-springboot-admin
{
  "description": "Pipeline processing for springboot-admin project logs"."processors":[{"grok": {
        "field": "message"."patterns": [
          """(? m)^\[%{INT:pid}\]%{SPACE}%{TIMESTAMP_ISO8601:createTime}%{SPACE}\ [%{DATA:threadName}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{JAVACLASS:javaClass}# (? <methodName>[a-zA-Z_]+):%{INT:linenumber}%{SPACE}-%{GREEDYDATA:message}"""]."pattern_definitions": {
          "METHODNAME": "[a-zA-Z_]+"
        },
        "on_failure":[{"set": {
              "field": "grok_fail_message"."value": "{{_ingest.on_failure_message }}"}}},"set": {
        "field": "pipelineTime"."value": "{{_ingest.timestamp}}"
      },
      "remove": {
        "field": "ecs"."ignore_failure": true
      },
      "convert": {
        "field": "pid"."type": "integer"."ignore_failure": true}}, {"convert": {
        "field": "linenumber"."type": "integer"."ignore_failure": true}}, {"date": {
        "field": "createTime"."formats": [
          "yyyy-MM-dd HH:mm:ss.SSS"]."timezone": "+ 8"."target_field": "@timestamp"."ignore_failure": true}}}]Copy the code

5. Prepare test data

[9708] 2021-05-13 11:14:51.873 [HTTP-nio-8080-exec-1] INFO org.springframework.web.servlet.DispatcherServlet#initServletBean:547 -Completed initialization in 1 ms [9708] The 2021-05-13 11:14:51. 910 [HTTP - nio - 8080 - exec - 1] ERROR com. Huan. Study. LogController# showLog: 32 - request: [/ showLog] exception occurred java.lang.ArithmeticException: / by zero at com.huan.study.LogController.showLog(LogController.java:30) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)Copy the code

6. Run FileBeat

./filebeat -e-c (FileBeat configuration file path)Copy the code

Explanation:

-eOutput logs toStderr, the default output tosysloglogs/filebeatFile.
-cThe specifiedfilebeat.ymlPath to the configuration file

7. View the results

inkibanaCreate the index schema on, and then view the log.

How to read the same file more than once

Delete the contents of the Data/Registry folder. Different filebeat installation, the position of the data directory, refer to the following documents at www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html

4. Data deduplication

We know that in ES, each document data has a document ID, which is automatically generated by ES by default, so duplicate document data can produce multiple documents. The solution is as follows:

Fingerprint if our data does not have a unique primary key otherwise we can use add_id
processors:
  # fingerprint prevents the same piece of data from existing more than once in es of output. (In order to demonstrate the use of the message field as a fingerprint, the actual situation should be different depending on the business is not used)
  - fingerprint:
      fields: ["message"]
      ignore_missing: false
      target_field: "@metadata._id"
      method: "sha256"
Copy the code

Filebeat encountered a pit using es ingest node pipeline

In the use offilebeatAs we know from the website,pipelineThis is written inoutputIn the.But in the process of testing found that written inoutputIt doesn’t work in this one, it has to be ininputFor this, see configuration file. Online discussion of the issue: Github.com/elastic/bea…

Vi. Reference documents

1, 2, www.elastic.co/guide/en/be, www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html… 3, www.elastic.co/guide/en/be… 4, www.elastic.co/guide/en/be… 5, www.elastic.co/guide/en/be… Filebeat on 6, making the output to es, the discussion of the pipeline is not effective in July and www.elastic.co/guide/en/el… 8, www.elastic.co/guide/en/el…

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Filebeat collects logs into ElsticSearch and processes them using ingest Node’s pipeline

A, requirements,

Second, the implementation

1. Preparation of filebeat.yml configuration file

2. Create a custom index template

3. Encrypt the password connected to es user

1. Create keystore

2. Add an ES_PASSWORD key

3. List how many keys there are in the keystore

4. Delete a key in the keystore

5. Change the es password in filebeat.yml

4. Use es ingest node pipeline to process data

5. Prepare test data

6. Run FileBeat

7. View the results

How to read the same file more than once

4. Data deduplication

Filebeat encountered a pit using es ingest node pipeline

Vi. Reference documents

Filebeat collects logs into ElsticSearch and processes them using ingest Node’s pipeline

A, requirements,

Second, the implementation

1. Preparation of filebeat.yml configuration file

2. Create a custom index template

3. Encrypt the password connected to es user

1. Create keystore

2. Add an ES_PASSWORD key

3. List how many keys there are in the keystore

4. Delete a key in the keystore

5. Change the es password in filebeat.yml

4. Use es ingest node pipeline to process data

5. Prepare test data

6. Run FileBeat

7. View the results

How to read the same file more than once

4. Data deduplication

Filebeat encountered a pit using es ingest node pipeline

Vi. Reference documents

Related Posts

Kong Yiji’s four ways of writing “Hui” aroused my reflection on the realization of strategic mode

Cloud Native – What are microservices? What is microservices Architecture?

Java non-stop monitoring & hot operation