The installation of the logstash

Download the installation package, download the address

Decompress the installation package

Basic configuration of logstash

There’s not a lot of configuration in logstash itself, This configuration is complicated by the fact that you can use beat to fetch data and then push it to elasticSearch (we will only talk about beat to fetch data). This configuration has a default configuration file named logstash-sample. Of course, this simple configuration does not meet our basic needs, but let’s take a look at the official configuration:

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

Copy the code

It can be seen that the two main configuration points in the configuration are input and output. In fact, we will often use filter in the subsequent use process, and its structure is shown in the figure below:

This configuration will be exemplified in a subsequent example.

Logstash common command

Start.\logstash. Bat -f.. \config\logstash-sample.conf

Filebeat installation

Download the installation package, download the address

Decompress the installation package

Filebeat configuration

Filebeat is one of Beats and a common one. Generally speaking, elasticSearch and LogStash can be used for multiple projects, but each project needs a corresponding FileBeat to collect logs according to the corresponding address.

Modify the config/filebeat. Yml

Filebeat. Inputs

  • Enable this input configuration:

    Inputs filebeat.inputs: Change enabled to true

  • Change the collection file address:

    Inputs Under filebeat.inputs, change all paths to their own addresses (two paths)

  • Configure the service name (this configuration corresponds to the logstash configuration, note!).

    Remove all comments (two) #fields for fields

    Added service: nginx under fields

    fields:
    service: nginx-log
Copy the code

  • Configure Multiline to match multi-line messages

    To give an example, you can adjust your log format to match multiple lines of log printed on one line

# to match all the regular format, and a log on a behavior (blank at the beginning of, for example, under Caused by opening) multiline. The pattern: '^ [[: space:]] + (ats | \. {3}) \ | b ^ under Caused by:' # true: Negate: false # Divide the current line with [after] or [before] into a group of multilines. match: afterCopy the code

Kibana part

Setup. kibana: Under host contact annotation, configure as your own address

Outputs of

  • Comment output to ElasticSearch

    Elasticsearch Output is annotated as output. Elasticsearch and hosts for Elasticsearch Output because logstash is used

  • Configure output to logstash

    Uncomment output.logstash and hosts

    Hosts Sets its own address

Filebeat Common command

Start:./filebeat -e -c filebeat.yml