This is the 11th day of my participation in the Novembermore Challenge.The final text challenge in 2021

Start the

The following is an example of a startup command:

docker run --log-opt max-size=100m --user root --net host --name filebeat -v $PWD/filebeat/data:/usr/share/filebeat/data  -v $PWD/filebeat/logs:/usr/share/filebeat/logs -v /logs:/usr/share/filebeat/logdata -e ES_HOSTS="es-1.dx.corp:9200,es-2.dx.corp:9200" -e LOG_PATH="/usr/share/filebeat/logdata/request_details.log" -d Filebeat: 5.4.3Copy the code

Note: You need to map the monitored log directory to the FileBeat container, and configure an environment variable LOG_PATH to specify the path in the FileBeat container. ES_HOSTS specifies the ES address, which is separated by commas (,). Note that the port number is HTTP. Configuration file can be remapped to containers, container path is/usr/share/filebeat/filebeat yml

Configuration and usage instructions

Close_inactive This parameter specifies how long the monitored file has not changed before the file handle is closed. The official recommendation is to set this parameter to a value greater than the maximum update interval for files. For example, if the file is updated every 5 seconds, set it to 1 minute. Default value: 5min. scan_frequency This parameter specifies the frequency (interval) at which Filebeat searches for new files. When a new file is created, Filebeat will start a harvester for it to monitor. The default value is 10s. The clean_INACTIVE: 72h configuration item should also be configured. The default value is 0 to indicate no cleanup, which means that the collected file description is never cleaned in the Registry file. After running for a while, Registry will become larger, which may cause problems. Ignore_older: 70h After clean_INACTIVE is set, ignore_older needs to be set and ensure that ignore_OLDER < clean_inactiveCopy the code

When logBack has finished cutting logs (renaming), the old Harvester is still monitoring the renamed log file, but since the file will not be updated again, it will close the harvester file after close_inactive time. When scan_frequency expires, Filebeat finds a new file in the directory and starts harvester to monitor that file. In this way, data can be transferred without loss or weight when cutting logs. Harvester is also restarted after scan_frequency for updates to the same file over a longer period of time

2020/08/20 06:32:18.206200 log.go:116: INFO File is inactive: /usr/share/filebeat/logdata/request_details.log. Closing because close_inactive of 5m0s reached.Copy the code
197758 log.go:91: INFO Harvester started for file: /usr/share/filebeat/logdata/request_details.logCopy the code

 

The Data/Registry file of FileBeat stores information about all logs that are collected.

A log in Registry in Linux records the following contents

[{ "source": "/usr/share/filebeat/logdata/request_details.log", "offset": 29973, "FileStateOS": { "inode": 5784066, "device": 2049}, "TIMESTAMP ": "2020-08-20T14:20:28.328896978+08:00"," TTL ": 259200000000000}, {"source": "/usr/share/filebeat/logdata/request_details.log", "offset": 1430, "FileStateOS": { "inode": 5784064, "device": 2049}, "timestamp": "2020-08-20T14:27:13.644183981+08:00", "TTL ": 259200000000000}, {"source": "/usr/share/filebeat/logdata/request_details.log", "offset": 1427, "FileStateOS": { "inode": 5784049, "device": 2049}, "TIMESTAMP ": "2020-08-20T14:27:44.14364326+08:00"," TTL ": 259200000000000}]Copy the code

The meanings of the fields in this record are respectively

Source Complete log file path offset Indicates the number of bytes of collected logs. Filestateos Inode log file inode number // Linux, Ls -i device Id of the disk where the log resides timestamp Timestamp of the last change in the log TTL TTL collection validity time. -1: collects logs as long as the logs existCopy the code
[dx@cdh3 logs]$ls -il request_details.* 5772009 -rw-r--r-- 1 root root 0 Mar 17 11:06 request_details.2020-03-17.0.log 5784066-rw-r --r-- 1 root root 29973 Aug 20 14:15 request_details.2020-08-20.log 5784064-rw-r --r-- 1 root root 1430 Log 5784049 -rw-r--r-- 1 root root 1427 Aug 20 14:27 request_details.logCopy the code

It can be seen that the offset corresponding to the three files 5784066, 5784064 and 5784049 in the Registry file is equal to the file size, indicating that the collection has been completed