At the beginning

In a distributed system, logging trace is a very to programmers headaches, in case of production problems, if it is need to open many more node server to tracking problem, if the downstream is also more nodes and calls to multiple services, it is more trouble, moreover, if there is no distributed link, in the case of production logging sliding rapidly, it is difficult to find out the problem. Therefore, it is necessary to build a distributed log system in the distributed system. The author uses the mature solution ELK+ Skywalking in the market to solve the problem. This paper will build a distributed log system from 0 to 1.

The effect

Kibana: You can view online error logs in Kibana. Trace_id indicates the unique link ID of the request. Skywalking: You can search for logs in Skywalking by using trace_id in 1

image.png

image.png

Architecture diagram

The document address: www.processon.com/view/link/6…

image.png

2. Elasticsearch1: used to store Skywalking link data 3. Filebeat: Used to store Skywalking link data 4. Logstash: Used to filter valid log information, such as collecting IP, TID and other information, defining index specification. Data store interconnects with ES 5. Elasticsearch2: stores log information filtered by logStash. This document stores error logs

Set up process

Machine configuration: two 32G servers, ES not on the cluster version, as needed expansion

1. Elasticsearch and Kibana setup

Set up before, will not go into here, specific see it www.jianshu.com/p/a69f8cefe…

2. Skywalking structures

Download address: archive.apache.org/dist/skywal… Download the apache- Skywalking – APm-es7-8.4.0.tar. gz file. Decompress the file and change the port number on the webui. Yml modify skywalking OAP service configuration file conf/ applica. yml and save logs to ES configuration, indicating that trace records are maintained for 7 days. RecordDataTTL: SW_STORAGE_ES_RECORD_DATA_TTL:7 otherMetricsDataTTL: SW_STORAGE_ES_OTHER_METRIC_DATA_TTL:45 monthMetricsDataTTL: SW_STORAGE_ES_MONTH_METRIC_DATA_TTL:18

2. Start apache-skywalking-apm-bin/bin/startup.sh The script will start two services, one is webUI services. 3. Upload apache-Skywalking – APm-ES7-8.4.0.tar. gz on application machine 02. Decompress ④ Modify the SpringBoot startup script (Java Agent probe technology) Java-JavaAgent: apache-Skywalk-apm-bin /agent/ Skywalk-agent.jar -Dskywalking.agent.service_name=${pro} -Dskywalking.collector.backend_service=xxx:11800 -jar xxx.jar >> Logs/Catalina. out & ⑤ Springboot integrates skywalking service

<! --skywalking--> <dependency> <groupId>org.apache.skywalking</groupId> <artifactId>apm-toolkit-logback-1.x</artifactId> <version>8.4.0</version> </dependency> logback-spring. XML configuration <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder"> <layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout"> <! Trace ID --> <Pattern>${APP_NAME}:${ServerIP}:${ServerPort} %d{YYYY-MM-DD HH: MM: ss.sss} %-5level %tid %t % Logger {36}: % MSG %n</Pattern> </layout> </encoder> Copy codeCopy the code

3. Filebeat structures

Create a filebeat-logstash. Yml multiline.pattern file on a JAVA application server. ‘^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}’

image.png

/ filebeat-e -c filebeat-logstash. Yml

4. The logstash

Conf message in the logstash-7.6.2/config file. Mapping index Indicates the index name. The index is created by day, and the scheduled task is saved in the latest X days

image.png

Sh nohup bin/logstash -f config/filebeat-grok.conf –config.reload. Automatic > logstash

image.png

Note: Start the Logstash first, then fileBeat

conclusion

1. Ensure the security of Kibana and Skywalking Web UI, and set the login password through Nginx. 2. For example, retain the logs of the last 7 days. 4. Skywalking enhanced: Provides the alarm service and GRPC log upload service. Cons: FileBeat and SkyWalking agent services need to be installed on every application machine