Lightweight distributed log tracking tool, 10 minutes to access, log tracking is easy

preface

With the proliferation of microservices, many companies are splitting their systems into microservices along business boundaries when it comes to troubleshooting and log checking. Because the service link runs through many microservice nodes, it is difficult to locate the log of a request and the log of upstream and downstream services.

At this time a lot of children’s shoes will start to consider on SkyWalking, Pinpoint distributed tracking system to solve, based on OpenTracing specification, and is usually non-invasive, and has a relatively friendly management interface to link Span query.

However, it takes a certain period of time to build a distributed tracking system, familiarize with and promote the system to the whole company, and it involves the storage cost of span nodes. Is it full collection or partial collection? For full collection, using SkyWalking’s storage as an example, an ES cluster requires at least five nodes to set up. This increases server costs. Moreover, if there are many microservice nodes, it is quite normal to generate dozens of gigabytes of data a day. You also need to increase the cost of server disks if you want to keep them longer.

Therefore, the open source project of this introduction is TLog. The background of TLog is to create a lightweight logging tracing solution, covering the popular logging framework and RPC framework, and allowing users to access it at the least cost to solve the pain point of logging tracing in microservices.

Of course, a distributed tracking system is the ultimate solution, and if your company is already on a distributed tracking system, TLog is not for you.

TLog provides one of the simplest ways to solve the log tracing problem. It does not collect logs and does not require additional storage space. It just automatically tags your business logs and automatically generates traceids throughout the entire link of your microservice. And provides information about upstream and downstream nodes. Suitable for small to medium enterprises and corporate projects that want to quickly solve log tracing problems.

For this reason, I have adapted three logging frameworks for TLog to support automatic detection adaptation. Support for Dubbo, Dubbox, And Spring Cloud RPC frameworks. More importantly, your project may not need TLog for 10 minutes

TLog project and features

Project Address:Gitee.com/bryan31/TLo…

TLog has the following features:

• Lightweight microservice log tracking is accomplished by tagging the log
• Non – invasive design of business code, simple to use, 10 minutes to access
• Supports common log4J, LOG4J2, and Logback logging frameworks, and provides automatic detection and adaptation
• Support dubbo, Dubbox, SpringCloud RPC framework
• Supports the configuration of a user-defined template for log labels
• Almost no performance loss

How to Access quickly

TLog supports automatic assembly of SpringBoot. In the SpringBoot environment, you only need the following two steps to access it!

Rely on

<dependency>
  <groupId>com.yomahub</groupId>
  <artifactId>tlog-all-spring-boot-starter</artifactId>
  <version>1.0.0</version>
</dependency>
Copy the code

At present, the JAR package has been uploaded to the central repository and can be directly relied on

Log framework adaptation

Just add a line of code to your startup class that automatically detects and enhances the logging frameworks used by your project, currently supporting log4J, Log4J2, and Logback.

@SpringBootApplication
public class Runner {

    static{AspectLogEnhance.enhance(); }// Perform log enhancement to automatically determine the log framework

    public static void main(String[] args) { SpringApplication.run(Runner.class, args); }}Copy the code

It is important to note that since this is a Javassit implementation, bytecode enhancements are required before the JVM loads the classes for the corresponding logging framework. So I’m going to use static blocks here. However, this one-line auto-adaptation may not take effect in the following two cases:

1. The Springboot/Spring startup class is added to the log definition. This does not take effect because the classLoad has already loaded the log framework before loading the static block.

2. You started with an external container such as Tomcat/Jboss/Jetty (no impact on springBoot built-in containers)

For these two cases, TLog also provides a separate adaptation of the three logging frameworks. You can simply replace the layout class in the log configuration file, as described below

Adaptation of RPC framework

In the Springboot environment, TLog automatically detects which RPC framework you are using and automatically ADAPTS.

After completing the above 2 steps, the final result is as follows (for example, dubbo+log4j) :

Consumer side code:

Log print:

The 2020-09-16 18:12:56, 748 [WARN] [TLOG] to regenerate the traceId [7161457983341056] > > com. Yomahub. TLOG. Web. TLogWebInterceptor: 39 2020-09-16 18:12:56,763 [INFO] <7161457983341056> Logback-dubbox-Consumer: Invoke method sayHello,name=jack >> Com. Yomahub. Tlog. Example. Dubbox. Controller. DemoController: 22 2020-09-16 18:12:56, 763 [INFO] < 7161457983341056 > test log aaaa > > com. Yomahub. Tlog. Example. Dubbox. Controller. DemoController: 23 2020-09-16 18:12:56, 763 [INFO] < 7161457983341056 > The test log BBBB > > com. Yomahub. Tlog. Example. Dubbox. Controller. DemoController: 24Copy the code

The Provider code:

Log print:

2020-09-16 18:12:56,854 [INFO] <7161457983341056> Logback-dubbox-Provider: Invoke method sayHello,name=jack >> Com. Yomahub. Tlog. Example. Dubbo. Service. Impl. DemoServiceImpl: 15 2020-09-16 18:12:56, 854 [INFO] < 7161457983341056 > Test log CCCC > > com. Yomahub. Tlog. Example. Dubbo. Service. The impl. DemoServiceImpl: 16 2020-09-16 18:12:56, 854 [INFO] < 7161457983341056 > test log DDDD > > com. Yomahub. Tlog. Example. Dubbo. Service. The impl. DemoServiceImpl: 17Copy the code

As you can see, after simple access, each microservice has a globally unique traceId running through each request, which works for all log outputs, making it easy to locate the log chain for a particular request.

Change the log label format

TLog allows you to customize the format of the log tag. By default, TLog only prints traceId in the <$traceId> template. Of course, you can customize the template. You can also add other headers

All you need to do is to change the format of the tag in springBoot’s application.properties and print the template as you define it

tlog.pattern=[$preApp][$preIp][$traceId]
Copy the code

$preApp: Name of the upstream microservice node

$preIp: indicates the IP address of the upstream microservice

$traceId: indicates the globally unique traceId

A separate adaptation of the Log framework

If your automated logging probe fails or you use an external container, you will need to make simple changes to the logging framework configuration in your project.

Log4j configuration file enhancements

You just need to change the layout implementation class

The Log4J templates are pretty much the same for each company, and only XML examples are shown here

      
<! DOCTYPElog4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration>
    <appender name="stdout" class="org.apache.log4j.ConsoleAppender">
        <! -- Replace with AspectLog4jPatternLayout-->
        <layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
            <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m >> %c:%L%n"/>
        </layout>
    </appender>
    <appender name="fileout" class="org.apache.log4j.DailyRollingFileAppender">
        <param name="File" value="./logs/test.log"/>
        <! -- Replace with AspectLog4jPatternLayout-->
        <layout class="com.yomahub.tlog.core.enhance.log4j.AspectLog4jPatternLayout">
            <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss,SSS} [%p] %m >> %c:%L%n"/>
        </layout>
    </appender>
    <root>
        <priority value="info" />
        <appender-ref ref="stdout"/>
        <appender-ref ref="fileout"/>
    </root>
</log4j:configuration>

Copy the code

Logback profile enhancements

Change the implementation class of encoder or change the implementation class of layout

Here is an XML example:

      
<configuration debug="false">
    <property name="APP_NAME" value="logtest"/>
    <property name="LOG_HOME" value="./logs" />
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <! -- Replace with AspectLogbackEncoder-->
		<encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
			  <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
		</encoder>
    </appender>
    <appender name="FILE"  class="ch.qos.logback.core.rolling.RollingFileAppender">
        <File>${LOG_HOME}/${APP_NAME}.log</File>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <FileNamePattern>${LOG_HOME}/${APP_NAME}.log.%d{yyyy-MM-dd}.%i.log</FileNamePattern>
            <MaxHistory>30</MaxHistory>
            <maxFileSize>1000MB</maxFileSize>
        </rollingPolicy>
        <! -- Replace with AspectLogbackEncoder-->
        <encoder class="com.yomahub.tlog.core.enhance.logback.AspectLogbackEncoder">
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n</pattern>
        </encoder>
    </appender>

    <! -- Log output level -->
    <root level="INFO">
        <appender-ref ref="STDOUT" />
        <appender-ref ref="FILE" />
    </root>
</configuration>

Copy the code

Log4j2 configuration file enhancements

Log4J2 is implemented as a plug-in, log4J2 has the function of automatically scanning plug-ins. So you don’t need to make any changes to the configuration file for it to take effect.

After the language

TLog is essentially a logging framework with log tracing capabilities that support the RPC framework. It took me about a week to write the framework to make it the lightest log-tracking solution that users can access at the least cost. TLog supports not only SpringBoot but also non-SpringBoot projects. For details, see the documentation on the project home page.

I have also written a detailed sample project for TLog, covering almost all of the scenarios involved, which is also on the project home page.

If you encounter difficulties in logging tracking in your project, try TLog, I hope this open source framework can help you, open source is not easy, if you like, please help STAR. Please feel free to contact me if you have any questions.

Pay attention to the author

“Yuan tribe” is a adhere to do the original technology technology share number, I hope you can pay attention to me, I will be a practical and original technical article every week, accompany you to go together, accompany you to grow up together. Follow the public account reply tlog can join the group chat, I will patiently answer your every question in use, will also carry out long-term maintenance and iteration of this project.