This article is a series

To view the previous article, scan the QR code at the bottom of the article and click on the Past Review – Log series to view all relevant articles

An overview of

Hello everyone, I am back to update. Last time we discussed why we use logging framework, this time we get to the root of the problem, why we need logging, and how to use logging?

Most developers struggle with how to export logs, when to export logs, and whether anyone will read them. Let’s think outside the developer’s box: Who needs logs? What kinds of logs are there? What do logs output? How do I output logs?

Who needs logs?

  • developersDevelopers need to output some variables during development to facilitate debugging. The correct way to output variables is to use logsSystem.outTo output, accidentally published online, will be the project manager pain criticism); Secondly, online problems are difficult to replay, and users’ statements are generally distorted. Moreover, many users delete the app and close the web page and leave when they find bugs
  • O&m personnel maintain the entire system most of the time. Logs help O&M personnel learn about the system status (many O&M systems are also connected to logs). If o&M personnel find abnormal information in logs, they can also inform the development team for troubleshooting
  • Operation personnel Yes, that is, operation personnel, such as the conversion rate of e-commerce, the completion rate of video websites, ordinary PV data, etc., can be counted through logs. With the popularity of big data technology, this part of the log accounts for an increasingly high proportion
  • Security personnel Although most enterprises do not attach importance to security, security can also be warned through logs, such as a user suddenly large amount of money transfer, for example, the database suddenly appeared a large number of unconditional paging search (drag) and so on

What kinds of logs are there?

  • Debug logs are used by developers to develop or trace problems online.
  • Diagnostic logs are used by O&M personnel to monitor the system and security personnel to analyze and alert.
  • Buried logs are generally used for operational decision analysis, but also for microservice invocation link tracing (o&M, debugging).
  • Audit logs are similar to diagnostic logs in that diagnostic logs favor O&M and audit logs favor security.

What do logs output?

Note: Log levels are explained below

  • The debug log
    • The DEBUG or TRACE level, such as method call parameters and network connection details, is used by developers to DEBUG programs
    • INFO level: important but risk-free information, such as environment initialization, parameters, environment cleanup, scheduled task execution, and remote call success
    • WARN level, possible risks and does not affect the system continue errors, such as system parameter configuration is not correct, the parameters of the user request is not correct (to specific output parameters convenient screen), or the performance of certain consumption scenarios, such as request on for too long a time, a SQL execution for more than two seconds, some third party call fails, If branches that are unlikely to be run, etc
    • The ERROR level is used to print stack information about program errors. It should not be used to output information other than program problems.throw) the
  • Diagnostic logs are generally output at the INFO level, request response time, and memory usage. These logs are opened when the monitoring system is connected online. You are advised to export them to an independent file in JSON format for easy analysis by external tools
  • Buried log services can be customized as required. For example, the conversion rate mentioned above can be generated when the user pays, and the playback rate can be generated once after the user plays. Generally, the logs can be generated at the INFO level
  • Most audit logs have a WARN or INFO level and can be exported for sensitive operations, such as login, transfer and payment, authorization cancellation, and deletion. You are advised to export audit logs to independent files in JSON format for convenient analysis by external tools

The output of debug logs is customized by the developer. The output of the other three logs should be customized based on actual service requirements.

Other points to note about logging

  1. Online logging should be as careful as possible. Think: can this location help troubleshoot the problem? Is the output relevant to troubleshooting? Is the output enough to troubleshoot the problem? Do a lot of output necessary information, not much output useless information (slow down the system, drown useful information)
  2. Super SessionId and RequestId should generate a super SessionId for each login of each user, no matter it is a single application or micro-service architecture, to facilitate the tracking and differentiation of a user. RequestId, which generates one RequestId per request and is used to trace a request. Microservices can also be used for link tracing
  3. Output logs in a single line. Otherwise, it is inconvenient to read and analyze logs by third-party systems or tools
  4. Within the company should develop a general log specification, including the format of the log, the variable name (hump, underline), separator (” = “or” : “, etc.), when the output (such as the provisions of the third party output INFO before and after log), the company’s log specification should be constantly optimized, adjust, find best practices of the company

OK, so much for the theory, let’s go back to the technical side.

Using the concept of

To learn how to use logging frameworks, you need to understand simple concepts such as Logger, Appenders, Layout, logging Level and Level Inheritance.

Logger (Log instance)

To output a log, call org.slf4j.LoggerFactory#getLogger(java.lang.class
) or org.slf4j.loggerFactory #getLogger(java.lang.string) generates an instance of logging, and the same parameters share the same instance.

Appenders

Log output, logback predefined output to console, files, Socket server, MySQL, PostgreSQL, Oracle and other databases, JMS and UNIX Syslog system calls, etc., can be used through configuration file configuration. Of course, the only two we use are console and file.

Layout

It controls the log output format. As described in the preceding section, log information is automatically output, for example: Date, thread, method name, etc. “you can use Layout to control, the actual use is very simple, write a Layout format to define the expression (pattern), using the method similar to Java SimpleDateFomat.

The level of logging

RFC 5424 (Page 11) specifies eight log levels, but SLF4j only defines five log levels: ERROR, WARN, INFO, DEBUG, and TRACE. The higher the configuration level, the less the log output is, as shown in the following figure

We saw five points on the slider is corresponding to the five levels, slide indicator can move around, indicator as a cut-off point, output indicator on the left side of the can, can output on the right side, left and right adjustment indicator can log output, slipped right can all output, slipped on the left side of the can reduce the output, so whether can completely shut down output? The answer is yes, the configuration file can also be configured to ALL and OFF, corresponding to ALL (equivalent to TRACE) and OFF, respectively.

Level of inheritance

With log levels in mind, let’s consider the following scenario:

  • INFO logs are generated for some important services and WARN logs are generated for others. At the same time, logs of all libraries and frameworks are disabled

Logback and LOG4j support the configuration of log instances. Now the problem is solved, but there is a new problem. If all the logs on the network are output INFO level, do you need to configure them one by one? Logback and log4j use the same single-root inheritance model, Object in Java and ROOT in log4j.

With inheritance, we just need to adjust ROOT to INFO level and refine the logger instance level for our business to meet most scenarios.

Codding of actual combat

Q: How many steps to put the elephant in the refrigerator? There are three steps: 1, import dependencies, 2, code output logs, 3, adjust configuration files. Step 1 has been mentioned before. If you haven’t seen it, please go to the official account for the past review, and go directly to Step 2.

Step 2

If Lombok is used in your project, you can annotate the class directly with @slf4J to get the logging instance, Static final org.slf4j.Logger Logger = loggerFactory.getLogger (testlog.class); To get the log instance

The log output method is as follows:

logger.trace("A TRACE Message");
logger.debug("A DEBUG Message");
logger.info("An INFO Message");
logger.warn("A WARN Message");
logger.error("An ERROR Message");
Copy the code

One caveat here is to use parameter placeholders instead of manually concatenating strings, as follows

String level = "Trace";
/ /
logger.trace("A " + level + " Message");
// The right thing to do
logger.trace("A {} Message", level);
Copy the code

Doing this can improve the efficiency, if not the output logs, the first kind of circumstance also can cause loss of performance string concatenation, the second will not have this problem (alibaba Java development manual (huashan version) stated here has a problem, a placeholder is more efficient because string processing delay as far as possible, if you don’t need the output log is not dealt with, The next principle analysis will be expanded), and we don’t need if (logger.istraceEnabled ()) to determine (performance loss is not high, but the code is much better).

Step 3

The configuration files need to distinguish logback from log4j2. The two frameworks have different but similar configuration files.

Logback Configuration file location

Logback supports both XML and Groovy scripting. Logback finds the location of configuration files as follows (how to change the location will be covered in a future article)

  1. Logback tries to find a file named logback-test.xml in the classpath.
  2. If no such file is found, Logback tries to find a file named logback.groovy in the classpath.
  3. If no such file is found, it checks the file logback.xml in the classpath.
  4. If it can’t find this kind of file by looking for the file meta-inf \ services \ ch qos. Logback. Classic. Spi. Configurator, If you have this file and contents are com. Qos. Logback. Classic. Spi. The Configurator implementation class full class name, loading the implementation class directly.
  5. If none of the above methods are successful, the Logback automatically configits itself using BasicConfigurator, which prints logs to the console.

XML in the Classpath root directory.

Logback configuration file writing rules

Logback configuration files fall into three main categories. One or more appenders define output locations (different file locations, network or database). One or more loggers for detailing the output levels and locations of different loggers; A ROOT, a special logger, is used to configure the ROOT Logger.

Let’s take a look at the following configuration file example

<?xml version="1.0" encoding="UTF-8"? >
<configuration scan="false" debug="false">

    <! Define log file storage location -->
    <property name="LOG_PATH" value="/var/log"/>

    <property name="CONSOLE_LOG_PATTERN"
              value="% d {HH: mm: ss. The SSS} % 5 level 10.10 thread] [% % 30.30 logger {29} \ \ \ % 4 l) - % MSG % n"/>
    <! -- Console output -->
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            <charset>utf-8</charset>
        </encoder>
    </appender>

    <! -- File log format (print log, no line number) -->
    <property name="FILE_LOG_PATTERN"
              value="% d {MM - dd yyyy - HH: MM: ss. The SSS} % 5 level 10.10 thread] [% % 30.30 logger {29} - % MSG % n"/>

    <appender name="FILE_ALL" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <! -- The path and name of the log file being recorded -->
        <file>${LOG_PATH}/log.log</file>
        <! -- Logger scroll policy, by date, by size -->
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <! -- YYYY-MM-DD by day -->
            <fileNamePattern>${LOG_PATH}/log-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <! -- Single file maximum 50M -->
            <maxFileSize>50MB</maxFileSize>
            <! -- Up to 5G of disk space, 500 files (no more than 5G in total) -->
            <maxHistory>500</maxHistory>
            <totalSizeCap>5GB</totalSizeCap>
        </rollingPolicy>
        <! -- Append log -->
        <append>true</append>
        <! -- Log file format -->
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>${FILE_LOG_PATTERN}</pattern>
            <charset>utf-8</charset>
        </encoder>
    </appender>

    <! -- Log output level STDOUT: console; FILE_ALL: file - >
    <root level="warn">
        <appender-ref ref="STDOUT"/>
    </root>
    <logger name="druid.sql" level="warn" additivity="true"/>
    <logger name="druid.sql.ResultSet" level="warn" additivity="true"/>
    <logger name="com.alibaba.druid.pool.DruidDataSource" level="debug" additivity="true">
        <appender-ref ref="FILE_ALL"/>
    </logger>
</configuration>
Copy the code

The configuration file defines two appenders, one that outputs to the console and the other to a file and automatically scrolls. Note that the property tag defines a variable and can be referenced with ${XXX}. CONSOLE_LOG_PATTERN and FILE_LOG_PATTERN define the console and file print format. SimpleDateFomat, which is written in a similar way to Java, will not be expanded here

  • Logback: logback. Qos. Ch/manual/layo…
  • Log4j:logging.apache.org/log4j/2.x/m…

Log4j2 Configuration file location

Log4j2 supports configuration files in XML, JSON, YAML, or Properties format. The search methods are as follows:

  1. Check the “log4j.configurationFile” system property and, if so, try to load the configuration using a ConfigurationFactory that matches the file extension.
  2. If the system property is not set, the property ConfigurationFactory looks for log4j2-test.properties in the classpath.
  3. If no such file is found, YAML ConfigurationFactory looks for log4j2-test. YAML or log4j2-test.yml in the classpath.
  4. If no such file is found, the JSON ConfigurationFactory looks for log4j2-test. JSON or log4j2-test.jsn in the classpath.
  5. If such a file is not found, the XML ConfigurationFactory looks for log4j2-test.xml in the classpath.
  6. If the test file is not found, the attribute ConfigurationFactory looks for log4j2.properties on the classpath.
  7. If the properties file is not found, YAML ConfigurationFactory looks for log4j2.yaml or log4j2.yml on the classpath.
  8. If the YAML file cannot be found, the JSON ConfigurationFactory looks for log4j2.json or log4j2.jsn on the classpath.
  9. If the JSON file cannot be found, the XML ConfigurationFactory tries to find log4j2.xml on the classpath.
  10. If the configuration file is not found, it is automatically configured using DefaultConfiguration to output the log to the console.

Instead of reading the longer text, just put log4j2.xml in the Classpath root directory

Log4j2 configuration file

Log4j is also a Logger and Appender configuration item. Log4j also has a ROOT special Logger. Appender supports more output locations than Logback, such as Kafka, Cassandra, Flume, etc.

<?xml version="1.0" encoding="UTF-8"? >
<Configuration status="debug" strict="true">
    <! ${XXX} -->
    <Properties>
        <Property name="baseDir">logs</Property>
    </Properties>

    <! Appenders specifies the output position -->
    <Appenders>
        <! -- Log scrolling? {date: yyyY-MM} : yyyy-mm: month ->
        <RollingFile name="RollingFile" fileName="${baseDir}/log.log"
                     filePattern="${baseDir}/? {date:yyyy-MM}/log-%d{yyyy-MM-dd-HH}-%i.log.gz">
            <! -- Log format -->
            <PatternLayout pattern="%d %p %c{1.} [%t] %m%n"/>
            <Policies>
                <! -- Time scroll (scroll directory by month, scroll file by hour) -->
                <TimeBasedTriggeringPolicy/>
                <! -- File size scroll (over 250M in 1 hour, forced scroll once) -->
                <SizeBasedTriggeringPolicy size="250 MB"/>
            </Policies>
            <! -- Maximum 100 files per day -->
            <DefaultRolloverStrategy max="100">
                <! If the total number of files is less than 100GB and the number of files is less than 10, the file will not be deleted.
                <Delete basePath="${baseDir}" maxDepth="2">
                    <IfFileName glob="*/app-*.log.gz">
                        <IfLastModified age="30d">
                            <IfAny>
                                <IfAccumulatedFileSize exceeds="100 GB"/>
                                <IfAccumulatedFileCount exceeds="10"/>
                            </IfAny>
                        </IfLastModified>
                    </IfFileName>
                </Delete>
            </DefaultRolloverStrategy>
        </RollingFile>
    </Appenders>

    <Loggers>
        <! -- Multiple loggers -->
        <Logger name="org.apache.logging.log4j.test2" level="debug" additivity="false">
            <AppenderRef ref="RollingFile"/>
        </Logger>

        <! -- a ROOT -->
        <Root level="trace">
            <AppenderRef ref="STDOUT"/>
        </Root>
    </Loggers>
</Configuration>
Copy the code

The log4j2 and logback configuration files are written in a similar way, even if additivity=”true” leads to repeated log output. After all, log4j1 and logback are both the works of Ceki.

conclusion

Thanks to the efforts of Ceki, there is little difference in logging usage (Logback and Log4j2, Google open source the Fluent logging framework Flogger in April 2018, Slf4j will also be supported in version 2.0, while Log4j2 lags behind again. But I think log4j2 has a lot of advantages, so stay tuned for more.) In addition, each company has different application scenarios, so you should follow the unified specifications of the company.

This article is more about basic usage, but the following articles will cover the comparison, principle and extended logging framework, please look forward to it.

Code related to this article

If you think your writing is good, ask for attention, ask for praise, ask for forwarding, if you have any questions or mistakes in the article, welcome to leave a comment and discuss.

Scan the public account to get the first updates

reference

Logging.apache.org/log4j/2.x/m…

Logback. Qos. Ch/manual/conf…