Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.
This article has participated in the “Digitalstar Project” and won a creative gift package to challenge the creative incentive money.
How to split response time?
In performance analysis, response time splitting is often a starting point. Because in a performance scenario, if the system reaches a bottleneck for whatever reason, then increasing the stress is bound to cause the response time to rise until it times out. After identifying the bottleneck, we need to find out where the problem is. The response time you see on the pressure tool goes through every system on the back end. So, when the response time gets longer, we need to know at what stage it gets longer. Let’s look at this picture.
SpringCloud typically has an architecture like this, and the split time should be fairly clear:
graph LR
A[jmeter] --> B[ingress]
B --> C[gateway]
C --> D[service]
D --> E[cache]
E --> F[DB]
Nginx-ingress = request_time; upstream_response_time = request_response_time;
172.16.106.116 - - [23/Feb/2021:13:52:21 +0800] "POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-" 8.659 8.660
Copy the codeThe last two columns are preceded by 8.659s of request time and followed by 8.660s of back-end response time.
At the same time, we can check the time consumed by Reactor Netty at gateway:
[Feb/ Feb/ 201:13:52:14 +0800]"POST/mall - order/order/generateOrder HTTP / 1.1" 500 133 8201 52 ms
Copy the codeIn the last column, it takes 52 ms.
SpringBoot Service = SpringBoot Service = SpringBoot Service
[Feb/ Feb/ 201:13:52:21 +0800]"POST/order/generateOrder HTTP / 1.1"500, 144,"-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" 504 ms 502 ms
Copy the codeThe request time consumed 504 ms and the response time consumed 502 ms.
In this example, the response time is explained step by step. Of course, if you are the following case, then one by one it is more hard, you can change another way.
Graph LR A service [A] - [service] B B B - C [service] C C - D service [D] - [service] E E DKnowing how long each system took, we can use a link monitoring tool to break down the time. For example, break it up like this:
How to enable Access Logs?
1, Nginx Ingress Access Log
We need to customize the log-format format in configMap:
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time
Copy the codeThe fields are described as follows:
- $remote_ADDR – IP address of the client that initiated the request. The RECORDED IP address is not necessarily the IP address of the real user client. It may be the public network mapping address of the private client or the proxy server address.
$remote_user– User name of the remote client, which is used to record the user name provided by the visitor for authentication, such as the user name for logging in to BaiduzuozeweiIf there is no login, it is blank.[$time_local]– The time the request was received (the time and time zone of the access, for example18/Jul/2018:17:00:01 +0800Time information last"+ 0800"Indicates that the server time zone is 8 hours after UTC.- “$request” – Request line from the client (request URI and HTTP protocol, this is the most useful information in PV logging to record what kind of request the server received)
- $status – The server returns the status code of the client, such as success is 200.
- $body_bytes_sent – The size of the body content of the file sent to the client, excluding the size of the response header (you can sum this value from each record of the log to give a rough estimate of server throughput)
- “$http_referer” – record the page from which the link was accessed (request header Referer content)
- “$http_user_agent” – Client browser information (request header user-agent content)
"$ http_x_forwarded_for"– Real IP address of the client. Usually, the Web server is placed after the reverse proxy. In this case, the client IP address cannot be obtained$remote_addThe IP address obtained is that of the reverse proxy server. The reverse proxy server can be added to the HTTP header for forwarding requestsx_forwarded_for** information to record the IP address of the original client and the server address requested by the original client.- $request_time – The total time of the request, in seconds (including the time to receive the request data from the client, the time to respond to the back-end application, and the time to send the response data to the client (excluding the time to write the log)
- $upstream_response_time – The response time, in seconds, to the upstream during a request (from the time the connection is established at the back end until data is received and the connection is closed)
The resulting log is as follows:
172.16.106.116 - - [23/Feb/2021:05:52:21 +0000] "POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"8.659 8.660 172.16.106.61 - - [23/Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 202:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"10.792 10.792 172.16.106.99 - - [Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"9.654 9.654 172.16.106.99 - - [Feb/ Feb/ 202:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 201:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"[Feb/ Feb/ 202:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-"20.058 20.058 172.16.106.61 - - [Feb/ 202:05:52:21 +0000]"POST/mall - order/order/generateOrder HTTP / 1.1" 499 0 "-" "Apache HttpClient / 4.5.12 (Java / 1.8.0 comes with _261)" "-" 8.943 8.943
Copy the codeReactor Netty Access Log
Spring – the cloud – gateway is based on the application of webflux, so to enable the Reactor Netty access log, you need to set up – Dreactor.net ty.. HTTP server accessLogEnabled = true.
Note:
It must be a Java System property, not a SpringBoot property.
We can configure the logging system to have a separate access log file. The following example creates a Logback configuration:
<appender name="accessLog" class="ch.qos.logback.core.FileAppender">
<file>access_log.log</file>
<encoder>
<pattern>%msg%n</pattern>
</encoder>
</appender>
<appender name="async" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="accessLog" />
</appender>
<logger name="reactor.netty.http.server.AccessLog" level="INFO" additivity="false">
<appender-ref ref="async"/>
</logger>
Copy the codeThe resulting log is as follows:
[Feb/ Feb/ 201:11:02:09 +0800]"GET /mall-portal/home/content? PageNum = 4 & pageSize = 1 HTTP / 1.1"50 ms 50 ms 50 - - [Feb/ Feb/ 201:50 + 50]"GET/mall - member/sso/info HTTP / 1.1"150 ms 150 ms 150 - - [Feb/ Feb/ 201:50 + 150]"POST/mall - member/sso/login HTTP / 1.1"150 ms 150 ms 150 - - [Feb/ Feb/ 201:50 + 150]"GET/mall - member/member/address/list of HTTP / 1.1"200 200 388 ms - - [Feb/ Feb/ 201:11:02:09 +0800]"POST/mall - cart/cart/addNew HTTP / 1.1"200 50 50 ms - - [Feb/ Feb/ 201:11:02:09 +0800]"GET/mall - cart/cart/list of HTTP / 1.1"200 2639 ms - - [Feb/ Feb/ 201:11:02:09 +0800]"POST/mall - cart/cart/addNew HTTP / 1.1"200 50 50 ms - - [Feb/ Feb/ 201:11:02:09 +0800]"GET/mall - order/order/detail / 1732755 HTTP / 1.1"500 150 ms 150 - - [23/Feb/ 201:11:02:09 + 150]"GET /mall-portal/home/content? PageNum = 4 & pageSize = 1 HTTP / 1.1" 200 3689 8201 15 ms
Copy the codeReactor-netty-0.8.5. release-sources.jar HTTP / / reactor/netty/server/AccessLog. We can understand the Java source code is how to achieve:
final class AccessLog {
static final Logger log = Loggers.getLogger("reactor.netty.http.server.AccessLog");
static final DateTimeFormatter DATE_TIME_FORMATTER =
DateTimeFormatter.ofPattern("dd/MMM/yyyy:HH:mm:ss Z", Locale.US);
static final String COMMON_LOG_FORMAT =
"{} - {} [{}] \"{} {} {}\" {} {} {} {} ms";
static final String MISSING = "-";
final String zonedDateTime;
String address;
CharSequence method;
CharSequence uri;
String protocol;
String user = MISSING;
CharSequence status;
long contentLength;
boolean chunked;
long startTime = System.currentTimeMillis();
int port;
AccessLog() {
this.zonedDateTime = ZonedDateTime.now().format(DATE_TIME_FORMATTER);
}
AccessLog address(String address) {
this.address = Objects.requireNonNull(address, "address");
return this;
}
AccessLog port(int port) {
this.port = port;
return this;
}
AccessLog method(CharSequence method) {
this.method = Objects.requireNonNull(method, "method");
return this;
}
AccessLog uri(CharSequence uri) {
this.uri = Objects.requireNonNull(uri, "uri");
return this;
}
AccessLog protocol(String protocol) {
this.protocol = Objects.requireNonNull(protocol, "protocol");
return this;
}
AccessLog status(CharSequence status) {
this.status = Objects.requireNonNull(status, "status");
return this;
}
AccessLog contentLength(long contentLength) {
this.contentLength = contentLength;
return this;
}
AccessLog increaseContentLength(long contentLength) {
if (chunked) {
this.contentLength += contentLength;
}
return this;
}
AccessLog chunked(boolean chunked) {
this.chunked = chunked;
return this;
}
long duration(a) {
return System.currentTimeMillis() - startTime;
}
void log(a) {
if (log.isInfoEnabled()) {
log.info(COMMON_LOG_FORMAT, address, user, zonedDateTime,
method, uri, protocol, status, (contentLength > -1? contentLength : MISSING), port, duration()); }}}Copy the codeAccessLog directly outputs logs through logger in the format of COMMON_LOG_FORMAT({} – {} [{}] “{} {} {}” {} {} {} ms). Address, User, zonedDateTime, Method, URI, Protocol, Status, contentLength, port, duration.
SpringBoot Access Log
To enable Access Log, we need to configure:
server:
tomcat:
basedir: /var/tmp
background-processor-delay: 30
redirect-context-root: true
uri-encoding: UTF-8
accesslog:
enabled: true
buffered: true
directory: ./log
file-date-format: .yyyy-MM-dd
pattern: '%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i" %D ms %F ms'
prefix: access_log
rename-on-rotate: false
request-attributes-enabled: false
rotate: true
suffix: .log
Copy the codeThe fields are described as follows:
- %h – IP address of the client that initiates the request. The RECORDED IP address is not necessarily the IP address of the real user client. It may be the public network mapping address of the private client or the proxy server address.
- % L – RFC 1413 identifier of the client (see). Only clients that implement the RFC 1413 specification can provide this information.
%u– User name of the remote client, which is used to record the user name provided by the visitor for authentication, such as the user name for logging in to BaiduzuozeweiIf there is no login, it is blank.%t– The time the request was received (the time and time zone of the access, for example18/Jul/2018:17:00:01 +0800Time information last"+ 0800"Indicates that the server time zone is 8 hours after UTC.- %{x-real_IP} I – Indicates the real IP address of the client
- %r – Request line from the client (request URI and HTTP protocol, this is the most useful information in PV logging to record what kind of request the server received)
- %>s – The server returns the client status code, such as success 200.
- %b – The size of the body content of the file sent to the client, excluding the size of the response header (you can sum this value from each record of the log to give a rough estimate of server throughput)
- %{Referer} I – From which page the link was accessed (request header Referer content)
- %D – Time to process the request in milliseconds
- %F – The response time of the client browser information submission, in milliseconds
The resulting log is as follows:
[root@svc-mall-admin-5dbb7467d8-q4wlq log]# ll
total 1736
-rw-r--r-- 1 root root 23967 Feb 21 00:00 access_log.2021-02-20.log
-rw-r--r-- 1 root root 668261 Feb 22 00:00 access_log.2021-02-21.log
-rw-r--r-- 1 root root 673517 Feb 23 00:00 access_log.2021-02-22.log
-rw-r--r-- 1 root root 385773 Feb 23 13:52 access_log.2021-02-23.log
[root@svc-mall-admin-5dbb7467d8-q4wlq log]# tail -f access_log.2021-02-23.log [Feb/ Feb/ 201:13:50:12 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"29 ms ms - - [27 /Feb/ 201:13:50:32 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"24 ms 30 ms - - [Feb/ Feb/ 201:13:50:52 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"23 ms ms - - [Feb/ Feb/ 201:13:51:12 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"18 ms 18 ms - - [Feb/ Feb/ 201:13:51:32 +0800]"GET HTTP / 1.1 / physical/info" 200 12 "-" "ReactorNetty / 0.9.7 RELEASE"4 ms 4 ms - - [Feb/ Feb/ 201:13:51:32 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"20 ms 20 ms - - [Feb/ Feb/ 201:13:51:52 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"24 ms 24 ms - - [Feb/ Feb/ 201:13:52:12 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"30 ms 30 ms - - [Feb/ Feb/ 201:13:52:32 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE"18 ms 18 ms - - [Feb/ Feb/ 201:13:52:52 +0800]"GET HTTP / 1.1 / physical health"200, 845,"-" "ReactorNetty / 0.9.7 RELEASE" 21 ms 20 ms
Copy the codeHow to improve log performance?
Mainly through the following two aspects:
- Increase log output to file LEVEL
- During the pressure test, you can set the service log level to Error
- Asynchronous log output reduces disk I/OS and improves disk performance
1. How to configure asynchronous logging?
The SpringBoot application comes with Logback and SLF4J dependencies, so the focus is on writing configuration files, and we don’t have to worry about what dependencies we need to introduce, or log dependency conflicts. The Logback framework loads a configuration file named logback-spring or logback in the classpath by default. Store all logs in one file. The file size grows with the running of the application and it is difficult to troubleshoot problems. The correct method is to separate error logs from other logs and store logs of different levels according to the time period.
For example:
<! DOCTYPEconfiguration>
<configuration>
<! -- Reference default log configuration -->
<include resource="org/springframework/boot/logging/logback/defaults.xml"/>
<! -- Use the default console log output implementation -->
<include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
<! -- Application name -->
<springProperty scope="context" name="APP_NAME" source="spring.application.name" defaultValue="springBoot"/>
<! -- Log file save path -->
<property name="LOG_FILE_PATH" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}}/logs}"/>
<! -- Log output to console -->
<appender name="CONSOLE-LOG" class="ch.qos.logback.core.ConsoleAppender">
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>[%d{yyyy-MM-dd' 'HH:mm:ss.sss}] [%C] [%t] [%L] [%-5p] %m%n</pattern>
</layout>
</appender>
<! -- Collect logs except for error level -->
<appender name="INFO-LOG" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>DENY</onMatch>
<onMismatch>ACCEPT</onMismatch>
</filter>
<encoder>
<! -- Set to default file log format -->
<pattern>${FILE_LOG_PATTERN}</pattern>
<charset>UTF-8</charset>
</encoder>
<! -- Scroll strategy -->
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<! -- -- -- > path
<fileNamePattern>${LOG_INFO_HOME}//%d.log</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
<! -- Scroll strategy -->
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<! -- Set path and file naming format -->
<fileNamePattern>${LOG_FILE_PATH}/info/${APP_NAME}-%d{yyyy-MM-dd}-%i.log</fileNamePattern>
<! -- set the size of the log file to 10M-->
<maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
<! -- Log file retention days (default: 30 days)
<maxHistory>${LOG_FILE_MAX_HISTORY:-30}</maxHistory>
</rollingPolicy>
</appender>
<! --ERROR log output to file -->
<appender name="FILE_ERROR"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<! ERROR level log -->
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<encoder>
<! -- Set to default file log format -->
<pattern>${FILE_LOG_PATTERN}</pattern>
<charset>UTF-8</charset>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<! -- Set the file naming format -->
<fileNamePattern>${LOG_FILE_PATH}/error/${APP_NAME}-%d{yyyy-MM-dd}-%i.log</fileNamePattern>
<! -- set the size of the log file to 10M-->
<maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
<! -- Log file retention days (default: 30 days)
<maxHistory>${LOG_FILE_MAX_HISTORY:-30}</maxHistory>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="CONSOLE-LOG" />
<appender-ref ref="INFO-LOG" />
<appender-ref ref="ERROR-LOG" />
</root>
</configuration>
Copy the codePart of label description:
<root>Tag, a mandatory tag that specifies the most basic log output level<appender-ref>Tag, add append
<append>Tag that specifies the collection policy for logs- The name property specifies the appender name
- The class attribute specifies the output strategy. There are usually two types, console output and file output, which is to persist the log. ConsoleAppender outputs logs to the console
<filter>Tag that you can use to specify a filtering policy<level>The tag specifies the type of filtering
<encoder>Tag, using the tag under<pattern>The tag specifies the log output format<rollingPolicy>The tag specifies a collection policy, such as time-based collection<fileNamePattern>The tag specifies the address where logs are generated and saved. By this configuration, logs are collected by day
The generated logs are as follows:
[root@svc-mall-admin-5dbb7467d8-q4wlq logs]# cd error/
[root@svc-mall-admin-5dbb7467d8-q4wlq error]# lltotal 1520 -rw-r--r-- 1 root root 0 Feb 20 23:36 mall-admin-2021-02-20-0.log -rw-r--r-- 1 root root 501172 Feb 21 20:37 mall-admin-2021-02-21-0.log -rw-r--r-- 1 root root 34632 Feb 22 11:27 mall-admin-2021-02-22-0.log -rw-r--r-- 1 root root 1012807 Feb 23 06:25 mall-admin-2021-02-23-0.logCopy the codeThe previous log configuration is based on synchronization. Disk I/O is performed every time logs are output to a file. Asynchronous log writing prevents disk I/OS that block threads and cause unnecessary performance loss. Writing logs asynchronously is as simple as adding an appender based on writing logs asynchronously and pointing to the configured appender.
As follows:
<! -- Asynchronous output -->
<appender name="ASYNC-INFO" class="ch.qos.logback.classic.AsyncAppender">
<! -- Do not lose logs. By default, TRACT, DEBUG, and INFO logs are discarded if 80% of the queue is full.
<discardingThreshold>0</discardingThreshold>
<! Change the default queue depth, which affects performance. The default is 256 -->
<queueSize>256</queueSize>
<! Add additional appenders, one at most -->
<appender-ref ref="INFO-LOG"/>
</appender>
<appender name="ASYNC-ERROR" class="ch.qos.logback.classic.AsyncAppender">
<! -- Do not lose logs. By default, TRACT, DEBUG, and INFO logs are discarded if 80% of the queue is full.
<discardingThreshold>0</discardingThreshold>
<! Change the default queue depth, which affects performance. The default is 256 -->
<queueSize>256</queueSize>
<! Add additional appenders, one at most -->
<appender-ref ref="ERROR-LOG"/>
</appender>
Copy the codeRelated information:
- Github.com/zuozewei/bl…
References:
- [1] : nginx – configmap – the resource
- [2] : Reactor Netty Access Logs
- [3] : Talk about reactor-Netty AccessLog
- [4] : Performance Test Actual Combat lecture 30