preface

Recently, I have been studying and upgrading the architecture of the company’s project. After continuous learning and trial and error, I finally determined a set of highly available architecture system based on K8S. In the next few issues, I will share the construction process and notes of this architecture system in the form of a series of articles, please look forward to it!

Due to the expansion of cluster and distributed scale, it is more and more necessary to monitor microservice links and collect logs. Therefore, after screening some schemes, SkyWalking is found to meet our expectations perfectly, with good implementation of link tracking and log collection.

SkyWalking profile

SkyWalking is an APM (Application Monitoring) system designed to be a microservice, cloud-native, container-based architecture. It mainly contains the following core functions

  1. Index analysis of services, running instances and APIS
  2. Link detection, checking for slow services and apis
  3. Monitor the infrastructure (VM, network, disk, database)
  4. Alerts if the threshold is exceeded
  5. , etc.

Open source: Apache/Skywalking

Official website: Apache SkyWalking

Integrate SpringCloud SkyWalking

1. Set up SkyWalking service

Before SkyWalking is used for link tracking and log collection, a set of SkyWalking services need to be set up first, and then the operating status and logs of SpringCloud can be sent to SkyWalking through agent for analysis and display.

There are many ways to build SkyWalking. Here I would like to introduce two docker-compose (non-highly available, quick start, convenient for testing and learning) and K8S (highly available, production level).

Docker – way to compose

The installation of Docker and Docker-compose is not the focus of this article, so you can check it yourself if necessary.

The following action starts three containers

  1. elasticsearchServes as a storage for Skywalking, storing link and log data, etc
  2. oapObservability Analysis Platform for data receipt and Analysis
  3. uiDisplay data on the Web
Create a directory to save the configuration file
mkdir -p /data/docker/admin/skywalking
Switch to the directory you just created
cd /data/docker/admin/skywalking
# Save the following docker-comemage. yml file to this directory
vi docker-compose.yml
# Pull image and start
docker-compose up -d
# check log
docker-compose logs -f
Copy the code

docker-compose.yml

version: '3.8'
services:
  elasticsearch:
    image: Docker. Elastic. Co/elasticsearch/elasticsearch: 7.14.1
    container_name: elasticsearch
    restart: always
    ports:
      - 9200: 9200
    healthcheck:
      test: ["CMD-SHELL"."curl --silent --fail localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - TZ=Asia/Shanghai
    ulimits:
      memlock:
        soft: - 1
        hard: - 1
  oap:
    image: Apache/skywalking - oap - server: 8.7.0 - es7
    container_name: oap
    depends_on:
      - elasticsearch
    links:
      - elasticsearch
    restart: always
    ports:
      - 11800: 11800
      - 12800: 12800
    healthcheck:
      test: ["CMD-SHELL"."/skywalking/bin/swctl"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    environment:
      TZ: Asia/Shanghai
      SW_STORAGE: elasticsearch7
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200
  ui:
    image: Apache/skywalking - UI: 8.7.0
    container_name: ui
    depends_on:
      - oap
    links:
      - oap
    restart: always
    ports:
      - 8088: 8080
    environment:
      TZ: Asia/Shanghai
      SW_OAP_ADDRESS: http://oap:12800

Copy the code

After startup, the browser access service IP :8080

k8s

Waiting for updates..

2. Download the Agent package

Click on the link to download, Skywalking – APM-8.7

For other versions, see the Apache archive site to find the corresponding version of the.tar.gz suffix package and download

Run the tar -zxvf apache-Skywalking -apm-8.7.0.tar.gz command or software to decompress the decompression

3. The Java command uses code to start the JAR package

Springcloud/SpringBoot is usually started by using java-jar xxx.jar. We just need to add the -JavaAgent parameter, as follows

The user-defined service name can be changed to the application name such as Lemes-Auth, the service IP is the IP address of the SkyWalking service constructed in the first step, and port 11800 is the port of the oAP container started

Jar =agent.service_name= user-defined service name, Collector. backend_service= service IP address: 11800-jar xx.jarCopy the code

After executing the command to start, access the following interface, and you can see the accessed link and invocation link in step 1 service IP :8080.

4. Enable log collection

This article mainly to log4j2 to introduce, other similar, you can find online tutorials. SpringCloud’s log4j2 integration is not the focus of this article, so Google it yourself.

Introduction of depend on

To enable log collection, you must add dependencies, as follows

<dependency>
    <groupId>org.apache.skywalking</groupId>
    <artifactId>apm-toolkit-log4j-2.x</artifactId>
    <version>8.7.0</version>
</dependency>
Copy the code

Modify log4j2. XML

To modify log4j2.xml, add the following two key points

  • add%traceIdTo print traceid
  • The statement GRPCLogClientAppender

The full content is as follows


      
<! -- Log levels and priorities: OFF > FATAL > ERROR > WARN > INFO > DEBUG > TRACE > ALL -->
<! -- Status after Configuration. This is used to set the internal output of log4j2. You do not need to set this. It can be set to OFF(OFF) or Error(output only Error messages). -->
<! --monitorInterval: Log4j automatically detects changes to configuration files and reconfigures itself, sets the interval -->
<configuration status="WARN" monitorInterval="30">

    <Properties>
        <Property name="log.path">logs/lemes-auth</Property>
        <Property name="logging.lemes.pattern">
            %d{yyyy-MM-dd HH:mm:ss.SSS} [%-5level] [%traceId] [%logger{50}.%M:%L] - %msg%n
        </Property>
    </Properties>

    <Appenders>
        <! -- Output console log configuration -->
        <Console name="Console" target="SYSTEM_OUT">
            <! -- Console only outputs messages of level and above (onMatch).
            <ThresholdFilter level="debug" onMatch="ACCEPT" onMismatch="DENY"/>
            <! -- Output log format -->
            <PatternLayout pattern="${logging.lemes.pattern}"/>
        </Console>

        <RollingRandomAccessFile name="debugRollingFile" fileName="${log.path}/debug.log"
                                 filePattern="${log.path}/debug/$${date:yyyy-MM}/debug.%d{yyyy-MM-dd}-%i.log.gz">
            <ThresholdFilter level="debug" onMatch="ACCEPT" onMismatch="DENY"/>
            <PatternLayout charset="UTF-8" pattern="${logging.lemes.pattern}"/>
            <Policies>
                <TimeBasedTriggeringPolicy interval="1"/>
                <SizeBasedTriggeringPolicy size="100 MB"/>
            </Policies>
            <DefaultRolloverStrategy max="30"/>
        </RollingRandomAccessFile>

        <GRPCLogClientAppender name="grpc-log">
            <PatternLayout pattern="${logging.lemes.pattern}"/>
        </GRPCLogClientAppender>
    </Appenders>
    <Loggers>
        <! -- ALL < TRACE < DEBUG < INFO < WARN < ERROR < FATAL < OFF -->
        <Logger name="com.lenovo.lemes" level="debug"/>
        <Logger name="org.apache.kafka" level="warn"/>
        <Root level="info">
            <AppenderRef ref="Console"/>
            <AppenderRef ref="debugRollingFile"/>
            <AppenderRef ref="grpc-log"/>
        </Root>
    </Loggers>
</configuration>

Copy the code

Log reporting is declared in the start command

Add parameters for reporting logs to the Agent in the previous step Plugin. The toolkit. The GRPC. Reporter. Server_host = service IP, plugin. The toolkit. The GRPC. Reporter. Server_port = 11800

Complete the following

java Jar =agent.service_name= user-defined service name,collector.backend_service= service IP address :11800,plugin.tool Kit. Log. GRPC. Reporter. Server_host = IP service, the plugin. The toolkit. The GRPC. Reporter. Server_port = 11800 - jar xx. The jarCopy the code

Log Collection Effect

In this case, the startup log displays traceid. N/A indicates A non-request log, and the log with traceid indicates an API request log

You can see our reported logs in Skywalking

Important: SkyWalking can view all logs of the current request in link tracing (different instances/modules)

5. Compatible with spring – cloud – gateway

After the link is set up, the traceId of the Gateway module is inconsistent with that of the service module.

This is because SkyWalking’s spring-cloud-gateway support is not by default, Therefore, copy agent/optional-plugins/ apM-spring-cloud-gateway-2.1.x-plugin-8.7.0.jar to agent/plugins and restart the plugins.

The last

These two features on SkyWalking were already very powerful and helped us optimize our programs, monitor for problems, and alert people to them. Log collection also solves the difficulty in querying logs in a large-scale distributed cluster.

SkyWalking also supports VM, browser, AND K8S monitoring, which will be updated as and when it is implemented.