Surveillance, it’s always been a topic for a long time. In addition to system monitoring, there is another one that is often overlooked, and today we will talk about this overlooked business monitoring.

Monitor what?

As a developer, you don’t just need to develop features, you need to know enough about the product or module you are responsible for, you need to pay attention to it all the time, and you need to have the feeling of first love.

For example, I want to know the success rate of placing an order, the average time it takes to place an order, and the number of failed orders due to lack of inventory.

Armed with this business metrics information, you will know what the product you are responsible for is doing and what improvements you need to make.

As for which metrics to monitor, follow your business. The purpose is to know the service status and trigger alarms at certain times.

implementation

Data is recorded based on buried points, recorded to local disk files, and then collected and stored by unified log collection program, statistical display and alarm.

The reason for the buried point approach is that you need to know the execution result of each business interface, whether it succeeded or failed, and why it failed. We define a business code for each exception, from which we can know the cause of the exception.

If you use logs to output data, separate the data files of service logs from those of system logs. Otherwise, it is difficult to collect and parse.

Logback can be configured with a separate appender. I only output the business data I buried in Json format.

<appender name="BIZ_FILE_APPENDER" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <encoder>
        <pattern>%m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}</pattern>
    </encoder>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        <FileNamePattern>${LOG_HOME_PATH_BIZ}/${APP_NAME}.%d{yyyy-MM-dd}.log</FileNamePattern>
        <MaxHistory>7</MaxHistory>
    </rollingPolicy>
    <filter class="ch.qos.logback.classic.filter.LevelFilter">
        <onMatch>ACCEPT</onMatch>
        <onMismatch>DENY</onMismatch>
    </filter>
</appender>
Copy the code

Log file contents:

{"biz":"confirm","bizCode":500,"domain":"storeOrder","bizId":"86081301","execTime":3,"count":"1","storeId":"1","userId": "740942"} {"biz":"confirm","bizCode":500,"domain":"storeOrder","bizId":"86081301","errorMessage":"/ by zero","execTime":14,"count":"1","storeId":"1","userId":"740942"}Copy the code

Annotations buried point

BizLog annotations are used for service monitoring burying points. The specific configuration fields are described as follows:

  • Domain: a domain, such as order and pay
  • Biz: Business, such as createOrder,cancelOrder
  • BizId: business ID, such as orderId
  • In addition: Extended information

In addition, you can configure multiple AdditionFields to obtain the data you want to buried. The data can be obtained from the request parameters or the corresponding parameters. For example, after the order is successfully created, the order ID will be returned, so bizId is obtained from the response parameters.

Code buried point

Some scenarios (payment callbacks, JOB tasks, etc.) are not good for annotation burying. We can do this by using code burying.

Log collection

We use the log service of Ali Cloud. We can directly configure Logtail to complete the collection work. The interface operation is very convenient.

Indicators show

Buried raw data

Chart statistical display

Indicators of the alarm

Now that the data is collected, it’s easy to focus on what metrics to look for and when to alert. For example, if placing an order frequently fails within a minute, you can set the number of failed alarms >N to trigger an alarm. When the alarm is received, you can immediately find out why placing an order fails.

About the author: Yin Jihuan, a simple technology lover, the author of Spring Cloud Micro-service – Full stack Technology and Case Analysis, Spring Cloud Micro-service Introduction combat and Progress, the founder of the public account Monkey World.