Usage posture of the log library

What is a log

The so-called Log is a time-ordered collection of some operations on the object specified by the system and its operation results. Log files are log files. Log files record information about the interaction between the system and users of the system. Log files are data collection methods that automatically capture the type, content, or time of the interaction between people and system terminals.

Log is used to record, user operations, system status, error information and other contents of the file, is an important part of a software system. A good log specification is of great significance to the analysis of system running status and the solution of on-line problems.

The log specification

In the development of software to print logs, need to pay attention to some problems, examples may not be complete, you can baidu related articles or view the literature at the bottom of the article:

Log important functions as well as possible.
Do not print unnecessary logs. Excessive unnecessary logs make it difficult to analyze logs.
Logs must be classified into levels such as DEBUG, WARN, INFO, and error.
It is best to print error stack information when unhandled errors are caught

Log library used by the Go language

Go language standard library provides a log library for us, in addition to this there are many log libraries, such as Logrus, Glog, Logx, Uber’s Zap and so on, for example, ZAP has many advantages:

A high performance
Rich Configuration Items
Multiple log levels
Support the hooks
Rich toolkit
The Sugar log is provided
Multiple log printing formats
.

Simple to use

package main

import (
	"errors"
	"go.uber.org/zap"
)

var logger *zap.Logger

func init(a) {
	logger, _ = zap.NewProduction()
}
func main(a) {
	logger.Error(
		"My name is baobao",
		zap.String("from"."Hulun Buir"),
		zap.Error(errors.New("no good")))

	logger.Info("Worked in the Ministry of national development of China!",
		zap.String("key"."Eat 🍚"),
		zap.String("key"."Sleep 😴"))
	defer logger.Sync()
}
Copy the code

Kratos log library principle analysis

When communicating with Teacher Tony privately about the realization concept of log library, Teacher Tony said: Since there are so many log libraries and they are easy to use, the following issues should be considered in Kratos’ log:

Unified log interface design

Organizational Structured logs

It also needs to be used at friendly log levels

Supports interconnection between multiple output sources, such as log-agent or 3rd log library

Kratos log library, not mandatory specific implementation mode, only provides adapters, users can realize the log function, only need to realize kratos/log Logger interface to access their favorite log system.

Kratos’ log library, in the design stage, referred to many excellent open source projects and the log system implementation of large factories, experienced many changes before presenting to everyone.

The composition of the log library

The kratos log library consists of the following files

Level. go defines log levels
Log. go Log core
Helper. Go log helper
Value. go implements dynamic values

Source code analysis

The core of kratos’ log library is log.go code is very simple and conforms to kratos’ design philosophy. Logger Interface is declared in log.go. Users only need to implement the interface to introduce their own log implementation, the main code is as follows:

log.go

package log

import (
	"context"
	"log"
)

var (
	// DefaultLogger is default logger.
	DefaultLogger Logger = NewStdLogger(log.Writer())
)

// Logger interface, which is used to implement the custom log library.
type Logger interface {
	Log(level Level, keyvals ...interface{}) error
}

type logger struct {
	logs      []Logger / / logger array
	prefix    []interface{} // Some default printed values, such as Valuer bound With With
	hasValuer bool // Whether Valuer is included
	ctx       context.Context / / context
}

func (c *logger) Log(level Level, keyvals ...interface{}) error {
	kvs := make([]interface{}, 0.len(c.prefix)+len(keyvals))
	kvs = append(kvs, c.prefix...)
        // Check whether valuer exists
	if c.hasValuer {
                / / bind valuer
		bindValues(c.ctx, kvs)
	}
	kvs = append(kvs, keyvals...)
        // Iterate over logs, calling all loggers to print the logs.
	for _, l := range c.logs {
		iferr := l.Log(level, kvs...) ; err ! =nil {
			return err
		}
	}
	return nil
}

// With with logger fields.
func With(l Logger, kv ...interface{}) Logger {
	// Determine whether the incoming logger can be declared as *logger
	if c, ok := l.(*logger); ok {
		// Make a c.prefix + keyvals interface array
		kvs := make([]interface{}, 0.len(c.prefix)+len(kv))
		// Process the printed content
		kvs = append(kvs, kv...)
		kvs = append(kvs, c.prefix...)
		// containsValuer() is used to check whether valuer exists in the KVS
		return &logger{
			logs:      c.logs,
			prefix:    kvs,
			hasValuer: containsValuer(kvs),
			ctx:       c.ctx,
		}
	}
	return &logger{logs: []Logger{l}, prefix: kv, hasValuer: containsValuer(kv)}
}

// WithContext binds CTX. Note that CTX must be non-null
func WithContext(ctx context.Context, l Logger) Logger {
	if c, ok := l.(*logger); ok {
		return &logger{
			logs:      c.logs,
			prefix:    c.prefix,
			hasValuer: c.hasValuer,
			ctx:       ctx,
		}
	}
	return &logger{logs: []Logger{l}, ctx: ctx}
}

// Multiple loggers are printed simultaneously
func MultiLogger(logs ... Logger) Logger {
	return &logger{logs: logs}
}

Copy the code

value.go

// Returns valuer.
func Value(ctx context.Context, v interface{}) interface{} {
	if v, ok := v.(Valuer); ok {
		return v(ctx)
	}
	return v
}

/ /... Omit some of the built-in Valuer implementations

/ / bind valuer
func bindValues(ctx context.Context, keyvals []interface{}) {
	for i := 1; i < len(keyvals); i += 2 {
		if v, ok := keyvals[i].(Valuer); ok {
			keyvals[i] = v(ctx)
		}
	}
}

// Whether valuer is included
func containsValuer(keyvals []interface{}) bool {
	for i := 1; i < len(keyvals); i += 2 {
		if _, ok := keyvals[i].(Valuer); ok {
			return true}}return false
}
Copy the code

helper.go

package log

import (
	"context"
	"fmt"
)

// Helper is a logger helper.
type Helper struct {
	logger Logger
}

// Create a Logger Helper instance
func NewHelper(logger Logger) *Helper {
	return &Helper{
		logger: logger,
	}
}

// With WithContext() returns a helper class containing a log of CTX, with some defined methods for printing the log by level
func (h *Helper) WithContext(ctx context.Context) *Helper {
	return &Helper{
		logger: WithContext(ctx, h.logger),
	}
}

func (h *Helper) Log(level Level, keyvals ...interface{}) {
	h.logger.Log(level, keyvals...)
}

func (h *Helper) Debug(a ...interface{}) {
	h.logger.Log(LevelDebug, "msg", fmt.Sprint(a...))
}

func (h *Helper) Debugf(format string, a ...interface{}) {
	h.logger.Log(LevelDebug, "msg", fmt.Sprintf(format, a...))
}

/ /... Omit some duplicate methods

Copy the code

Understand the invocation logic through unit tests

func TestInfo(t *testing.T) {
	logger := DefaultLogger
	logger = With(logger, "ts", DefaultTimestamp, "caller", DefaultCaller)
	logger.Log(LevelInfo, "key1"."value1")}Copy the code

The single test first declares a logger, using the default DefaultLogger
Call the With() function in log.go, passing in logger and two dynamic values, DefaultTimestamp and DefaultCaller.
The With method is called to see if the argument l type can be converted to *logger
If it can be converted, assign the incoming KV to logger.prefix, then call containsValuer() in value.go to determine if there is a Valuer value in the incoming KV, and assign the result to context.hasvaluer. Finally, the Logger object is returned
Logger{logs: []Logger{l}, prefix: kv, hasValuer: containsValuer(kv)}
The logger struct’s Log method is then called when the Log is printed
Log()The method is preallocated firstkeyvalsSpace, and then judgehasValuerIf fortrueThe callvaluer.goIn thebindValuer()And the incomingctxAnd then to getvaluerThe value of theif v, ok := v.(Valuer); ok { return v() }

8. Run logger.logs for the last time to print logs

Method of use

Use Logger to print logs

logger := log.DefaultLogger
logger.Log(LevelInfo, "key1"."value1")
Copy the code

Use the Helper to print logs

log := log.NewHelper(DefaultLogger)
log.Debug("test debug")
log.Info("test info")
log.Warn("test warn")
log.Error("test error")
Copy the code

Use the valuer

logger := DefaultLogger
logger = With(logger, "ts", DefaultTimestamp, "caller", DefaultCaller)
logger.Log(LevelInfo, "msg"."helloworld")
Copy the code

Print multiple loggers simultaneously

out := log.NewStdLogger(os.Stdout)
err := log.NewStdLogger(os.Stderr)
l := log.With(MultiLogger(out, err))
l.Log(LevelInfo, "msg"."test")
Copy the code

Use the context

logger := log.With(NewStdLogger(os.Stdout),
	"trace", Trace(),
)
log := log.NewHelper(logger)
ctx := context.WithValue(context.Background(), "trace_id"."2233")
log.WithContext(ctx).Info("got trace!")
Copy the code

Use filter to filter logs

You can use log.newfilter () to filter some fields in logs that should not be printed in plain text, such as password.

Filter logs by level

l := log.NewHelper(log.NewFilter(log.DefaultLogger, log.FilterLevel(log.LevelWarn)))
l.Log(LevelDebug, "msg1"."te1st debug")
l.Debug("test debug")
l.Debugf("test %s"."debug")
l.Debugw("log"."test debug")
l.Warn("warn log")
Copy the code

Logs are filtered by key

l := log.NewHelper(log.NewFilter(log.DefaultLogger, log.FilterKey("password")))
l.Debugw("password"."123456")
Copy the code

Logs are filtered by value

l := log.NewHelper(log.NewFilter(log.DefaultLogger, log.FilterValue("kratos")))
l.Debugw("name"."kratos")
Copy the code

Filter logs by hook Func

l := log.NewHelper(log.NewFilter(log.DefaultLogger, log.FilterFunc(testFilterFunc)))
l.Debug("debug level")
l.Infow("password"."123456")
func testFilterFunc(level Level, keyvals ...interface{}) bool {
	if level == LevelWarn {
		return true
	}
	for i := 0; i < len(keyvals); i++ {
		if keyvals[i] == "password" {
			keyvals[i+1] = "* * *"}}return false
}
Copy the code

Using Zap to implement the logging interface of Kratos

The code is very simple, only less than 100 lines of code, just for your reference.

implementation

// kratos/examples/log/zap.go
package logger

import (
	"fmt"
        "os"
        
	"github.com/go-kratos/kratos/v2/log"
	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
	"gopkg.in/natefinch/lumberjack.v2"
)

var _ log.Logger = (*ZapLogger)(nil)

// Zap structure
type ZapLogger struct {
	log  *zap.Logger
	Sync func(a) error
}

// Create a ZapLogger instance
func NewZapLogger(encoder zapcore.EncoderConfig, level zap.AtomicLevel, opts ... zap.Option) *ZapLogger {
	writeSyncer := getLogWriter()
	/ / set zapcore
	core := zapcore.NewCore(
		zapcore.NewConsoleEncoder(encoder),
		zapcore.NewMultiWriteSyncer(
			zapcore.AddSync(os.Stdout),
		), level)
	// New a * zap.logger
	zapLogger := zap.New(core, opts...)
	return &ZapLogger{log: zapLogger, Sync: zapLogger.Sync}
}

The // Log method implements the Logger interface in kratos/ Log /log.go
func (l *ZapLogger) Log(level log.Level, keyvals ...interface{}) error {
	if len(keyvals) == 0 || len(keyvals)%2! =0{
        	l.log.Warn(fmt.Sprint("Keyvalues must appear in pairs: ", keyvals))
		return nil
	}
	// Use zap.Field when KV is passed in
	var data []zap.Field
	for i := 0; i < len(keyvals); i += 2 {
		data = append(data, zap.Any(fmt.Sprint(keyvals[i]), fmt.Sprint(keyvals[i+1)))}switch level {
	case log.LevelDebug:
		l.log.Debug("", data...)
	case log.LevelInfo:
		l.log.Info("", data...)
	case log.LevelWarn:
		l.log.Warn("", data...)
	case log.LevelError:
		l.log.Error("", data...)
	}
	return nil
}

// Log automatic cutting, using lumberjack implementation
func getLogWriter(a) zapcore.WriteSyncer {
	lumberJackLogger := &lumberjack.Logger{
		Filename:   "./test.log",
		MaxSize:    10,
		MaxBackups: 5,
		MaxAge:     30,
		Compress:   false,}return zapcore.AddSync(lumberJackLogger)
}

Copy the code

Method of use

// kratos/examples/log/zap_test.go
package logger

import (
	"testing"

	"github.com/go-kratos/kratos/v2/log"
	"go.uber.org/zap"
	"go.uber.org/zap/zapcore"
)

func TestZapLogger(t *testing.T) {
	encoder := zapcore.EncoderConfig{
		TimeKey:        "t",
		LevelKey:       "level",
		NameKey:        "logger",
		CallerKey:      "caller",
		MessageKey:     "msg",
		StacktraceKey:  "stack",
		EncodeTime:     zapcore.ISO8601TimeEncoder,
		LineEnding:     zapcore.DefaultLineEnding,
		EncodeLevel:    zapcore.LowercaseLevelEncoder,
		EncodeDuration: zapcore.SecondsDurationEncoder,
		EncodeCaller:   zapcore.FullCallerEncoder,
	}
	logger := NewZapLogger(
		encoder,
		zap.NewAtomicLevelAt(zapcore.DebugLevel),
		zap.AddStacktrace(
			zap.NewAtomicLevelAt(zapcore.ErrorLevel)),
		zap.AddCallerSkip(2),
		zap.Development(),
	)
	zlog := log.NewHelper(logger)
	zlog.Infow("name"."Advanced Go language")
	defer logger.Sync()
}
Copy the code

reference

Discussion issue on log libraries
Uber log library Zap Uber/Zap
Log cutover library LumberJack
Zap-based log demo log example

What is a log

The log specification

Log library used by the Go language

Simple to use

Kratos log library principle analysis

The composition of the log library

Source code analysis

log.go

value.go

helper.go

Understand the invocation logic through unit tests

Method of use

Use Logger to print logs

Use the Helper to print logs

Use the valuer

Print multiple loggers simultaneously

Use the context

Use filter to filter logs

Filter logs by level

Logs are filtered by key

Logs are filtered by value

Filter logs by hook Func

Using Zap to implement the logging interface of Kratos

implementation

Method of use

reference

Related Posts

Reading Notes on The Midas Touch: The Secrets of Web Design for Visitors

ElasticSearch does not support grouped queries?

K8s practice | how to solve the problem of the security isolation of multi-tenant cluster?