Build the wheel: Write a journaling framework
Intro
Logging framework has a lot of, such as log4net nlog/serilog/Microsoft extensions, logging, etc., how to do when switching logging framework need not modify the code, you just need to switch different loggingProvider is ok, The lowest cost to reduce the cost of switching logging framework, in this consideration to write a logging framework for different logging framework to write an adaptation, need to use what logging framework, configure it, business code does not need to change.
V0
The initial log strongly depends on the log4net, log4net is I use the first log frame, so for a long time are using it for logging in, but because is strongly dependent on, is to want to change the logging framework with very afflictive, want to change a lot of code, do not accord with the basic principles of closed, so there will be a first version of the journal.
V1
The first version of logging refers to the implementation of Microsoft’s logging framework, which is roughly structured as follows:
public interface ILogHelperLogFactory
{
ILogger CreateLogger(string categoryName);
bool AddProvider(ILogHelperProvider provider);
}
public interface ILogHelperLogger
{
bool IsEnabled(LogHelperLogLevel logLevel);
void Log(LogHelperLogLevel logLevel, Exception exception, string message);
}
public enum LogHelperLogLevel
{
/// <summary>
/// All logging levels
/// </summary>
All = 0,
/// <summary>
/// A trace logging level
/// </summary>
Trace = 1,
/// <summary>
/// A debug logging level
/// </summary>
Debug = 2,
/// <summary>
/// A info logging level
/// </summary>
Info = 4,
/// <summary>
/// A warn logging level
/// </summary>
Warn = 8,
/// <summary>
/// An error logging level
/// </summary>
Error = 16,
/// <summary>
/// A fatal logging level
/// </summary>
Fatal = 32,
/// <summary>
/// None
/// </summary>
None = 64
}
public interface ILogHelperProvider
{
ILogHelperLogger CreateLogger(string categoryName);
}
Copy the codeLogger. Info/ logger.Error/logger.Error/logger.Info/ logger.Error/logger.Info/ logger.Error
public static void Log(this ILogHelperLogger logger, LogHelperLevel loggerLevel, string msg) => logger.Log(loggerLevel, null, msg);
#region Info
public static void Info(this ILogHelperLogger logger, string msg, params object[] parameters)
{
if (parameters == null || parameters.Length == 0)
{
logger.Log(LogHelperLevel.Info, msg);
}
else
{
logger.Log(LogHelperLevel.Info, null, msg.FormatWith(parameters));
}
}
public static void Info(this ILogHelperLogger logger, Exception ex, string msg) => logger.Log(LogHelperLevel.Info, ex, msg);
public static void Info(this ILogHelperLogger logger, Exception ex) => logger.Log(LogHelperLevel.Info, ex, ex? .Message);
#endregion Info
/ /... The others are similar and I won't go into detail here
Copy the codeTo customize logging, implement an ILogHelperProvider, which implements an ILogHelperLogger, Log4net can implement a Log4NetLogHelperProvider, which only needs to implement the corresponding ILogHelperProvider when changing to another logging framework, but it is still weak in terms of functionality
If you don’t want to record Debug logs, for example, or Error logs in a Logger, it’s a bit of a hassle. You can only do this with log4Net. Increased LoggingFilter can target the Provider/Logger/LogLevel/Exception to set the filter, filter does not need to record the log, it is also a reference for Microsoft’s log frame filter, but the implementation is different, Interested partners can do their own in-depth research.
V2
Version V2 added AddFilter method to ILogFactory interface, defined as follows:
/// <summary>
/// Add logs filter
/// </summary>
/// <param name="filterFunc">filterFunc, logProviderType/categoryName/Exception, whether to write log</param>
bool AddFilter(Func<Type, string, LogHelperLogLevel, Exception, bool> filterFunc);
Copy the codeThen we define some extension methods to make it easier to use:
public static ILogHelperFactory WithMinimumLevel(this ILogHelperFactory logHelperFactory, LogHelperLevel logLevel)
{
return logHelperFactory.WithFilter(level => level >= logLevel);
}
public static ILogHelperFactory WithFilter(this ILogHelperFactory logHelperFactory, Func<LogHelperLevel, bool> filterFunc)
{
logHelperFactory.AddFilter((type, categoryName, logLevel, exception) => filterFunc.Invoke(logLevel));
return logHelperFactory;
}
public static ILogHelperFactory WithFilter(this ILogHelperFactory logHelperFactory, Func<string, LogHelperLevel, bool> filterFunc)
{
logHelperFactory.AddFilter((type, categoryName, logLevel, exception) => filterFunc.Invoke(categoryName, logLevel));
return logHelperFactory;
}
public static ILogHelperFactory WithFilter(this ILogHelperFactory logHelperFactory, Func<Type, string, LogHelperLevel, bool> filterFunc)
{
logHelperFactory.AddFilter((type, categoryName, logLevel, exception) => filterFunc.Invoke(type, categoryName, logLevel));
return logHelperFactory;
}
public static ILogHelperFactory WithFilter(this ILogHelperFactory logHelperFactory, Func<Type, string, LogHelperLevel, Exception, bool> filterFunc)
{
logHelperFactory.AddFilter(filterFunc);
return logHelperFactory;
}
Copy the codeWe only want to define the Logger Filter and the Provider Filter. We don’t need to use all parameters. The logging Filter is now implemented and Serilog has been used for logging for some time. I felt that some of Serilog’s designs were excellent and elegant, so I decided to use some of Serilog’s designs in my own logging framework. For example:
-
The Serilog extension is called Sink, the log output place, Serilog has a custom Sink, it is very easy to implement only one interface, do not need to implement a Logger, in this regard, Serilog is better than Microsoft logging framework. And for LogEvent log is more convenient for batch operation, has the need to can know about the PeriodBatching Serilog https://github.com/serilog/serilog-sinks-periodicbatching
-
Serilog can customize some Enricher to enrich the log content, such as the log request context, log environment, or some fixed attribute information
-
MessageTemplate is a similar concept in Microsoft’s logging framework, but it’s not obvious, and I rarely used it before Serilog, The Microsoft logging framework can write logger.LogInfo(“hello {name}”,”world”) as the first argument to MessageTemplate or its internal Format
With so many benefits, I decided to incorporate these features into my logging framework
V3
The introduction of LoggingEvent
LogHelperLoggingEvent = Serilog LogEvent = Serilog LogEvent
public class LogHelperLoggingEvent : ICloneable
{
public string CategoryName { get ; set ; }
public DateTimeOffset DateTime { get ; set ; }
public string MessageTemplate { get ; set ; }
public string Message { get ; set ; }
public Exception Exception { get ; set ; }
public LogHelperLogLevel LogLevel { get ; set ; }
public Dictionary
Properties { get ; set ; }
,>
public LogHelperLoggingEvent Copy => ( LogHelperLoggingEvent)Clone();
public object Clone()
{
var newEvent = (LogHelperLoggingEvent )MemberwiseClone();
if ( Properties ! = null)
{
newEvent .Properties = new Dictionary< string, object>();
foreach ( var property in Properties)
{
newEvent .Properties[ property.Key ] = property. Value;
}
}
return newEvent;
}
}
Copy the codeIn Event, a dictionary of Properties is defined to enrich the log content, and ICloneable interface is implemented to facilitate the copying of objects. In order to strongly type, a Copy method is added to return a strongly typed object
Transform LogProvider
To reduce the complexity of extending an ILogProvider, we will simplify the ILogProvider by simply logging like the Serilog Sink, regardless of whether a Logger is created or not
The definition after transformation is as follows:
public interface ILogHelperProvider
{
Task Log(LogHelperLoggingEvent loggingEvent);
}
Copy the code(This returns a Task, maybe void is enough, depending on your needs)
So when you implement the LogProvider, you only need to implement the interface, and you don’t need to implement a Logger
Increase the Enricher
Enricher definition:
public interface ILogHelperLoggingEnricher
{
void Enrich(LogHelperLoggingEvent loggingEvent);
}
Copy the codeThere is a built-in PropertyEnricher for adding simple properties
internal class PropertyLoggingEnricher : ILogHelperLoggingEnricher
{
private readonly string _propertyName ;
private readonly Func< LogHelperLoggingEvent, object > _propertyValueFactory;
private readonly bool _overwrite ;
private readonly Func< LogHelperLoggingEvent, bool > _logPropertyPredict = null;
public PropertyLoggingEnricher (string propertyName , object propertyValue, bool overwrite = false) : this (propertyName , (loggingEvent ) => propertyValue , overwrite )
{
}
public PropertyLoggingEnricher (string propertyName , Func
propertyValueFactory ,
,>
bool overwrite = false) : this(propertyName , propertyValueFactory, null, overwrite )
{
}
public PropertyLoggingEnricher (string propertyName , Func
propertyValueFactory , Func
logPropertyPredict,
,>
bool overwrite = false)
{
_propertyName = propertyName;
_propertyValueFactory = propertyValueFactory;
_logPropertyPredict = logPropertyPredict;
_overwrite = overwrite;
}
public void Enrich( LogHelperLoggingEvent loggingEvent )
{
if ( _logPropertyPredict? .Invoke (loggingEvent) ! = false)
{
loggingEvent .AddProperty( _propertyName, _propertyValueFactory , _overwrite);
}
}
}
Copy the codeAdd an AddEnricher method to ILogFactory
/// <summary>
/// add log enricher
/// </summary>
/// <param name="enricher">log enricher</param>
/// <returns></returns>
bool AddEnricher(ILogHelperLoggingEnricher enricher);
Copy the codeThis way we can enrich the Properties in LoggingEvent with Enricher when logging
To facilitate Property manipulation, we have added some extension methods:
public static ILogHelperFactory WithEnricher<TEnricher>(this ILogHelperFactory logHelperFactory,
TEnricher enricher) where TEnricher : ILogHelperLoggingEnricher
{
logHelperFactory.AddEnricher(enricher);
return logHelperFactory;
}
public static ILogHelperFactory WithEnricher<TEnricher>(this ILogHelperFactory logHelperFactory) where TEnricher : ILogHelperLoggingEnricher, new()
{
logHelperFactory.AddEnricher(new TEnricher());
return logHelperFactory;
}
public static ILogHelperFactory EnrichWithProperty(this ILogHelperFactory logHelperFactory, string propertyName, object value, bool overwrite = false)
{
logHelperFactory.AddEnricher(new PropertyLoggingEnricher(propertyName, value, overwrite));
return logHelperFactory;
}
public static ILogHelperFactory EnrichWithProperty(this ILogHelperFactory logHelperFactory, string propertyName, Func<LogHelperLoggingEvent> valueFactory, bool overwrite = false)
{
logHelperFactory.AddEnricher(new PropertyLoggingEnricher(propertyName, valueFactory, overwrite));
return logHelperFactory;
}
public static ILogHelperFactory EnrichWithProperty(this ILogHelperFactory logHelperFactory, string propertyName, object value, Func<LogHelperLoggingEvent, bool> predict, bool overwrite = false)
{
logHelperFactory.AddEnricher(new PropertyLoggingEnricher(propertyName, e => value, predict, overwrite));
return logHelperFactory;
}
public static ILogHelperFactory EnrichWithProperty(this ILogHelperFactory logHelperFactory, string propertyName, Func<LogHelperLoggingEvent, object> valueFactory, Func<LogHelperLoggingEvent, bool> predict, bool overwrite = false)
{
logHelperFactory.AddEnricher(new PropertyLoggingEnricher(propertyName, valueFactory, predict, overwrite));
return logHelperFactory;
}
Copy the codeMessageTemplate
MessageTemplate has been added from LoggingEvent above, so we introduced the logging formatting of Microsoft Logging Framework, converting MessageTemplate and Parameters to Message and Properties. Specific reference https://github.com/WeihanLi/WeihanLi.Common/blob/276cc49cfda511f9b7b3bb8344ee52441c4a3b23/src/WeihanLi.Common/Logging/Lo ggingFormatter.cs
internal struct FormattedLogValue
{
public string Msg { get; set; }
public Dictionary<string, object> Values { get; set; }
public FormattedLogValue(string msg, Dictionary<string, object> values)
{
Msg = msg;
Values = values;
}
}
internal static class LoggingFormatter
{
public static FormattedLogValue Format(string msgTemplate, object[] values)
{
if (values == null || values.Length == 0)
return new FormattedLogValue(msgTemplate, null);
var formatter = new LogValuesFormatter(msgTemplate);
var msg = formatter.Format(values);
var dic = formatter.GetValues(values)
.ToDictionary(x => x.Key, x => x.Value);
return new FormattedLogValue(msg, dic);
}
}
Copy the codeThis way we can support messageTemplate and modify our Logger
public interface ILogHelperLogger
{
void Log(LogHelperLogLevel logLevel, Exception exception, string messageTemplate, params object[] parameters);
bool IsEnabled(LogHelperLogLevel logLevel);
}
Copy the codeUnlike the above, we added parameters
Let’s update our extension method. The above extension method is directly formatted in string.Format
public static void Info(this ILogHelperLogger logger, string msg, params object[] parameters)
{
logger.Log(LogHelperLogLevel.Info, null, msg, parameters);
}
public static void Info(this ILogHelperLogger logger, Exception ex, string msg) => logger.Log(LogHelperLogLevel.Info, ex, msg);
public static void Info(this ILogHelperLogger logger, Exception ex) => logger.Log(LogHelperLogLevel.Info, ex, ex? .Message);
Copy the codeAt this point, the function is basically complete, but from the perspective of API, I feel that the current ILogFactory is too heavy, these AddProvider/ AddEnricher/ AddFilter should be the internal properties of ILogFactory, through configuration to complete. Should not be its interface method, so there is the next version
V4
This version mainly introduces LoggingBuilder, through LoggingBuilder to configure the internal LogFactory required Provider/ Enricher/ Filter, Their original configuration method and extension method are changed to ILogHelperLoggingBuilder
public interface ILogHelperLoggingBuilder
{
/// <summary>
/// Adds an ILogHelperProvider to the logging system.
/// </summary>
/// <param name="provider">The ILogHelperProvider.</param>
bool AddProvider (ILogHelperProvider provider);
/// <summary>
/// add log enricher
/// </summary>
/// <param name="enricher">log enricher</param>
/// <returns></returns>
bool AddEnricher (ILogHelperLoggingEnricher enricher);
/// <summary>
/// Add logs filter
/// </summary>
/// <param name="filterFunc">filterFunc, logProviderType/categoryName/Exception, whether to write log</param>
bool AddFilter (Func< Type, string, LogHelperLogLevel, Exception , bool > filterFunc );
///// <summary>
///// config period batching
///// </summary>
///// <param name="period">period</param>
///// <param name="batchSize">batchSize</param>
//void PeriodBatchingConfig(TimeSpan period, int batchSize);
/// <summary>
/// Build for LogFactory
/// </summary>
/// <returns></returns>
ILogHelperFactory Build ();
}
Copy the codeAdd logging configuration:
public static class LogHelper
{
private static ILogHelperFactory LogFactory { get; private set; } = NullLogHelperFactory.Instance;
public static void ConfigureLogging(Action<ILogHelperLoggingBuilder> configureAction)
{
var loggingBuilder = new LogHelperLoggingBuilder();
configureAction? .Invoke(loggingBuilder);
LogFactory = loggingBuilder.Build();
}
public static ILogHelperLogger GetLogger<T>() => LogFactory.GetLogger(typeof(T));
public static ILogHelperLogger GetLogger(Type type) => LogFactory.GetLogger(type);
public static ILogHelperLogger GetLogger(string categoryName)
{
return LogFactory.CreateLogger(categoryName);
}
}
Copy the codeFinal use:
internal class LoggingTest
{
private static readonly ILogHelperLogger Logger = LogHelper.GetLogger<LoggingTest>();
public static void MainTest()
{
var abc = "1233";
LogHelper.ConfigureLogging(builder =>
{
builder
.AddLog4Net()
//.AddSerilog(loggerConfig => loggerConfig.WriteTo.Console())
.WithMinimumLevel(LogHelperLogLevel.Info)
.WithFilter((category, level) => level > LogHelperLogLevel.Error && category.StartsWith("System"))
.EnrichWithProperty("Entry0", ApplicationHelper.ApplicationName)
.EnrichWithProperty("Entry1", ApplicationHelper.ApplicationName, E => e.logLevel >= logHelperLoglevel. Error)// Property is added only when LogLevel is Error or higher
;
});
Logger.Debug("12333 {abc}", abc);
Logger.Trace("122334334");
Logger.Info($"122334334 {abc}");
Logger.Warn("12333, err:{err}", "hahaha");
Logger.Error("122334334");
Logger.Fatal("12333");
}
}
Copy the codeMore
Adding LoggingEvent also attempts to create a batch submission log, just as PeriodBatchingConfig is defined above. However, when periodPeriodis actually used, some providers do not support setting the log time, as the time is recorded internally. However, if you just use your own extension and don’t use an external logging framework such as Log4net, I think you can still do it and improve efficiency. Currently, I mainly use Serilog and Log4net. I won’t update it for the time being
Things to address in the next version
-
ILogProvider logs and returns a Task
-
Serilog’s Filter is based on LogEvent, which is simpler and can be filtered according to the Properties in LogEvent. So AddFilter API can update AddFilter (Func < LogHelperLoggingEvent, bool > filter)
Reference
-
https://github.com/serilog/serilog
-
https://github.com/serilog/serilog-sinks-periodicbatching
-
https://github.com/aspnet/Logging
-
https://github.com/aspnet/Extensions/tree/master/src/Logging
-
https://github.com/WeihanLi/WeihanLi.Common/tree/dev/src/WeihanLi.Common/Logging