A programmer can’t do without a log.

A log is like a notebook that records information about a program as it runs.

The log file

There are many things we can do with logs.

Functions of Logs

1. Record the usage of the system and interfaces, such as request logs

2. Record and analyze user behavior, such as website visit logs

3. Debug program, and the console function is similar, but the content of the console will not be saved to the file, and the log can be saved for a long time.

4. Help us troubleshoot and locate errors. For example, if the system throws an exception, the exception information is recorded in a log and can be retrieved later.

5. Analyzing logs can optimize code logic and improve system performance and stability.

Logs have so many uses, but too many can be a headache for developers. For large systems, programmers often have to look at thousands or tens of thousands of lines of logs, often looking at the log to see dizziness.

However, there are a lot of tricks to dealing with journals. fishskin shares his story with journals. fishskin

The story is divided into 7 stages, from the log to see doubt life, to play with ten million logs in the palm of the hand, what did fish do?

Fish skin and log of love and hate

The first phase has no logs

At the beginning of building the new system, for the sake of convenience, Yupi did not connect any log frame to the system, nor did it record any log. The whole project was very clean and smooth. When debugging is needed, the information is printed directly to the console with the output function, and when an exception occurs, the stack is printed directly.

Is really nothing light, great!

Unfortunately, it didn’t last long. After the project was launched, suddenly one day, the system went wrong, the data could not be checked out, and my colleagues came to visit me.

“It’s not a problem!” laughed The fish.

Then log on to the server, enter the project directory, and blink.

The project directory is still clean and silky. I don’t keep any fucking logs!

Oh, great. We’re not gonna find anything. Go ahead and add logging to your project.

The second phase introduces the logging library

The Java language has many excellent logging libraries, such as Logback and Log4j2, which provide many convenient methods for logging and printing logs. You can use one of these libraries directly without having to implement it yourself. Since the fishskin project uses the Spring Boot framework for development, it can directly use its default log library Logback.

You can add the logback. XML configuration file to configure the storage path and format of log files. The Logback framework automatically compresses logs every day and deletes them after a certain number of days to save disk space. The maximum number of storage days can also be specified in the configuration file.

The configuration file looks something like this:

Create a log object on the class to print logs:

Logger logger = LoggerFactory.getLogger(MyApp.class);

You can then use this object to log:

catch(Exception e) { logger.error(“app error”, e); }

After adding the configuration file, start the project and you can see the generated log file. Oh yeah, the next time there’s a problem with the system, you don’t have to worry about the lack of information to get it wrong!

After the system had been running for an hour, the colleague came back again. This time, the fish felt confident and said with a smile, “It’s not a problem!”

When I opened the log file, I was amazed. There were thousands of lines of logs. How could I know which line of logs was an error message?

Is that all you can do for me? Use the Linux command to filter out the log lines with the “ERROR” field

cat application.log | grep ‘ERROR’

Although the problem was solved, the filter command had to be typed again every time an error was reported, and as the file grew larger, the command execution speed became slower and slower.

Can you separate all error logs from normal logs and put them in separate files?

Phase 3 Log classification

Fortunately, common logging frameworks provide log storage tiering, which can be implemented by modifying configuration files.

Modify the logback. XML configuration file to output ERROR logs to the ERROR. Log file to implement log classification:

When the project is started, logs are written to application.log and error.log files as expected. When the system is abnormal again, fish skin only need to open the error. Log file, error information in a glance!

After the system ran for a period of time, fishskin launched a very important service and recorded quite a lot of business logs. Although the error logs can be viewed separately, the logs of core services and normal logs of other services are stored in application.log. If you want to view only the logs of core services, you still need to use command filtering, which is quite troublesome.

Is there any way to keep the core business logs in a separate file?

Phase four is segregated by class

Fortunately, the Logback logging framework allows you to log logs generated by different classes to separate files by modifying the configuration file. For example, logging all requests generated by RequestAOP classes to Request.log:

The request. Log file is automatically generated when the project is started. Open this file and you can view all the request logs for flow analysis and so on.

Later, as the system became more and more visited, a single server could no longer meet the demand for concurrency, so Fishskin added three more machines to provide services together.

One day, the system went wrong again, my colleague came to the door, fishskin thought: letter not letter minute to solve the bug for you!

A fierce operation such as tiger, log in to a server to check the log, the result of the error log is empty, than the skin of the pocket are clean.

Strange, how can not find the error message?

Yes, there are four machines now, so the request could be on another machine, so the error log could be on another machine too!

Alas, there is no way, one by one to log into the server to find the error message.

In fact, four machines can also tolerate, but later with the increase of concurrency, fish skin is responsible for the system has ten machines, each time to check the log to log on ten machines to find! And with hundreds of thousands of rows of single log data, no matter how much you slice it, it seems too much.

Oh, George, this is terrible! Is there any way I can read my journal in one place?

How about logging directly to the database?

No, no, the amount of log data is too large to be stored in the database. And the speed of writing database is limited by network transmission and so on, relatively slow.

What to do? All right, let’s get some sleep.

Phase 5 Log reporting and centralized management

“Hey, boy, do you want strength?”

“Nonsense, who wouldn’t want it!

“Ever heard of ELK? He will show you where to go.”

Fishskin woke up from a dream, yes, you can use ELK to build a distributed log collection system ah!

ELK, short for Elasticsearch, Logstash, and Kibana, is not a single piece of software, but a solution to a whole set of problems.

Elasticsearch (ES for short) is a full-text search engine that can store and search massive amounts of data efficiently.

Logstash is a data pipeline that collects data from a variety of sources, such as MySQL databases, transfers it from one place to another, parses it, and transforms it.

Kibana is a data visualization platform for displaying data stored in Elasticsearch. On Kibana, we can not only see all the raw log information, but also customize all kinds of beautiful and intuitive visualizations.

The Logstash stash is used to collect all the data on each machine and transfer it to Elasticsearch for storage. Finally, the data is displayed through Kibana. Then you can easily view and analyze all the data using Kibana.

Since you can solve the problem, then access ELK ~

However, using ELK is equivalent to introducing three new components into the system. Considering that the more components the system uses, the higher the complexity, the more difficult it is to maintain; In addition, Logstash is heavy and consumes a lot of CPU and memory. So fishskin decided to use Elasticsearch as a database instead of a Logstash database.

Elasticsearch is integrated in Spring Boot, log data is stored in Elasticsearch through the API provided by the dependency package, and log data is displayed in Kibana.

Maven introduces dependencies:

Interface to access ES:

@Repositorypublic interface UserRepository extends ElasticsearchRepository<HouseIndexTemplate, Long> {

}

The idea is good, but the reality is bloody cruel.

While Spring Boot is convenient to access Elasticsearch, it would be too intrusive to change the whole project by replacing the entire logging code with the ES code. In addition, it takes much longer to save logs to ES than to write files asynchronously. When the number of concurrent files is large, log writing fails occasionally. So in the middle of the code change, Fishskin gave up and directly restored all the changed code.

Fishskin pondered again, is there a way to write the log to ES without changing a line of code?

Phase 6 Log broker

If you do not change any code, the logs generated by each machine are still recorded independently in the log file of the current machine. It is very difficult to view the logs on each machine through a single interface.

If the data in the log file is automatically synchronized to ES, it can be easily viewed through Kibana.

Who’s going to do the synchronization? Do I need to write my own scheduled task program to upload log files to ES? In that case, I might as well continue to change the original project code.

Is it possible to get an agent to help me with this?

Just like we throw spicy chicken every day, we just throw the spicy chicken into the spicy chicken bucket at the gate of the community, and then the spicy chicken truck will help us transport the spicy chicken to the spicy chicken station for centralized processing.

Our log file is the spicy chicken, the agent is the spicy chicken truck, and the ES is the spicy chicken station.

ELK has been upgraded to ElasticStack for a long time. In addition to the three components mentioned above, there is also a Beats.

Beats is a lightweight data collector that provides different components for different data types.

To upload log file data to ES for storage, use Filebeat. Filebeat is a lightweight log collector that provides a lightweight way to forward and summarize logs and files, making it easy to deal with hundreds or even thousands of logs generated by servers, VMS, and containers.

Filebeat is an agent that collects logs from each machine and transfers them to Logstash for processing or directly to Elasticsearch for storage. So you don’t have to change the project code at all!

ElasticStack architecture is as follows:

So how do you use Filebeat?

It’s as simple as installing Filebeat directly on the server where the log files reside, and then defining the input (the path to the log files to collect) and output (where to send the collected data) in its configuration file. For example, in the following configuration, system logs are collected and transmitted to Logstash:

Inputs filebeat.inputs: -type: log Paths :- /var/log/system.log output.logstash: hosts: [“127.0.0.1:5044”]

I got it! This is so good! I used to be overwhelmed by a few thousand lines of logs. Now, it’s easy to view and analyze hundreds of thousands or millions of logs at the flick of a finger by opening the Kibana console.

Feel like a general, these logs are my rule of the soldiers, have to obediently listen to my orders, not happy!

Phase 7 Improves the log architecture

The Elasticsearch stack allows you to easily manage a large number of logs in a centralized manner. Elasticsearch also supports horizontal scaling, allowing you to store tens of millions of logs.

However, ElasticStack is not invincible when there are too many logs per second. While Filebeat, Elasticsearch, and Kibana are all powerful, Logstash is often the Achilles heel!

Because Logstash accepts logs collected by multiple Filebeat at the same time, the more machines there are, the more Filebeat will be deployed, and the greater the Logstash pressure will be. Although we can also increase the number of Logstash nodes just like expanding ES, it cannot fundamentally solve the problem. When the log magnitude increases to a certain extent, not only Logstash but also ES cluster may not be able to support!

Therefore, we need to access some middleware for buffering, the preferred reliable and high-performance message queue Kafka (relying on distributed coordination tool Zookeeper).

Finally, the complete architecture of the distributed log collection system is as follows:

So far, the fish will finally log ten million play in the palm, this feeling is really too cool.

If you’re suffering from journaling, try to set up a good journaling system.

Finally, I would like to share my own experience of logging:

1. Don’t rely too much on your journal. Write everything down.

2. Reduce the length of the log while ensuring that it is understandable, such as simplifying this is an apple to Apple.

3. Level and classify logs. DEBUG logs are generated only in the development and test environment, not in the production environment.

4. Configure a unified log format to facilitate subsequent analysis.

5. Don’t use logging as a tool to store data! Caution The log information cannot contain sensitive information, and do not make it public.

Fish skin from 0 to 1, experienced seven stages, successfully played with ten million logs in the palm of the hand. In fact, whether it is learning or practical application, we all need to have the spirit of continuous practice, exploration and optimization.

Will be ling extremely, list of small mountains.

Source: mp.weixin.qq.com/s/8lfPzt9NG…