“This is the 26th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021.”
One, the introduction
In the previous article, we have introduced that when using JMeter non-GUI mode for pressure measurement, we can use InfluxDB+Grafana to monitor real-time performance test results, and Tegraf+InfluxDB+Grafana can also be used to monitor server performance. Although the Grafana kanban can show the number of requests and failure rates for transaction execution. But we also want to know why it failed.
Not all HTTP request failures are caused by 500, but sometimes 200. The response assertion simply checks for the presence of a given string in the response data, and if not, the request failed. But what is our actual response data over this period of time? It is important to know that debugging your application during performance testing is important. We often use Ali Cloud or physical machine cluster to pressure test, even if we record the response data in the log, we may not be able to get the data immediately. We can only wait for the end of the pressure test to SSH/FTP access the host to check the logs. We cannot collect these large amounts of unstructured text data using InfluxDB as we can with performance test results. Because InfluxDB as a sequential database is not designed to retrieve text.
One of the simple lightweight logging solutions is to use ElasticSearch+FileBeats+Kibana to collect and analyze error response data.
Second, the background
1, Filebeat
Filebeat is a new addition to the ELK protocol stack, a lightweight open source log file data collector implemented in the GO language. Filebeat is installed on the server as a proxy to monitor log directories or specific log files, either forwarding logs to Logstash for parsing or sending them directly to ElasticSearch for indexing. Filebeat documentation, configuration is simple, natural and support the ELK, for Apache, Nginx, System, MySQL logs provide services such as the default configuration, collection, analysis and display a dragon.
As shown below, the configuration of Filebeat is straightforward
filebeat:
spool_size: 1024 You can save up to 1024 pieces of data to send at the same time
idle_timeout: "5s" Otherwise it has to be sent every 5 seconds
registry_file: "registry" # File read location record file, will be placed in the current working directory.
config_dir: "path/to/configs/contains/many/yaml" If the configuration is too long, you can split the configuration by directory loading
prospectors: # Those with the same configuration parameters can be classified as a prospector
-
fields:
log_source: "sample" # Similar to logstash add_fields, where "log_source" is used to identify which project the log comes from
paths:
- /var/log/system.log # indicate where to read the file
- /var/log/wifi.log
include_lines: ["^ERR"."^WARN"] Send only logs containing these words
exclude_lines: ["^OK"] # do not send logs containing these words
-
document_type: "apache" # define the value of _type when writing ES
ignore_older: "24h" No longer listen to files that have not been updated for more than 24 hours.
scan_frequency: "10s" Scan the directory every 10 seconds to update the list of files matching wildcards
tail_files: false Whether to start reading from the end of the file
harvester_buffer_size: 16384 # 16384 bytes are read each time the file is actually read
backoff: "1s" Check the file every 1 second to see if there is a new line to read
paths:
- "/var/log/apache/*" # wildcards can be used
exclude_files: ["/var/log/apache/error.log"]
-
input_type: "stdin" # in addition to "log", there is "stdin"
multiline: # multiline merge
pattern: '^[[:space:]]'
negate: false
match: after
output.elasticsearch:
hosts: ["127.0.0.1:9200"] # The elasticsearch host
Copy the codeLogs sent by Filebeat contain the following fields:
- Beat. hostname: indicates the name of the host on which beat is running
- Beat. name: The name of the shipper configuration section. If this is not set, it is equal to beat.hostname
- @timestamp: the time when the contents of the line were read
- Type Indicates the content set by: document_type
- Input_type: from “log” or “stdin”
- Source: indicates the full path of the file name
- Offset: indicates the start offset of the log
- “Message” : log content
- Fields: Other fixed fields added are stored in this object
2, Elasticsearch
Elasticsearch is an open source, highly extensible, distributed full text search engine that can store and retrieve data in near real time. Its scalability is very good, can be extended to hundreds of servers. Elasticsearch is strong in full text search, and InfluxDB is good at timing data, so it is still a case by case analysis. Elasticsearch works best if you need to keep a log and query it frequently, such as our JMeter log. If you only rely on logs to display status and query occasionally, InfluxDB is appropriate.
3, Kibana
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. Kibana provides the ability to search, view, and interact with data stored in the Elasticsearch index. Users can easily perform advanced data analysis and visualize data in a variety of charts, tables, and maps. Fibana is not as beautiful as Grafana on the chart display, but Kibana is very handy for retrieving logs from Elasticsearch.
Iii. Overall structure
4. Log collection architecture
Five, installation and configuration
1. Download and configure ElasticSearch
Can directly refer to the website of the tutorial, built the wheels will not repeat here Website tutorial address: www.elastic.co/downloads/e…
After the installation is complete, you can access ElasticSearch by using http://elasticsearch-host-ip:9200
2. Download and configure Kibana
Refer to the website tutorial: www.elastic.co/downloads/k…
Run kibana.bat/.sh to make sure you can access the Kibana home page using http://kibana-host-ip:5601
3. Download and configure FileBeat
Refer to the website tutorial www.elastic.co/downloads/b… We need to deploy a FileBeat node for each press. FileBeat collects log data and sends it to ElasticSearch for storage.
Update filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- D: \ BaiduNetdiskDownload \ jmeter \ apache - jmeter - 4.0 \ bin \ jmeter log
multiline.pattern: ^ [0-9] {4} - [0-9] {2} - [0-9] {2}
multiline.negate: true
multiline.match: after
output.elasticsearch:
hosts: ["127.0.0.1:9200"]
Copy the codeBy default, FileBeat logs each line in a log file as a separate log entry. Sometimes JMeter exceptions can span multiple lines. So we need to configure Filebeat.yml in multi-line mode.
Jmeter.log Each log entry has its time stamp (YYYY-MM-DD). So, we can configure the mode to intercept from the timestamp, and if there is no timestamp, FileBeat can append the line to the previous line according to the configuration.
Once FileBeat is started, log files will be monitored and data will be sent to the ElasticSearch store every time the log file is updated.
Vi. JMeter log collection
We created a very simple test, as shown below, with only the Debug Sampler, using a BeanShell Assertion to listen for writing returned data to a log file when any errors occur.
Once the pressure test starts, FileBeat will start collecting information from the log file and forward it to ElasticSearch storage, where we can retrieve the detailed log through Kibana.
If we click the small arrow to expand the details, the message section below displays the log details we are interested in.
Seven, summary
In addition to real-time performance test results and real-time performance data, we were able to collect real-time response data for failed requests. This is useful when we are running distributed load tests for a long time. This setting helps us examine the response data to understand application health and test tool behavior when a request transaction suddenly fails.
This article only throws out a brick to introduce jade, if you are interested, you can refer to the tutorial in-depth practice.
Related information:
- Github.com/zuozewei/bl…