In previous articles, I have introduced some articles on machine learning. In today’s article, I prepared a new data set to do a further exercise. Hopefully you’ll get a better understanding of this. If you want to learn more about machine learning exercises, check out the previous post:
– Elastic: Machine learning practice – single metric job
– Elastic: machine learning practice – Multi metric job
– Elastic: Machine learning practice – Population Job
– Elastic: machine learning practice – categorization
To prepare data
We now download our experimental data at the following address:
git clone https://github.com/liu-xiao-guo/machine-learning-elastic-datasets
Copy the codeAfter downloading this data, we went to the clone directory and typed the following command:
tar xvf server_metrics.tar.gz
Copy the codeUnzip the compressed files and unzip all files to the server-metrics folder. We go to this folder. In this folder, there are three.sh files:
upload_server_metrics.sh
upload_server_metrics_noauth.sh
Copy the codeIf your Elasticsearch machine has a password, you can change your username, password and URL to upload_server_metrics.sh and import the data. If you don’t have a password, you can use upload_server_metrics_noauth.sh to import it. Before running this script, make sure the.sh file is executable:
chmod a+x upload_server_metrics.sh
chmod a+x upload_server_metrics_noauth.sh
Copy the codeFor my case, I use upload_server_metrics_noauth.sh:
./upload_server_metrics_noauth.sh
Copy the code$ ./upload_server_metrics_noauth.sh == Script for creating index and uploading data == == Deleting old index "server-metrics"== {"acknowledged":true} == Creating Index - server-metrics == {"acknowledged":true,"shards_acknowledged":true,"index":"server-metrics"} == Bulk uploading data to index... Server-metrics_7 uploaded Server-metrics_8 uploaded Server-metrics_9 uploaded Server-metrics_10 uploaded Server-metrics_11 uploaded Server-metrics_12 uploaded Server-metrics_13 uploaded Server-metrics_14 uploaded Server-metrics_15 uploaded Server-metrics_16 uploaded Server-metrics_17 uploaded Server-metrics_18 uploaded Server-metrics_19 uploaded Server-metrics_20 uploaded done - output to /dev/null == Check upload health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open server-metrics 9y3JcUu_SYu1Z-adDDWhCg 1 0 Skipped over: to be deflected or skipped over. 5. Skipped over: to be deflected or skipped over. 5. Skipped over: to be deflected or skipped over. Server-metrics uploadedCopy the codeIt shows that we have created an index called server-metrics.
GET _cat/indices
Copy the code
We can see that an index called server-metrics has been created. We create an index pattern named “server-*” for it and display it in Discover.
In this dataset, there are 630,000 pieces of data. Focus on April 1-27, 2017. The content format of each data is as follows:
{” @ timestamp “:” the 2017-03-23 T13:00:00, “” accept” : 36320, “deny” : 4156, “the host” : “server_2”, “response” : 2.4558210155, “service” : “app_ 3″,”total”:40476}
This data is access statistics for some servers. Total is the total number of visits. In today’s exercise, we will use this metric for analysis. We can create the following visualization in the visualization:
The graph above shows the total number of requests over time. We will use machine learning methods to determine what unusual events occur in the time series data.
Create a Single metric job
In this exercise, we will use Elastic 7.8 to demonstrate this. The interface may differ from previous versions, but the flow is the same:
Click on Machine Learning above:
Click Manage Jobs above (if you already have at least one ML task) or Create Jobs above (if you don’t already have one ML task).
Select our server-* index pattern:
Choose Single metric:
Choose to use the entire data set and click the Next button:
Above, we select the total number of requests field. Above, we can see High sum (total) and Low sum (total). When we select High sum (total), it means that when sum (total) exceeds the threshold of the value predicted by machine learning, it will be marked as an exception, without considering the case below the downline. Similarly, when we select Low sum (total), it means that when sum (total) is below the lower limit predicted by machine learning, the event will be marked as an exception, but not above the threshold. When we select Sum (total), it indicates that both higher and lower downlines will be taken into account and both will produce exception events. For our case, we choose Sum (total):
Typically the bucket span for machine learning is set to be between 5 and 60 minutes. This number depends on the actual amount of data collected. The larger the value, the more likely it is that an exception will be missed if the exception occurs within a very small time interval. See the last section of the Elastic: Machine Learning Practices – Single Metric Job in our previous article. In our case, we chose 30m, or 30 minutes. Click the Next button:
Let’s name this machine learning task total-Request and click the Next button:
It says everything is fine. Let’s click the Next button:
Click on the Create job:
Above, we can see that the curve shown above is the same as what we saw earlier using Visualization. The shaded part above is the range of values predicted by the model based on machine learning. At the beginning, you can see a shaded part, which represents the stage where machine learning is building models. Click the View Results button above:
From above, we can use the two direction buttons to resize the window. This way we can see the details of the exception.
For example, I choose the first place where the anomaly occurs and zoom in. We can see some more detail:
Above, we can see that the actual value is far in advance of the predicted value range and is therefore considered an outlier event. At the bottom of the display, we can see a list of exception events. Click the arrow character:
We can see the details of this exception event. We can also click the Forecast button to predict what will happen in the next few days:
Click the Forecast button:
So let’s put 7d, which is 7 days. Click the Run button:
Above, we can see the data forecast for the next seven days.
Example Creating a Multi Metric Job
Next, let’s create a multi-metric machine learning task. In this exercise, we want to track both the request value and the response time. We also wanted to analyze each application to see their request and Respone time.
Click on the Machine Learning:
Click On Manage Jobs:
One of the new metrics we added was High mean (response). The reason is that we want to find a service or server that takes too long to respond. With the number of requests and the response time being too long, we can easily locate the problem.
Above, we chose Split Field as service, which means our machine learning will run separately for each service. In normal cases, this field will automatically be included in the influencer. We also include hosts in the influencer because we want to know which hosts are playing a key role when exceptions occur. Click Next above:
Click the Next button:
Click the Next button:
Machine learning analysis was performed separately for each service and metric. Click to View the results:
We can click on an exception in app_4 to see:
Click to enter the abnormal event of April 10th, 2017, 13:00:
We can find that server_1 and server_2 have great influence on APP_4. We need to look at it.
We can also view exceptions to host separately:
On the left side above, we can see the influence of the first few apps.
Create a population job
Next, let’s create a Population job. Population job is to detect outliers in the Population. That is, each indicator is not compared to its own historical data, but to its peers. Take a common example, for example, the same team, according to the normal range of salary increases, generally in the 2% to 15%, but one person’s salary increase is 80%. This is obviously on the same level as everyone else’s salary increase. This is the exception. For example, in the picture below:
Zoralda displays and compares entity differently. This user transfers a lot more data than anyone else and stands out from the crowd.
To do this exercise, we need to create a new index. Let’s start by downloading the following files:
git clone https://github.com/liu-xiao-guo/machine-learning-datasets-useractivity
Copy the codeWhen we are done downloading, go to the directory we downloaded and type the following command:
chmod a+x ingest-data.sh
./ingest-data.sh
Copy the codeThe data format of the document is as follows:
{"username": "Frederica", "@timestamp": "2017-04-16t21:06:58.963073 ", "bytesSent": 313}Copy the codeA newly imported index called User-Activity is available in Kibana. We need to create an index pattern called user-activity for this index.
I can view the mean Data transfer diagram based on each user through visualization:
Open Kibana machine Learning:
Select the user – the activity:
Select the Next button:
Click to View the results:
We see some unusual entities. Click on an exception above:
Click the arrow above to see the details of this exception:
