In Kibana’s visualization tools, there is a visualization tool called Timelion. Timelion is a visualization tool for time series in Kibana. Time series visualization is the visualization of analyzing data in chronological order. Timelion can be used to draw 2d diagrams, time plots on the X axis using Timelion, you can combine independent data sources in the same visualization file. Using relatively simple syntax, you can perform advanced mathematical calculations such as dividing and subtracting indices, compute derivatives and moving averages, and of course visualize the results of these calculations.

In all, there are about fifty different features (some still experimental) that can be used to slice and slice the data set to be analyzed. This article will give you a primer on Timelion and its functional syntax.

What are the advantages over just using plain bars or lines for visualization? Timelion takes a different approach. Instead of using a visual editor to create diagrams, you can use Timelion specific syntax to define diagrams by chaining functions together. This syntax allows you to do things that are not available in classic dot series graphs – such as plot data from different indexes or data sources into a single graph.

This tutorial will start with a brief introduction to the Timelion UI in Kibana, then explain the Timelion syntax and show some use cases that traditional Kibana visualizations cannot or still cannot do.

 

To prepare data

In today’s tutorial we will use Makelogs to produce our test data. If you’re still not familiar with this tool, check out my previous article “Logstash: Creating test Logs with Makelogs.” We use the following command to create the test data:

makelogs -d 30 -c 10000
Copy the code

In this way, we created 10,000 test data and covered 30 days of data. To check the generated logstuck-0 index, run the following command:

GET _cat/indices
Copy the code

From the above we can see that there are already 10004 entries in this index. We must create an index pattern called logstash-* for this logstash-0.

 

The user interface

We can enter Timelion in the following way:

By default, the above screen is displayed.

The input box on the right side of the window displays an expression for the currently selected graph. All the expressions you will see in this tutorial will be inserted into this text box. Using the “Save” button in the menu, you can store the entire Timelion worksheet and all of its graphics, or you can store the currently focused graphics as visual content that can be placed on any dashboard.

The date range of the data currently displayed may be affected by the well-known date picker at the top right of the page. To set the scale of the x axis, use the selection box next to the expression input. By default, it is set to “Auto”, which automatically determines the appropriate scale based on the selected time range. If you want to force for example one data point per day, you can set it to 1D.

 

Timelion expressions

The simplest expression – also automatically used for the new graph expression is as follows:

.es(*)
Copy the code

The Timelion function always begins with a dot, followed by the function name, followed by parentheses (asterisks in this case) containing all of the function’s arguments.

The.es (or.elasticSearch, if you prefer to type long words) feature collects data from ElasticSearch and plots it over time.

If you do not specify an index in the expression (you will see how to do this later), all indexes of Elasticsearch will be queried for data. You can change this default setting in “Advanced Settings for Kibana” by changing timelion: es.default_index setting.

By default, the.es function will only count documents, resulting in a chart showing the number of documents over a period of time.

If you enter a simple expression, even if you select a date range that contains data, you’ll only get a flat line: chances are your data doesn’t use @TIMESTAMP as the name of the primary time field. You can change the default name by using the timelion:es. Timefield setting in Advanced Settings, or by using the timefield parameter in the.es function for a single series. You will see how to set parameters in the next section.

 

Function parameters

Function can take multiple arguments, as can.es functions. Each parameter has a name that you can use to set its value in parentheses.

Parameters also have an order, which is completed by the autocomplete function. If you do not specify a parameter name, Timelion assigns values to the parameters in order, and these values are listed in the document.

The first argument to the.es function is q (for queries), which is a Query String used to filter this series of data. You can also refer to this parameter explicitly by its name, and I always recommend that you do this immediately after passing multiple arguments to a function. Therefore, the following two expressions are equivalent:

.es(*)
.es(q=*)
Copy the code

Multiple parameters are separated by commas. The.es function also has another parameter called index, which can be used to specify the index mode for the series, so the query will not be executed again against all indexes (or any of the values that will be changed above in advanced Settings).

.es(q=*, index=logstash-*)
Copy the code

If the value of a parameter contains Spaces or commas, the value must be enclosed in single or double quotation marks. Otherwise, you can omit the quotes.

.es(q='some query', index=logstash-*)
Copy the code

Chaining functions

Many Timelion functions are used to modify a range of data. You can apply them via functions in chain expressions. The.label function is one such function. You can use it to change the tags of the series:

.es(q=country:de).label(Germany)
Copy the code

You can also chain multiple expressions, as described later in this tutorial.

Multiple series

One of the advantages of using Timelion is the ability to add multiple time series to a chart. In an expression, multiple sequences must simply be separated by commas:

.es(q=de), .es(q=us)
Copy the code

You can now also use chain functionality on each series, for example to specify tags:

.es(q=CN).label(China), .es(q=IN).label('India')
Copy the code

Now that we’ve covered the basics of how to use Timelion, we can jump to a more in-depth explanation of several features and the ways you can use them.

 

Timelion functions

Data source functions

The data source capability can be used to load data into a graph. We have already seen the.es data function, which loads data from Elasticsearch. Timelion provides additional sources for loading data, which we’ll explain in this section.

Elasticsearch

Before looking at other data source functions, we should first look again at the.es function. It offers some functionality that we haven’t seen yet.

Because dividing the value of a particular field by row is a common use case (which can be done in a general visualization using the term aggregation), the.es function takes a parameter called split.

The value of the split argument must be a field name followed by a colon, followed by a number indicating how many of the uppermost values should get a row. The following expression shows the traffic for the first four countries in the sample data (the country code is stored in the geo-.dest field) :

.es(split=geo.dest:4)
Copy the code

The metric parameter controls the calculation of the y value. By default, the.es function places the number of documents on the Y-axis. You can use the metric parameter to specify another aggregation of metrics that should be used to calculate values at a particular time. The value must be the name of a single value indicator aggregation, followed by a colon, followed by the name of the field on which the aggregation is to be evaluated.

Valid names for aggregations are: AVG calculates the average of a field, sum sums, cardinality retrieves the number of unique values ina field, and min and Max retrieve the minimum or maximum value of a field.

If we wanted to modify the above expression to show the maximum number of bytes transferred in that country to show the first four countries (to see which countries are responsible for my bandwidth), we could use the metric parameter as follows:

.es(split=geo.dest:4, metric=sum:bytes)
Copy the code

You can also use multiple split parameters to create sub-buckets (as in the visualization) and multiple metric parameters to plot multiple values at each point in time.

 

static/value

.static (or alias.value) simply draws a plain horizontal line at the given value. This is useful for drawing visual thresholds on a graph. You can also mark the series directly using parameter tags:

.es(), .static(100, label='good visitor level')
Copy the code

 

Offset data source

Each data source function takes a parameter called offset. It can be used to offset data to a specific time range. This is useful for comparing data from different time ranges. The offset parameter accepts positive and negative values in units. The valid units are S for seconds, M for minutes, H for hours, D for days, W for weeks, M for months, and Y for years. It offsets the input by the specified offset before drawing it.

To compare the number of visits from the current page visitor (based on our sample data) to a week ago, use the following expression:

 

Styling functions

Some of the functionality provided by Timelion is primarily for style setting.

Line style

By default, Timelion draws data using lines. There are three functions to modify the actual graphic types:.lines,.bars, and.points. They each support a set of parameters to modify such as line width, dot size, etc. To see all available parameters, review the documentation within your application.

You can see some arguments demonstrated in the following expression:

Colors

To manually specify a particular series of colors, use the.color function. It expects the color to be an HTML color name or a hexadecimal value in the format of # RRGGBB. The following example illustrates usage:

If you use the split argument, and therefore multiple lines, you can specify multiple colors separated by colons:

 

Math functions

Timelion provides several mathematical calculations on time series. If you want the absolute value on the Y-axis, you can link to the.abs function. You can use the.log function to compute the logarithm of all values (and optionally specify the logarithm base).

The.cusum function can be used to calculate the sum of all values, that is, the value at a particular point in time is the sum of all previous points. Note that the cumulative sum is only calculated from the start of the chart, not from time.

. Derivative function can be used to calculate derivative of time series, i.e. slope of curve:

The.mvavg (long.movingaverage) function smooths a sequence by applying a movingaverage to it. This function needs its first argument (window), which specifies the size of the window as a value and a unit of time (see offset above) to calculate the moving average.

The following expression and its results demonstrate the smoothing effect of moving averages:

Trend

Another useful feature of Timelion is the ability to draw trend lines into graphs. You can use the.trend chain function on any series to plot a trend line for that series.

.es(q=CN).trend()
Copy the code

Different scales

Adding multiple series to a chart sometimes creates a problem where the ratios of values for the two series are completely different. For example, the number of documents in the following two indexes is completely different orders of magnitude.

If you change scale, it’s hard to see what the red is. If we are not interested in the actual values, but only in the way the curve changes over time, we can use the.range function to limit the sequence to the new range of values, that is, overriding the minimum and maximum values of the sequence, but leaving the shape unchanged.

In this way, we can now see the correlation between the two series. Of course, you never forget that correlation does not imply causation. We can simplify the query above by using the Timelions grouping feature, where we can group multiple series using parentheses and link functions to the entire group:

(. Es (index = logstash - *), the es (index = metricbeat - *)). The range (0, 10)Copy the code

Since using a range query would lose the actual value, the.yaxis function might be more convenient in our use case. It will assign a sequence to the different y-axes. You must specify the axis number it should use (the default y axis is 1). It can optionally modify the Y-axis using min, Max, label, color, and position parameters.

Now to achieve a similar effect to the.range query, but without losing the actual value, you can use the following expression to move the two series to different y axes.

There is another extension issue that is almost impossible to solve with classic Kibana virtualization. Imagine that you are plotting the total number of bytes of server log data. If you look at an entire week of data, each data point on the X-axis will now represent an hour’s worth of data, so take the sum of all bytes transmitted in an hour as a value. When other time ranges are selected, such as four hours later, each data point now represents only one minute, so the Y-axis value is lower. The picture below shows this, with a one-week time frame on the left and a four-hour time frame on the right. As you can see, the Y-axis on the left is higher.

Sometimes this behavior is not what you want, because no matter what time range you’re looking at, you might want to compare absolute values to each other, or you might want the values to make sense to you, for example because you want to see all the values in bytes per second, because you might be used to calculating bandwidth in that unit.

The.scale_interval function does solve this problem. It takes a parameter (named interval) that accepts a unit of time (as shown above). No matter what time range you are looking for, Timelion calculates all values for that time interval. To see the number of bytes per second, specify a value of 1s and to see a value of 1m bytes per minute:

.es(metric=sum:bytes).scale_interval(1s)
Copy the code

This doesn’t change the shape of the graph at all, it just scales the actual value that appears to you when printed on the Y-axis and hovered over the graph.

 

Calculation sequence

Another important advantage of Timelion is that it can be calculated sequentially. You can add, subtract, multiply or divide sequences by numbers or even other sequences.

Such a use case is to correlate information with other information. If we use sample Web server data to analyze visitors from India compared to China, we can display the results using the following expression:

We can see that there are many more visitors from China than from India, but of course China has a larger population than India. We can divide each series by the country’s actual population using the timeline:

Es (CN). Divide (14.005), the es (IN). Divide (13.24)Copy the code

For the convenience of calculation, the population of China is written as 1.40005 billion instead of the real number 14,0005 million. Also India’s population is 1.324 billion.

From the above, we can see that even in terms of visitors per head, China still has more visitors than India.

You can use all basic operations with.add (or.plus),.subtract,.multiply, and.divide functions. Each accepts a static value or another series (as shown in the example above).

 

Conditional choice

Timelion provides.min and.max functions. These will take a list of multiple series or values and will always return the minimum or maximum of all series/values (depending on what function you are using).

For example, you can set a graph to a specific value in this way:

.min(.es(), 25)
Copy the code

Another use case using.min and.max is to conditionally color a graph. You can say for example if it exceeds a certain threshold (4000 in this case), please highlight a sequence like this:

.es().bars(stack=false).color(#F44336), .min(.es(), 4000).bars(stack=false).color(#8BC34A)
Copy the code

This will first draw regular.es() data as a red bar, and then draw the same data, but with an upper limit of 4000. This way, the green bar will override the red, and the red bar will only show thresholds above 4000.

If you want to provide a more complex solution for conditional selection, you can use the.if (or long.condition) function to provide conditions and values for conditions that are true or false.

Rashid Khan has a great blog post, I only have one. Condition (), which explains these features in detail and shows beautiful use cases for their use.