In my previous article “Beats: Collecting logs and Importing Elasticsearch using Rsyslog on Linux”, I explained how to use Rsyslog to collect logs on Linux and send them to Elasticsearch. In that solution, we configured Rsyslog as a client mode and sent logs to Elasticsearch. We structure our logs in Logstash.

In today’s article, I’ll take a different approach. We have configured Rsyslog for both server and client mode. With server mode, I can collect the logs of various applications or other systems together and send them to the Logstash file through client mode. We can even structure logs directly using the configuration of Rsyslog. Unless we need special handling, we can even leave out the filter part in the Logstash.

introduce

Understanding the millions of log lines generated by an organization can be a daunting challenge. On the one hand, these log lines provide a view of application performance, server performance metrics, and security. On the other hand, log management and analysis can be time-consuming, which can hinder the adoption of these increasingly necessary services.

Open source software such as Rsyslog, Elasticsearch, and Logstash provide tools to transfer, transform, and store log data.

In this tutorial, you will learn how to create a centralized Rsyslog server to store log files from multiple systems and send them to Elasticsearch server using Logstash. From there, you can decide how best to analyze the data.

The target

This tutorial teaches you how to centralize syslog generated or received logs, especially a variant called Rsyslog. Syslog and syslog-based tools such as Rsyslog gather important information from the kernel and from the many programs that run to keep UniX-like servers running. Because syslog is a standard, not just a program, many software projects support sending data to syslog. By centralizing this data, you can more easily audit security, monitor application behavior, and keep track of other important server information.

You can then forward data from a centralized or converged Rsyslog server to Logstash, which further parses and enrichis your log data before sending it to Elasticsearch.

The ultimate goal of this tutorial is to:

  • Set up a single client (or forward) Rsyslog server
  • Set up a single server (or collection) Rsyslog server to receive logs from the Rsyslog client
  • Set up the Logstash instance to receive messages from the RSyslog collection server
  • Set up the Elasticsearch server to receive data from the Logstash

The installation

I’m not going to do installation descriptions in today’s exercise. I hope you have installed your own:

  • Elasticsearch
  • Kibana
  • Logstash
  • Rsyslog (For details, see Beats: Collecting Logs and Importing Elasticsearch using RSyslog on Linux)

– Rsyslog for a system will be configured in server mode

– Rsyslog of other systems will be configured in client mode

To illustrate, I deployed Elastic Stack and Rsyslog on Ubuntu 20.04 Linux machines. Rsyslog on this machine will be configured in server mode. The IP address of my machine is 192.168.0.4. You can get it by:

ifconfig -a | grep 192
Copy the code
192 $ifconfig -a | grep inet 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255Copy the code

Or:

ifconfig -a
Copy the code

You can deploy Rsyslog on another Linux machine and configure it in client mode as follows. Their syslogs are sent to the centrally processed Rsyslog Server and eventually imported into Elastiicsearch via Logstash.

Configure Rsyslog as server mode to collect data in a centralized manner

In this section, we configure rsyslog-server as a central server capable of receiving data from other syslog servers on port 514. To configure rsyslog-server to receive data from other syslog servers, edit /etc/rsyslog.conf on rsyslog-server:

/etc/rsyslog.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
Copy the code

Find the lines that have been commented out in rsyslog.conf and uncomment them. The end result looks like the one shown above. In this way, we enable BOTH TCP and UDP.

The first line of each section loads the IMUDP and IMTCP modules on module(load=” imUDP “) and module(load=” imTCP “) respectively. Imudp indicates the input module UDP, and IMTCP indicates the input module TCP. These modules listen for incoming data from other system logging servers.

The second line of each section (input(type=” IMUDP “port=”514″) and input(type=” IMTCP” port=”514″) indicate that Rsyslog should start the respective UDP and TCP servers for these protocols, These protocols listen on port 514, which is the syslog default port.

Let’s restart the rsyslog service:

sudo service rsyslog restart
Copy the code

You can run the following command to check whether the rsyslog service is running properly:

service rsyslog status
Copy the code

Tip: You can use the following command to check the rsyslog configuration file:

sudo rsyslogd -N1
Copy the code

You can use the following command to check the run logs of the Rsyslog service:

journalctl -u rsyslog
Copy the code

Configure Rsyslog of other systems to send logs to the centralized server

We configure Rsyslog in client mode on other Linux machines and send logs from this machine to Rsyslog Server. This server is our centralized log processing and uploading service. In my case, its IP address is 192.168.0.4.

In the Ubuntu installation, we find /etc/rsyslog.d:

# pwd
/etc/rsyslog.d
root@liuxgu:/etc/rsyslog.d# ls
20-ufw.conf  50-default.conf
Copy the code

Let’s modify the 50-default.conf file. Add the following line before log by Facility at the top of the file to replace private_IP_of_ryslog_server with the private IP of your central server:

*.*                         @private_ip_of_ryslog_server:514
Copy the code

First part of line (.) It means we want to send all messages. Although it is beyond the scope of this tutorial, you can configure Rsyslog to send only certain messages. The rest of the line explains how and where to send the data. In our example, the @ symbol before the IP address tells Rsyslog to send the message using UDP. Change this to @@ to use TCP. Next is the private IP address of rsyslog-Server with rsyslog and Logstash installed. The number after the colon is the port number to use.

Restart rsyslog to enable changes:

sudo service rsyslog restart
Copy the code

A: congratulations! You are now sending system log messages to the central server!

Format the log as JSON

Elasticsearch requires that all documents it receives be in JSON format, and Rsyslog provides a way to do this via templates.

In this step, we will configure our central Rsyslog server to format log data using JSON templates, then send it to Logstash, and then send it to Elasticsearch.

Go back to the rsyslog-Server central server and create a new configuration file to format the message as JSON before sending it to Logstash:

sudo vi /etc/rsyslog.d/01-json-template.conf
Copy the code

Copy the following into the file exactly as shown:

/etc/rsyslog.d/01-json-template.conf

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}
Copy the code

In addition to the first and last, notice that the lines generated by this template have a comma at the beginning of them. This is to maintain the JSON structure and help keep the file readable by arranging everything neatly. This template formats your messages in the way that Elasticsearch and Logstash expect to receive them. Here’s what they look like:

{
  "@timestamp" : "2015-11-18T18:45:00Z",
  "@version" : "1",
  "message" : "Your syslog message here",
  "sysloghost" : "hostname.example.com",
  "severity" : "info",
  "facility" : "daemon",
  "programname" : "my_program",
  "procid" : "1234"
}
Copy the code

Tip: If you want to customize log data, the rsyslog.com documentation shows the variables available in Rsyslog. However, you have to send it to Logstash in JSON format, and then to Elasticsearch.

The data being sent is not yet in this format. The next step shows configuring the server to use this template file.

The configuration central server sends logs to the Logstash

Now that we have a template file with the correct JSON format defined, let’s configure the Rsyslog central server to send data to the Logstash server, which is on the same server as this tutorial.

On startup, rsyslog looks at the file in /etc/rsyslog.d and creates its configuration from it. Let’s add our own configuration file to extend the configuration.

Create the/etc/rsyslog. D / 60 – the output. The conf:

sudo vi /etc/rsyslog.d/60-output.conf
Copy the code

Copy the following lines to this file:

# This line sends all lines to defined IP address at port 10514, # using the "json-template" format template *.* @private_ip_logstash:10514; json-templateCopy the code

 

The *.* at the beginning indicates the rest of the line that processes all log messages. The @ symbol indicates the use of UDP (using @@ instead of TCP). The IP address or host name after @ is the location for forwarding messages. In our example, we use the private IP address of the Rsyslog central server because the Rsyslog central server and the Logstash server are installed on the same server. This must match the private IP address that you configure the Logstash to listen on in the next step.

Next comes the port number. This tutorial uses port 50515. Note that the Logstash server must listen on the same port using the same protocol. The last section is our template file, which shows how to format data before passing it around.

Do not restart Rsyslog. First, we must configure the Logstash to receive messages.

Configure Logstash to receive JSON information

Let’s start with the Logstash file and create the following configuration file:

/etc/logstash/conf.d/syslog.conf

# pwd
/etc/logstash/conf.d
root@liuxgu:/etc/logstash/conf.d# ls
syslog.conf
Copy the code
input { udp { host => "logstash_private_ip" port => 50515 codec => "json" type => "rsyslog" } } output { elasticsearch {  hosts => ["https://elasticsearch_private_ip:9200"] user => elastic password => password ssl_certificate_verification =>  true cacert => "/etc/logstash/config/certs/ca.crt" } # stdout { codec => rubydebug } }Copy the code

On top, you have to make the substitution according to your own time IP address. Above, my rsyslog central server and Elasticsearch are both on the server at 192.168.0.4. If you use security, configure the certificate as above. Otherwise, the security part can be omitted in the above configuration. In the above configuration, we do not use any filter to do any processing on the data. We have structured the data in Rsyslog using JSON templates. If we need to further manipulate the data, we can use filter.

By definition, the system logging protocol is UDP, so this configuration reflects that standard.

In the input block, set the Logstash host address by replacing the logstash_private_IP with the private IP address of the Rsyslog server, on which the Logstash host is also installed.

The input block configates the Logstash to listen on port 50515, so it does not compete with syslog instances on the same machine. Ports less than 1024 need to run Logstash as root, which is not a good security practice.

Be sure to replace elasticSearch_private_IP with the private IP address of your Elasticsearch server. The output block shows a simple conditional configuration. Its goal is to allow only matching events through. In this case, this is just an event of type “rsyslog”.

We can use the following command to start the Logstash:

sudo service logstash restart
Copy the code

Or use the following command to restart rsyslog:

sudo service rsyslog restart
Copy the code

Tip: To troubleshoot Logstash, use sudo service Logstash stop to stop the service and run it in the foreground with a detailed message:

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf --verbose
Copy the code

It will contain common information, such as authentication using the Logstash IP address and UDP port:

Starting UDP listener {:address=>"192.168.0.4:50515", :level=>:info}
Copy the code

Validation data collection

We opened Kibana. We need to create an index schema called logstash-* for the collected data. We can view the imported data in Discover:

From the above, we can see that the fields displayed are indeed structured data that we are working with using the JSON template.

We can use Telnet to send some data to a central Rsyslog server. We connect to the central Rsyslog server via Telnet and send it some information:

Telnet 192.168.0.4 514Copy the code

Above, 192.168.0.4 is the address of my Rsyslog central server. We send it the following log message:

<30>Aug  4 10:52:20 cooltest logstash[13329]:           "type" => "rsyslog"
Copy the code

Above, we set the syslogHost value to nice. Wait a moment, let’s do a search on the Kibana interface:

You can view the latest log information in /var/log/syslog:

In Kibana, we can check the following information:

This shows that our Rsyslog central server is working properly.

conclusion

Your log is now in Elasticsearch. What’s next? Consider reading about what Kibana can do to visualize Elasticsearch data, including line and bar charts, pie charts, maps, and more. Kibana 101: Getting Started with Visualization explains how to search and visualize logs using the Kibana Web interface.