Three open source log aggregation tools

Log aggregation systems can help with troubleshooting and other tasks, and here are three top tool options.

How is measure aggregation different from log aggregation? Can’t logs contain measures? Can’t log aggregation systems do the same things as measure aggregation systems?

These are the questions I often hear. I’ve also seen vendors use their log aggregation systems as a solution to all observable problems. Log aggregation is a valuable tool, but it is generally not a good tool for time series data.

Two valuable features in a time series metric aggregation system are periodic intervals and a storage system specifically tailored for time series data. Interval rules allow users to obtain true mathematical results consistently. If a log aggregation system collects metrics regularly, it might work in the same way. However, the storage system is not optimized for typical query types in metric aggregation systems. Processing these queries using the storage system in the log aggregation tool requires more resources and time.

So, we know that a log aggregation system may not be suitable for time series data, but what good is it? Log aggregation systems are a good way to gather event data, which is a very important unconventional activity; an example might be access logging for Web services. Because we want to know who is accessing our system and when. Another example is triggering an application error condition — because it is not a normal operating condition, it can be valuable in troubleshooting.

Some logging rules:

• Including time stamps

• Formatting with JSON

• Don’t record unimportant events

• Log all application errors

• A log warning may be logged

• Enabling Logging

• Write messages in human-readable form

• Do not record information during production

• Do not record anything that a human cannot read or react to

Cloud computing Costs

The cloud seems to be an attractive choice when looking at log aggregation tools. However, there can be significant costs. When aggregated across hundreds or thousands of hosts and applications, logs represent large amounts of data. In cloud-based systems, data extraction, storage, and retrieval are very expensive.

As a reference point for a real system, a cluster of about 500 nodes and several hundred applications generates 200GB of log data per day. The system may have room for improvement, but in many SaaS products, even cutting it in half costs nearly $10,000 a month. This usually involves keeping data for only 30 days, which is not long enough if you want to look at trend data from year to year.

This is not to discourage the use of these systems, as they are very valuable – especially for smaller organizations. The point is to point out that doing so can have significant costs, which can be frustrating when realized. The remainder of this article focuses on self-hosted open source and commercial solutions.

Tool selection

1.ELK

ELK, short for Elasticsearch, Logstash, and Kibana, is the most popular open source log aggregation tool on the market. It is used by Netflix, Facebook, Microsoft, LinkedIn and Cisco. All three components are developed and maintained by Elastic. Elasticsearch is essentially a full-text NoSQL search engine implementation. Logstash is a log pipeline system that extracts data, transforms it, and loads it into a store like Elasticsearch. Kibana is the visualization layer on top of Elasticsearch.

A few years ago, Beat came along, and Beat was the data collector. They simplify the process of sending data to Logstash. Users can install Beat to properly export NGINX logs or Envoy proxy logs for effective use in Elasticsearch without needing to know the correct syntax for each type of log.

When installing the production-level ELK stack, you may include other parts, such as Kafka, Redis, and NGINX. In addition, Logstash is often replaced with Fluentd, which we’ll discuss later. The system was complicated to operate and led to many problems and complaints in the early days. These have been largely fixed, but it’s still a complex system, so if you’re on a smaller scale, you might not want to try it.

Don’t worry, there is a paid service that logz.io runs for you. But if you have a lot of data, it’s priced a little bit higher. Of course, you might be smaller, you might not have much data. If you can’t afford logz.io, check out services like AWS Elasticsearch Service (ES). ES is a service provided by Amazon Web Services (AWS) that makes Elasticsearch easy and fast to run. It also has tools to import all AWS logs into ES using Lambda and S3. This is a cheaper option, but requires some management and has some limitations.

Stack’s parent company Elastic offers a more powerful product using an open core model that provides additional options for analysis tools and reports. It can also be hosted on Google Cloud Platform or AWS. This is probably the best choice, as this combination of tools and hosting platforms offers a cheaper solution than most SaaS options and still provides a lot of value. The system can effectively replace or provide the functionality of security information and event management (SIEM) systems.

The ELK stack also provides excellent visualization tools through Kibana, but it lacks alerting capabilities. Elastic offers alerts as a paid X-pack add-on, but nothing is built into the open source system. Yelp has created a solution to this problem called ElastAlert, and there may be others. This additional software is quite powerful, but it adds complexity to an already complex system.

2.Graylog

Graylog has been gaining popularity recently, but it was created by Lennart Koopmann in 2010. Two years later, a company of the same name was born. Although its use is increasing, it still lags far behind ELK. This also means less functionality for community development, but it keeps pace with the ELK stack. Graylog received accolades in the Go community with the launch of the Graylog Collector Sidecar written in Go.

Graylog uses Elasticsearch, MongoDB, and the Graylog server in the background. This makes it as complex as the ELK stack, if not more so. However, Graylog has alerts built into the open source version, as well as a few other notable features such as streaming, message rewriting, and geolocation.

Streaming features allow data to be routed to specific streams in real time as it is being processed. With this feature, users can see all database errors in a single stream and Web server errors in different streams. Alerts can even be based on these flows when new items are added or thresholds are exceeded. Latency can be one of the biggest problems with log aggregation systems, and streaming eliminates this problem in Graylog. Once the logs are in, they can be streamed to other systems without adequate processing.

The message rewriting feature uses the open source rules engine Drools. This allows all incoming messages to be evaluated against user-defined rule files to allow messages to be deleted (called blacklists), fields to be added or removed, or messages to be modified.

Perhaps the coolest feature is Graylog’s geolocation feature, which allows IP addresses to be plotted on a map. This is a fairly common feature and can be used in Kibana, but it adds a lot of value — especially if you want to use it as a SIEM system. Geolocation is available in the open source version.

Graylog charges for support of the open source version. It also provides an open core model for the enterprise version with archiving, audit logging, and other support. There aren’t many support or hosting options, so if you don’t use Graylog’s services, you’re probably on your own.

3.Fluentd

Fluentd was developed at Treasure Data and has been adopted by CNCF as an incubation project. It is written in C and Ruby and is recommended by AWS and Google Cloud. Fluentd has become a common alternative to Logstash in many deployments. It acts as a local aggregation tool for collecting all node logs and sending them to the central storage system. It is not a log aggregation system.

It uses a powerful plug-in system to quickly and easily integrate different data sources and data outputs. With over 500 plug-ins available, most use cases should be covered, which sounds like an opportunity to give back to the open source community.

Fluentd is a common choice in Kubernetes environments because of its low memory requirements (only a few tens of megabytes) and high throughput. In an environment like Kubernetes, where each pod has a Fluentd Sidecar, memory consumption increases linearly with the creation of each new pod. Using Fluentd will greatly reduce system utilization. This has become a common problem for tools developed in Java that are designed to run one per node, where memory overhead is not a major issue.