The Elastic Stack is an excellent open source Stack for log aggregation and analysis. It represents Elasticsearch (NoSQL database and search server), Logstash (log transfer and resolution service), and Kibana (Web interface that connects users to Elasticsearch database and enables visualization and search options for system operation users). With a large open source community, Elastic Stack is very popular and I’m happy to work with it. In this article, we’ll walk you through a simple installation of Elastic Stack on AWS or Amazon Web Services.
The following instructions will guide you through the steps involved in creating an effective sandbox environment. Since the production setup is more comprehensive, we decided to detail how to change the configuration of each component to be ready for use in a production environment.
We’ll start by describing the environment, then step through how each component is installed, and finally configure the sandbox server to send its system logs to Logstash and view them through Kibana.
AWS Environment: Learn about putting Elastic Stack on AWS
We ran the tutorial on a single INSTANCE of AWS Ubuntu 18.04 stored locally on an instance of M4.large. We started an EC2 instance in the public subnet of the VPC, and then we set up the security group (firewall) to allow access from anywhere using SSH and TCP 5601 (Kibana).
Production tip: A production installation requires a minimum of three EC2 instances – one for each component, each with an additional EBS SSD volume.
To complete this tutorial, you must register an account with AWS and create an EC2 instance of Ubuntu 18.04. In order to enable the addresses of our ports 9200 and 5601 to be accessed externally, we must open these two ports by setting inbound rules:
Add two Custom TCP configurations:
So we have opened the two ports 5601 and 9200.
Install the Elasticsearch
Elasticsearch is a widely used database and search server that is a major part of the ELK setup.
The benefits of Elasticsearch include:
- Easy to install and use
- Powerful internal search technology (Lucene)
- RESTful Web interface
- Data can be used in SCHEMa-free JSON documents (noSQL)
- Open source
There are several ways to install Elasticsearch, but we will use the DEB package.
To begin installing Elasticsearch, add the following repository key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Copy the codeInstall apt-transport-https package:
sudo apt-get install apt-transport-https
Copy the codeAdd the following Elasticsearch list to the key:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Copy the codeTo install the version of Elasticsearch that only includes Apache 2.0 license features, use:
echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Copy the codeUpdate your system and install Elasticsearch with the following command:
sudo apt-get update && sudo apt-get install elasticsearch
Copy the codeIn the following positions open Elasticsearch configuration file: / etc/Elasticsearch/Elasticsearch yml:
sudo vi /etc/elasticsearch/elasticsearch.yml
Copy the codeAnd apply the following configuration:
Network. Host: "0.0.0.0" http.port: 9200 cluster.initial_master_nodes: ["<PrivateIP"]Copy the codeSet network. Host to 0.0.0.0. This will allow Elasticsearch to bind to all network interfaces on our machine. Refer to the article for a detailed description. Alternatively, if we want the PrivateIP of our machine, we can use the following command:
ifconfig
Copy the codeStart Elasticsearch:
sudo service elasticsearch start
Copy the codeIf for some reason we modify the elasticSearch. yml configuration file, we can restart our service using the following command:
sudo systemctl restart elasticsearch
Copy the codeVerify the installation with the curl command:
curl http://localhost:9200
Copy the codeIf I see something like this, then our installation was successful:
{ "name" : "ip-172-31-2-44", "cluster_name" : "elasticsearch", "cluster_uuid" : "rQ9x67KPTs2D_75b1A1LZQ", "version" : {" number ":" 7.6.0 ", "build_flavor" : "default", "build_type" : "deb," "build_hash" : "Seven f634e9f44834fbc12724506cc1da681b0c3b1e3", "build_date" : "the 2020-02-06 T00:09:00. 449973 z", "build_snapshot" : False, "lucene_version" : "8.4.0", "minimum_wire_compatibility_version" : "Minimum_index_compatibility_version" : "6.0.0-beta1"}, "tagline" : "You Know, for Search"}Copy the codeWe can also type our address in the address bar of our browser:
To make the service start at machine startup, type the following command:
sudo update-rc.d elasticsearch defaults 95 10
Copy the codeProduction tip: Do not open any other ports to the world, such as 9200! There are many robots that will search the 9200 and execute regular scripts instead of computers. Do not bind Elasticsearch to a public IP address.
Install the Logstash
Logstash is an open source tool that collects, parses, and stores logs for future use and allows for quick log analysis. Logstash can be used to aggregate logs from multiple sources, such as clusters of Docker instances, and parse them from lines of text into a structured format like JSON. In Elastic Stack, Logstash enables Elasticsearch to store and index logs.
Install the Java
Logstash requires Java 8 or Java 11 installed:
sudo apt-get install default-jre
Copy the codeVerify that Java is installed:
java -version
Copy the codeIf the output from the previous command looks like the following, you know you’re heading in the right direction:
Openjdk Runtime Environment (build 11.0.6+ 10-post-ubuntu119.04.1 64-bit Server VM (Build 11.0.6+ 10-post-ubuntu-1Ubuntu118.04.1, Mixed mode, Sharing)Copy the codeUse the following command to install the Logstash:
sudo apt-get install logstash
Copy the codeExample: Collect Apache access logs using Logstash
We first download a test file using the following command:
wget https://github.com/liu-xiao-guo/logstash_multi-pipeline/blob/master/apache-daily-access.log
Copy the codeIf we run the above command in our home, we can find a file called apache-daily-access.log.
Create a Logstash configuration file:
sudo vi/etc/logstash/conf.d/apache-01.conf
Copy the codeEnter the following configuration:
input {
file {
path => "/home/ubuntu/apache-daily-access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
Copy the codeThis file tells the Logstash to collect the local /home/ubuntu/apache-daily-access.log file and send it to Elasticsearch for indexing.
The input section specifies the file (path) to collect and the desired format. The filters section tells Logstash how to process data using grok, Date, and GeoIP filters. The output section defines the location to which the Logstash data is transported — in this case, the local Elasticsearch.
In this example, we use localhost as the Elasticsearch host name. However, in a real production setup, the Elasticsearch hostname will be different because Logstash and Elasticsearch should be hosted on different machines.
Finally, start the Logstash to read the configuration:
sudo service logstash start
Copy the codeTo ensure that data is indexed, use:
sudo curl -XGET 'localhost:9200/_cat/indices? v&pretty'Copy the codeYou should see the new Logstash index created
sudo curl -XGET 'localhost:9200/_cat/indices? v&pretty' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open Kibana_task_manager_1 X5tnbl34QsyMyOvpDUM7lQ 1 0 2 1 29.7 KB 29.7 KB Green open. apm-agent-configuration PNnvaxiHQomiPCuYIoEP7Q 1 00 283B 283B green Open ILM-history-1-000001 hWfdS8rvRDyc38zcva0KJg 1 0 60 15.6 KB 15.6 KB Yellow Open Logstash -2020.02.20-000001 qbIz0PcKT7yC1NL7uQot9g 1 1 957 0 162.9 KB 162.9 KB Green open.kibana_1 JUlEHYinSLCbfONHxarrZw 1 0 60 22.6 KB 22.6 KBCopy the codeAbove, we can see the index of logStash -2020.02.20-000001.
In practice, we might use Beats to collect data and import it into the Logstash. In this case, we need to open an extra port address to import the data into the Logstash.
Install Kibana on AWS
Kibana is an open source data visualization plugin for Elasticsearch. It provides visualization over content indexed on the Elasticsearch cluster. Users can create bar charts, line charts and scatter charts. The pie chart. And mapping on top of a lot of data.
Among other things, Kibana makes logging super easy and even fun, and its graphical Web interface enables beginners to perform powerful log searches.
To install Kibana, use the following command:
sudo apt-get install kibana
Copy the codeOpen the Kibana configuration file and enter the following configuration:
Server. port: 5601 server.host: "0.0.0.0" elasticSearch. hosts: ["http://localhost:9200"]Copy the codeKibana start:
sudo service kibana start
Copy the codeIf for some reason we modify the above kibana.yml file, we can restart Kibana using the following command:
sudo systemctl restart kibana
Copy the codeTest: We typed the following link in our browser’s address bar:
If you see the output above, it indicates that our Kibana installation was successful.
The next step for Kibana is to define the Elasticsearch index schema.
What does “index pattern” mean, and why do we have to configure it? Logstash creates a new Elasticsearch index (database) every day. The index name is as follows: logstash- YYYY.mm.DD- for example, “logstash-2020.02.20-000001” for the index we created above on April 16, 2019.
Kibana works on top of these Elasticsearch indexes, so it needs to know which index you want to use. Go to Management-> Kibana -> Index Pattern. Kibana automatically recognizes the Logstash index, so all you need to do is define it with ‘Logstash -* :
In the next step, we will select the @TIMESTAMP timestamp field and click the “Create Index Pattern” button to define the pattern in Kibana.
Production tip: In this tutorial, we will access Kibana directly through an application server on its port 5601, but in a production environment, you might want to put a reverse proxy server (such as Nginx) in front of it.
To view your log, go to the ‘Discover’ page in Kibana:
As you can see, creating an entire pipeline for log shipping, storage, and viewing is not a daunting task. In the past, storing and analyzing logs was an incredible technique that required dealing with huge unstructured text files. But the future looks brighter and simpler.
The next step
If you want to learn more about Elastic, you can check out my previous post:
- Elastic: Beginner’s Guide
- To harden and secure our installation, see the article “Elasticsearch: Setting Elastic Account Security”