The Internet of Things (IoT) describes a network of physical objects (” things “) embedded with sensors, software, and other technologies for the purpose of connecting and exchanging data with other devices and systems over the Internet. The Internet of Things is a big web. It’s bigger and more extensive than the Internet. With the growth of the Internet of Things, a lot of data can be uploaded to Elasticsearch and used as an ideal storage space. Elasticsearch can store petabytes of data and analyze data in real time. Elasticsearch can use machine learning to analyze data for exceptions and alert for exceptions. MQTT protocol is widely used in the Internet of Things. Use MQTT input to read data for small and mobile devices over lightweight messaging protocols optimized for high latency or unreliable networks. How do we pass MQTT data into Elasticsearch?

Filebeat in the Elastic Stack already supports this protocol. We can use MQTT Input as the fileBeat input. This input connects to the MQTT broker, subscribes to the selected topic and parses the data into a common message line. Everything happens before row filtering, multi-line, and JSON decoding, so this input can be used in conjunction with these Settings.

In today’s example I’ll show you how to import data into Elasticsearch using FileBeat. You can also refer to my previous article “Elastic: Passing MQTT Data into Elasticsearch”. In today’s presentation, I will use two computers to demonstrate:

As shown above, we will deploy the Elastic Stack on a MacOS machine and install FileBeat and MQTT on another Ubuntu 20.04 machine. The MQTT client can send information to the MQTT broker, which is sent to FileBeat. Finally fileBeat sends the received information to Elasticsearch for storage and analysis.

 

The installation

Elastic Stack installation

See the previous article “Elastic: A Beginner’s Guide” to deploy the Elastic Stack locally. Before running Elasticsearch, we need to make the following adjustments:

Network. The host: 0.0.0.0 discovery. Type: single - nodeCopy the code

I made Elasticsearch bind to all of its network interfaces by setting network.host to 0.0.0.0. Localhost :9200 can access Elasticsearch, and its private IP address: MAC :9200 can access Elasticsearch. We do this because Ubuntu needs to access another computer, so we can’t just bind localhost:9200.

We don’t need to do any special configuration to install Kibana. Once Elastic Stack is installed, we can access the Kibana interface directly:

 

Install the MQTT Broker and client

In Ubuntu 20.04, it can’t be as Elastic as my previous article “Elastic: Install MQTT Broker directly as described in “Pass MQTT data to Elasticsearch” because the previous release was Ubuntu 18.04. If you type the following command:

sudo apt-get update
sudo apt-get install -y python-paho-mqtt
Copy the code

The above command causes the installation to fail. We need to install as follows:

sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt-get update
sudo apt-get install mosquitto
sudo apt-get install mosquitto-clients
sudo apt clean
Copy the code

Once mosquitmosquitos are installed, we can check whether the mosquitos service has been successfully run using the following command:

service mosquitto status
Copy the code

If you see the mosquito-active state above, then mosquito-service has been successfully installed.

 

Install Filebeat

On Ubuntu 20.04 machines, we can use the following commands to install:

Curl - L - O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.1-linux-x86_64.tar.gz tar XZVF Filebeat - 7.10.1 - Linux - x86_64. Tar. GzCopy the code

We can also install this by deb:

The curl - L - O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.10.1-amd64.deb sudo DPKG -i Filebeat 7.10.1 - amd64. DebCopy the code

In this case, I used the latest 7.10.1 release to install. You will need to make adjustments to your own distribution of Elasticsearch. Theoretically it needs to be the same as the larger version of the Elastcsearch distribution.

We went to fileBeat’s configuration file filebeat.yml to modify it. Here, we use the first installation method. Let’s modify the existing filebeat.yml file:

filebeat.yml

filebeat.inputs:
- type: mqtt
  hosts:
    - tcp://localhost:1883
  topics:
    - test
  tags: ["mqtt_test"]
  fields:
    app_id: "engine"

...

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["mac:9200"]

...
Copy the code

On top, I added the following section:

- type: mqtt
  hosts:
    - tcp://localhost:1883
  topics:
    - test
  tags: ["mqtt_test"]
  fields:
    app_id: "engine"
Copy the code

Support for MQTT has been added. We filled in the address of the MQTT broker TCP ://localhost:1883, and I added the test topic I wanted to listen on. To facilitate statistics and search, I added tags and fields. These depend on the needs of your own application.

In order for the data to reach the MacOS computer, we changed the output. elasticSearch section so that the data can be imported into ElasticSearch. We use the following command to run fileBeat:

./filebeat -e
Copy the code

 

Send data to Elasticsearch

So far, we have completed all installations. Enter the following command in Ubuntu OS terminal:

mosquitto_pub -m "this is so nice" -t "test"
Copy the code

Then we looked at it on MacOS:

The GET filebeat - 7.10.1 / _searchCopy the code

{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : {" total ": {" value" : 2, the "base" : "eq"}, "max_score" : 1.0, "hits" : [{" _index ": "Filebeat - 7.10.1-2021.01.11-000001", "_type" : "_doc", "_id" : "EiXn8HYByAmDKr1llqJE", "_score" : 1.0, "_source" : {"@timestamp" : "2021-01-11t10:04:36.917z ", "message" : "this is so nice"," MQTT ": {"qos" :0, "retained" : false, "topic" : "test", "duplicate" : false, "message_id" : 0 }, "tags" : [ "mqtt_test" ], "input" : { "type" : "mqtt" }, "fields" : { "app_id" : "engine" }, "host" : { "containerized" : false, "ip" : [" 192.168.0.4 fe80: : ", "76 dc: 60 a9:72 BC: 6 bl3", "192.168.122.1", "172.17.0.1]", "MAC" : [ "1c:69:7a:0f:5b:05", "94:e6:f7:be:e2:d6", "52:54:00:d8:80:5e", "52:54:00:d8:80:5e", "02:42:7d:13:fd:d9" ], "hostname" : "liuxgu", "name" : "liuxgu", "architecture" : "x86_64", "os" : { "name" : "Ubuntu", "kernel" : "5.4.0-58-generic", "Codename" : "focal", "platform" : "Ubuntu ", "version" :" 20.04.1LTS (Fossa)", "family" : "debian" }, "id" : "65e5a6f1012248ab8c3802e90b108b7d" }, "agent" : { "type" : "filebeat", "version" : "7.10.1", "hostname" : "liuxgu", "ephemeral_id" : "A52FF3AF-BC21-402B-9AE7-2BF942345DBD ", "07 e79686 - f3fc - 4444-8620-42 e1c83cd82f", "name" : "liuxgu"}, "ecs" : {" version ":" 1.6.0 "}}}}}]Copy the code

The “this is so nice” message was successfully imported into Elasticsearch.