Taking common iot usage scenarios as an example, this paper introduces how to use EMQ X MQTT server + InfluxDB + Grafana to build iot data visualization platform, which can show the time series data uploaded by iot devices conveniently.

The device data and data storage scheme connected to the platform in the Internet of Things project have the following characteristics:

  • The dimension, frequency and number of devices of data collection are quite large, and the amount of data collected is relatively large, which puts great pressure on the access throughput of the message server and the storage space of the back-end database.
  • Data are reported, transmitted, and stored in accordance with the collection period.

It is therefore a good choice to use a temporal database in an iot project. Sequential databases offer significant performance improvements, including higher occupancy, faster large-scale queries, and better data compression. After data is stored in the database, the data visualization platform is often required to count and present the data according to the rules, so as to realize the business requirements such as data monitoring and indicator statistics, so as to give full play to the value of the data.

The business scenario

Assume that there is an existing batch of devices, each device has a Client ID, and all devices send data to the corresponding topic on the MQTT server through the MQTT protocol. The topic design is as follows:

devices/{client_id}/messages
Copy the code

Each device sends the temperature and humidity data collected by the sensor in JSON format.

{
    "temperature": 30."humidity" : 20
}
Copy the code

Now you need to store data in real time so that you can view it at any later time. The specific requirements are as follows:

  • Each device reports data every five seconds, and the database stores each piece of data.
  • The visualization system displays the average, maximum, and minimum temperature and humidity in any time period and the average temperature and humidity in all time periods.

The final display effect of the visual platform is shown below. In the upper right corner of the dashboard, you can select the time interval and automatic refresh time. When the device sends data continuously, the data value on the dashboard changes accordingly, achieving a comprehensive visual effect of functions.

Plan to introduce

At present, there are many Internet of Things message-oriented middleware, timing database and data visualization products in the market. Combining data collection and reporting, networking access, message storage and visualization functions, The combination of EMQ X (high-performance IOT MQTT messaging middleware) + InfluxDB (time series database) + Grafana (beautiful and powerful visual monitoring indicator presentation tool) is undoubtedly the best iot data visualization integration solution.

The overall architecture of the scheme is shown in the figure below:

  • EMQ X: EMQ X is an open source MQTT messaging server based on the highly concurrent Erlang/OTP language platform that supports mega connections and a distributed cluster architecture with a published-subscribe pattern. EMQ X has a number of built-in features out of the box. The Enterprise edition of EMQ X Enterprise supports high-performance storage of device messages to InfluxDB via a rules engine or message persistence plug-in, with open source users handling the message storage process themselves.
  • InfluxDB: InfluxDB is an open source time-series database created by InfluxData. Written by Go, it focuses on high-performance query and storage of time-series data. InfluxDB is widely used for monitoring data of storage systems and real-time data in the IoT industry.
  • Grafana: Grafana is a cross-platform, open source metrics analysis and visualization tool that visually displays data collected through flexible configuration queries. It can quickly and flexibly create client-side charts. The official library is rich in dashboard plug-ins, such as heat maps, line charts, charts and other display methods. Supports data sources such as Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch and KairosDB. You can create custom alarm rules and notify them to other message processing services or components.

Implementation steps

All components used in this paper have Docker images. Except for EMQ X, which needs to be downloaded and installed (it is convenient to modify some configurations), both InfluxDB and Grafana are built by Docker. The detailed installation steps will not be described in this paper.

The official websites of the three components have installation packages and tutorials for different operating systems:

  • EMQ X website
  • InfluxDB website
  • Grafana website

EMQ X installation

The installation

Visit the EMQ X download page to download the installation package suitable for your operating system. Since data persistence is an enterprise function, you need to download EMQ X Enterprise edition (you can apply for a trial License). At the time of this writing, the latest version of EMQ X Enterprise is V3.4.5, which is required for this tutorial. The steps to start the tutorial are as follows:

Decompress the downloaded installation packageUnzip emqx - ee - macosx - v3.4.4.zipcd emqx

Copy the License file to the EMQ X specified directory etc/. You need to apply for a trial License or purchase authorization to obtain the Licensecp .. /emqx.lic ./etc## Start EMQ X in console mode
./bin/emqx console
Copy the code

Modify the configuration

The following configuration files are required for this article:

  1. License file, EMQ X Enterprise edition License file, overwrite with available licenses:

    etc/emqx.lic
Copy the code

  2. EMQ X InfluxDB Message storage plug-in configuration file used to configure the InfluxDB connection information and select the incoming Topic:

    etc/plugins/emqx_backend_influxdb.conf
Copy the code

    Fill in the following plug-in configuration information based on the actual deployment:

    Backend. Influxdb. Pool1. Server = 127.0.0.1:8089 backend. Influxdb. Pool1. Pool_size = 5## Whether or not set timestamp when encoding InfluxDB line
backend.influxdb.pool1.set_timestamp = true

## Store Publish Message
## Change the default topic filter here because the business only needs devices/{client_id}/messages topics
backend.influxdb.hook.message.publish.1 = {"topic": "devices/+/messages"."action": {"function": "on_message_publish"}, "pool": "pool1"}
Copy the code

  3. EMQ X InfluxDB Message storage plug-in message template file used to define message parsing import templates:

    ## Template file
data/templates/emqx_backend_influxdb_example.tmpl

## Rename to
data/templates/emqx_backend_influxdb.tmpl
Copy the code

    Since MQTT messages cannot be written directly to InfluxDB, EMQ X provides the emqx_backend_influxdb. TMPL template file to convert MQTT messages to DataPoint that can be written to InfluxDB:

    {
  "devices/+/messages": {
    "measurement": "devices"."tags": {
      "client_id": "$client_id"
    },
    "fields": {
      "temperature": ["$payload"."temperature"]."humidity": ["$payload"."humidity"]},"timestamp": "$timestamp"}}Copy the code

    For more information on the use of EMQ X InfluxDB, see [InfluxDB Data Storage](

InfluxDB installation

Install with Docker, map data folder to port 8089 udp and port 8086 (used by Grafana) :

EMQ X supports only the InfluxDB UDP channel and needs the support of the influx_UDP plug-in. The database name is specified as DB

Use the influx_UDP plugin
git clone https://github.com/palkan/influx_udp.git

Go to the plugin directory
cd influx_udp

Create and start the container using the plug-in configuration
docker run --name=influxdb --rm -d -p 8086:8086 -p 8089:8089/udp \
	-v ${PWD}/files/influxdb.conf:/etc/influxdb/influxdb.conf \
  -e INFLUXDB_DB=db \
  influxdb:latest

## Check the running status of the container after startup
docker ps -a

Copy the code

At this point, you can restart EMQ X and start the plug-in to apply the above configuration:

./bin/emqx stop

./bin/emqx start

Or use the console mode to see more information
./bin/emqx console

## Start the plug-in
./bin/emqx_ctl plugins load emqx_backend_influxdb

## After successful startup, the following message will be displayed
Plugin emqx_backend_influxdb loaded successfully.

Copy the code

Grafana installation

Install and start Grafana from Docker using the following command:

docker run -d --name=grafana -p 3000:3000 grafana/grafana
Copy the code

Log in to the Grafana visual panel using the default user name and password of admin admin. After login, change the password as prompted. Use the new password to log in to the main page:

Write analog data

Before visualizing configuration, you need to write simulated data to preview effects during configuration.

The following script simulates a scenario in which 100 devices report simulated temperature and humidity data every 5 seconds for the past 12 hours and send it to EMQ X. After installing the Node.js platform, readers can start by using the following command:

npm install mqtt mockjs --save
node mock.js
Copy the code

After the simulation script is executed, data is written to the InfluxDB database. Run the following command to enter the InfluxDB container and view the data:

Enter the docker container
docker exec -it influxdb bash

## Enter the command line influxdb
root@581bde65650d:/# influx

## Switch to db database
use db;

## Query data
select * from devices limit 1;

## Query resultsname: Devices time client_id humidity temperature -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 1574578725608000000 mock_client_1 54.33 98.5Copy the code

Attached: The simulation script is as follows:

// Node.js
// mock.js
const mqtt = require('mqtt')
const Mock = require('mockjs')

class MockData {
  constructor(clientNum = 20) {
    this.EMQX_SERVER = 'mqtt://localhost:1883'
    this.clientNum = clientNum
    this.clients = {}
    this.startMock()
  }

  async startMock() {
    const now = Date.now()
    // last 12h every 5s
    for (let ts = now - 12 * 3600 * 1000; ts <= now; ts += 5 * 1000) {
      for (let i = 0; i < this.clientNum; i++) {
        const clientId = `mock_client_${i}`
        const client = this.clients[clientId] || await this.createClient(clientId)
        this.clients[clientId] = client
        const mockData = this.getMockData()
        client.publish(`devices/${clientId}/messages`.JSON.stringify(mockData))
        console.log(`${clientId} send temperature ${mockData.temperature} humidity ${mockData.humidity}`)}}}/** * Init a virtual mqtt client * @param {string} clientId ClientID */
  createClient(clientId) {
    return new Promise((resolve, reject) = > {
      const client = mqtt.connect(this.EMQX_SERVER, {
        clientId,
      })
      client.on('connect', () = > {console.log('client s% connected', clientId)
        resolve(client)
      })
      client.on('error', (e) => {
        reject(e)
      })
    })
  }

  /** * Generate mock data */
  getMockData() {
    return {
      temperature: parseFloat(Mock.Random.float(22.100).toFixed(2)),
      humidity: parseFloat(Mock.Random.float(12.86).toFixed(2)),}}}// startup
new MockData(100)

Copy the code

Visual configuration

After the components are installed and simulated data is successfully written, configure the data visualized on the Grafana visual interface.

Add data source

Add the data source, that is, the data source information displayed. The InfluxDB data source is selected and the connection parameters are input for configuration. By default, the key configuration information is as follows:

  • The Grafana container network does not interact with each other because the Grafana container network does not interact with each other because of the influence of the Docker installation. Enter the current server Intranet/LAN address instead of127.0.0.1 或 localhost;
  • Auth: InfluxDB Enabled by default no authentication mode is configured. Enter the value based on the actual situation.
  • The Database:dbFor EMQ X to write the default database name.

Adding a dashboard

After the data source is added, add the data dashboard information to be displayed. The Dashboard is a collection of multiple visual panels. After clicking New Dashboard, select Add Query to Add the data panel by querying:

Creating the panel requires four steps: Queries, Visualization, General, and Alert, which are created based on your business requirements.

Average temperature and humidity panel

Using Grafana’s visual query build tool, find the average of all devices:

  • FROM: To select datameasurementAccording to theemqx_backend_influxdb.tmplFile configuration, heremeasurement 为 devices;
  • SELECT: SELECT and calculate the fields. These two queries need to be usedAggregationFunction processing, respectively selecttemperature mean 和 humidity mean, query and calculate the average values of the temperature and humidity fields.
  • GROUP BY: The default time range is used for aggregation.
    • time($__interval)Function representation$__intervalTime interval data, such astime(5s)Said from each5Take the value from the original data in seconds for calculation (calculation in SELECT)
    • fillParameter Indicates the default value when there is no valuenullThe data point is not shown on the graph when the data point is not shown on the graph.
    • tagThis parameter is optional. The value is displayed according to the specified tag.
  • ALIAS BY: ALIAS of the query for easy visual viewing.

Visualization is not changed by default. Change the panel name to Device Temperature and Humidity Mean Value in General. If you need to monitor alarms for services, you can choreograph alarm rules in Alert. This function is not used for visual display only.

When you’re done, click the back button in the upper left corner to successfully add a data panel to the Dashboard. Click the top navigation bar to save the icon, and enter the Dashboard name to complete the creation of the Dashboard.

Maximum and minimum temperature and humidity panel

Continue to click the Dashboard Add Panel button to Add the temperature maximum and minimum charts. The operation procedure is the same as adding the average value. Only the SELECT method field in the query is adjusted to the Max and min methods in Selectors.

Total average temperature and humidity, number of data panel

Continue to click the Dashboard’s Add Panel button to Add the total average temperature and humidity and the number of data bars panel. The operation procedure is similar to the above two steps. Use the count and mean methods respectively to operate the specified fields, and cancel the GROUP BY field to complete the query. Choose Gauge as the chart type in the Visualization configuration.

Save the dashboard, and drag and drop each data panel to size and position it to get a visually good data dashboard. When the final report is complete, it will look like it did at the beginning of the article.

conclusion

So far, we have completed the construction of EMQ X + InfluxDB + Grafana Internet of Things data visualization platform. In this article, readers can learn that the InfluxDB + Grafana visualization system based on THE rich expansion capabilities of EMQ X can be developed very quickly and flexibly in the data visualization solution for the storage, computational analysis and presentation of massive data. Learning more about Grafana’s other features allows users to customize better data visualization and even monitoring and alerting systems.

Copyright notice: this article is the original EMQ, reprint please indicate the source.

Link: www.emqx.io/cn/blog/lig…