1. Introduction

In our last article, we initially realized the visual analysis of Nginx logs, which can intuitively understand the visual output of different dimensions of the site in each time period. We’ll cover another type of kibana visualization chart — maps — in the next section.

2. Preparation

2.1 geoip plugin

As we detailed in our previous ELK feature: Day3 — Logstash & Filebeat Configuration Supplement, the geoIP plugin in Logstash is used to identify the location of the visiting IP address.

Logstash reference GeoIP plug-in sample configuration:

input {}

filter {
    grok { ... }
    }
    geoip {
      source => "remote_addr"
      target => "geoip"
    }
}

output {}
Copy the code

2.2 Type of geoIP field

According to the official documentation guidelines, to use map-related features in Kibana, you must include a field of type GEO_point in the index. Correctly identified ES content in Kibana will look like this:

2.3 Supplementary description about mapping Settings in ElasticSearch

The geoip.location field is a float for elasticSearch, which uses rc_index_pattern-* as the index name in the previous exploration.

But when we modify the logstash configuration file to store the log content in logstash-nginx-log-*, we see that geoip.location is automatically recognized as geo_point:

Index Management -> Indices; logstuck-nginx-log-2021.08.21; This is where the geoIP field types are set:

The above mapping Settings are not available in rc_index_pattern-* indexes. “Index Management -> Index Templates -> Legacy Index Templates -> logstash -> Mappings”

Elasticsearch has a new index template named Logstash. All indexes that start with a logstash-* will automatically be named with the default mapping setting. That is, if we continue to use the index RC_index_PATTERN *, we need additional Settings to normalize the content under the geoIP field, but we can save some steps by simply starting the index name with logstash-.

Because of space limitations, we will not go into the index configuration here, but we will use the logstash-nginx-log-* index name.

TIPS

Use curl to query the index-template setting of ES:

The curl - XGET 'http://192.168.0.212:9200/_template/logstash? pretty'Copy the code

Click to expand the sample to return results

{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "logstash-policy",
          "rollover_alias" : "logstash"
        },
        "number_of_shards" : "1",
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            },
            "match_mapping_type" : "string"
          }
        },
        {
          "string_fields" : {
            "mapping" : {
              "norms" : false,
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              }
            },
            "match_mapping_type" : "string",
            "match" : "*"
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "geoip" : {
          "dynamic" : true,
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
        "@version" : {
          "type" : "keyword"
        }
      }
    },
    "aliases" : { }
  }
}
Copy the code

3. Kibana map Settings

3.1 Use custom maps

By default, Kibana provides Maps through the Elastic Maps Service, based on Maps from Natual Earth and OpenStreetMap.

However, there are some inaccuracies in the Chinese map of the built-in map, and many details are missing compared with our traditional Chinese map. We can choose other basic maps by modifying the configuration. In this article, we will use Amap in Kibana.

Modify the kibana.yml configuration file as follows:

#/etc/kibana/kibana.yml
server.port: 5601
server.host: "192.168.0.213"
server.name: "rc-application-test-kibana"
elasticsearch.hosts: "http://192.168.0.212:9200"
logging.dest: /var/log/kibana/kibana.log
logging.verbose: false

map.tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'
Copy the code

After restarting Kibana service, we can go to the main menu of Kibana page -> Analytics -> Maps to open the map, we can find that we have used the Amap we set in the configuration file:

3.2 Creating a Heat Map

The steps are shown in the GIF:

You can also save the heat map to an existing dashboard:

4. To summarize

In this paper, we focus on the map tool provided by Kibana. Through this heat map, we can intuitively see that the main source of visits to the site is concentrated in guangdong Pearl River Delta region, which is a very powerful tool.

If it is in the logistics industry or other business scenarios strongly related to geographic information, you can use this mapping function to explore more feasibility.

In addition, in the experimental environment we configured, when the Logstash output to es, we used the date parameter to output to a different index every day. If we want to set the index automatically on a daily basis, it is necessary to use es’s index-template functionality, which we will explore further when we have the opportunity.

5. Reference documents

Elasticsearch Mapping

Elasticsearch geo_point field type description

Kibana uses custom maps

Kibana heat map

ELK Feature: Day5 — Using Kibana to display map heat maps