Extracting data through the API is not one of the most common use cases for Elastic Stack users, but it can be useful in some cases. For example, developers who use REST apis to wrap their database services might be interested in analyzing this data for business intelligence purposes. In today’s article, we’ll show you how to use the ELK stack to analyze a Service API.

Whatever the reason, Elastic Stack provides some easy ways to integrate with the API. One of these methods is the Logstash HTTP Poller input plug-in. In the following example, I’ll use this plug-in to get some weather data exposed through a public API.

To perform these steps and build your own pipeline for HTTP API analysis in the Elastic Stack, you need to set up the following:

  • Install your own Elasticsearch and Kibana
  • How to install a Logstash in an Elastic stack: Install your own Logstash
  • Set up an account at the openweathermap.org/api link. You can find registered at openweathermap.org/guide way and find their own API keys in home.openweathermap.org/api_keys.

 

To prepare data

As mentioned above, we use the OpenWeatherMap API to collect current weather data. The dataset includes statistical information collected from 40,000 weather stations on current weather conditions around the world.

The data itself can be provided in JSON, XML, or HTML format, but the default format is JSON, which is perfect for our use case because we are using Elasticsearch to index the data. There are several ways to call data – you can do this by providing the city name (the method we’ll use here), city ID, geographic coordinates, or zip code. You can also call multiple cities within specific coordinates.

The following example invokes weather statistics in a bounding box containing longitude and latitude coordinates:

http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10&appid=YourAppKey
Copy the code

Above, bbox is defined as [lon-left, Lat-bottom, Lon-right, Lat-top, Zoom], that is, it represents a rectangular region. In the appID above, we have to fill in our App Key to work.

The API returns data that gives us all the statistics we need to measure current weather conditions, including temperature, humidity, atmospheric pressure, wind speed and direction.

Here’s an example:

{"cod":200,"calctime":0.00256473,"cnt":15,"list":[{"id":2563191,"dt":1581912559,"name":"Birkirkara","coord":{"Lon":14.46,"Lat":35.9},"main":{"temp":8,"feels_like":6.94,"temp_min":8,"temp_max":8,"pressure":1028,"humidity":93},"visibility":10000,"wind":{"speed":0.5},"rain":null,"snow":null,"clouds":{"today":1},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210247,"dt":1581912558,"name":"Tripoli","coord":{"Lon":13.19,"Lat":32.88},"main":{"temp":14.62,"feels_like":12.78,"temp_min":14.62,"temp_max":14.62,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":59},"wind":{"speed":1.53,"deg":42},"rain":null,"snow":null,"clouds":{"today":12},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]},{"id":2216885,"dt":1581912558,"name":"Zawiya","coord":{"Lon":12.73,"Lat":32.75},"main":{"temp":10.33,"feels_like":8.14,"temp_min":10.33,"temp_max":10.33,"pressure":1027,"sea_level":1027,"grnd_level":1015,"humidity":77},"wind":{"speed":1.96,"deg":87},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2212771,"dt":1581912558,"name":"Sabratah","coord":{"Lon":12.49,"Lat":32.79},"main":{"temp":14.59,"feels_like":12.27,"temp_min":14.59,"temp_max":14.59,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":57},"wind":{"speed":2.05,"deg":57},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2215163,"dt":1581912558,"name":"Masallatah","coord":{"Lon":14,"Lat":32.62},"main":{"temp":8.75,"feels_like":6.93,"temp_min":8.75,"temp_max":8.75,"pressure":1027,"sea_level":1027,"grnd_level":995,"humidity":77},"wind":{"speed":0.98,"deg":309},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2219905,"dt":1581912558,"name":"Al Khums","coord":{"Lon":14.26,"Lat":32.65},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2208425,"dt":1581912558,"name":"Zuwarah","coord":{"Lon":12.08,"Lat":32.93},"main":{"temp":12.65,"feels_like":10.83,"temp_min":12.65,"temp_max":12.65,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":64},"wind":{"speed":1.3,"deg":63},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210221,"dt":1581912558,"name":"Tarhuna","coord":{"Lon":13.63,"Lat":32.44},"main":{"temp":7.58,"feels_like":6.11,"temp_min":7.58,"temp_max":7.58,"pressure":1027,"sea_level":1027,"grnd_level":992,"humidity":85},"wind":{"speed":0.56,"deg":142},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208485,"dt":1581912558,"name":"Zlitan","coord":{"Lon":14.57,"Lat":32.47},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2217362,"dt":1581912558,"name":"Gharyan","coord":{"Lon":13.02,"Lat":32.17},"main":{"temp":5.59,"feels_like":4.01,"temp_min":5.59,"temp_max":5.59,"pressure":1027,"sea_level":1027,"grnd_level":940,"humidity":90},"wind":{"speed":0.4,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523693,"dt":1581912495,"name":"Pozzallo","coord":{"Lon":14.85,"Lat":36.73},"main":{"temp":5.56,"feels_like":2.34,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2524119,"dt":1581912495,"name":"Modica","coord":{"Lon":14.77,"Lat":36.85},"main":{"temp":5.73,"feels_like":2.54,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208791,"dt":1581912558,"name":"Yafran","coord":{"Lon":12.53,"Lat":32.06},"main":{"temp":7.01,"feels_like":5.31,"temp_min":7.01,"temp_max":7.01,"pressure":1027,"sea_level":1027,"grnd_level":959,"humidity":84},"wind":{"speed":0.68,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523581,"dt":1581912494,"name":"Rosolini","coord":{"Lon":14.95,"Lat":36.82},"main":{"temp":5.32,"feels_like":2.06,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523650,"dt":1581912495,"name":"Ragusa","coord":{"Lon":14.72,"Lat":36.93},"main":{"temp":5.52,"feels_like":2.29,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":22},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]}]}
Copy the code

 

Configuration Logstash

The way to import data returned through the API into the Elastic Stack is to use the Logstash HTTP Poller input plug-in, which aggregates, decoders, and ships data from defined endpoint urls. Let’s start by creating a configuration file called logstash_OpenWeathermap.conf.

Logstash input

The input section defines the HTTP_poller input plug-in – URL endpoint for polling, request timeout, cron-based schedule (every 5 minutes), and the codec (JSON) to use. The metadatA_target setting is optional and adds certain fields to the response about poller performance.

input { http_poller { urls => { url => "Http://api.openweathermap.org/data/2.5/weather?q=London, uk&APPID = 7 dbe7341764f682c2242e744c4f167b0 & units = metric"} request_timeout => 60 schedule => { every => "5m"} codec => "json" metadata_target => "http_poller_metadata" } }Copy the code

The API we want to insert here contains the following query parameters after calling the URL:

  • Q = London, UK — Request to return weather data for London
  • APPID = – OpenWeatherMap API key
  • Units = metrics — Convert the unit format to degrees Celsius

The interface to the microservice is shown in the URL above. It returns:

{" coord ": {" lon" : 0.13, "lat" : 51.51}, "weather" : [{" id ": 802," main ":" Clouds ", "description" : "Scattered clouds", "icon" : "3 n"}], "the base", "crisis", "main" : {" temp ": 6.7," feels_like ": 0.46," temp_min ": 5, "temp_max": 0, "limit ": 0," limit ": 0, "limit ": 0," limit ": 0, "limit ": 0," limit ": 0, "limit ": 0," limit ": 0, "limit ": 0," limit ": 0, "limit ": 0," limit ": 0, "limit ": 0 230 }, "clouds": { "all": 40 }, "dt": 1581916140, "sys": { "type": 1, "id": 1414, "country": "GB", "sunrise": 1581923520, "sunset": 1581959845 }, "timezone": 0, "id": 2643743, "name": "London", "cod": 200 }Copy the code

You can use jsonformatter.org/json-viewer to format the results returned by the API.

Logstash filter

Because the API response uses JSON, Logstash requires almost no processing or parsing. Therefore, we can now leave the filter section blank.

filter {}
Copy the code

Logstash output

The following part is pretty straightforward:

output {
   	elasticsearch {
     	index => "openweather"
     	document_type => "_doc"
     	hosts => "localhost:9200"
	} 
 }
Copy the code

Above, we import the data into Elasticsearch at localhost:9200 with the index name openWeather.

The final longstash_openweather.conf is as follows:

input { http_poller { urls => { url => "Http://api.openweathermap.org/data/2.5/weather?q=London, uk&APPID = 7 dbe7341764f682c2242e744c4f167b0 & units = metric"} request_timeout => 60 schedule => { every => "1m"} codec => "json" metadata_target => "http_poller_metadata" } } filter {} output { stdout { codec => rubydebug } elasticsearch { index => "openweather" document_type => "_doc" hosts => "localhost:9200" } }Copy the code

Start the Logstash. We run the Logstash installation directory:

./bin/logstash -f ~/data/logstash_openweather.conf
Copy the code

You will see a new Elasticsearch index created using weather data. Define a new index schema in Kibana to start the analysis:

 

Analyze the data

We can create an index pattern called OpenWeather. You can then view it with Discover:

 

Using the OpenWeather Index pattern, we can make our Dashboard: