The Nodejs Elasticsearch tutorial example is today’s main topic. Elasticsearch is an open source search engine that has become very popular due to its high performance and distributed architecture. Elasticsearch is built on top of Apache Lucene, a high-performance text search engine library. Although Elasticsearch can perform data storage and retrieval, its primary purpose is not to be used as a database. Rather, it is a search engine (server) whose primary goal is to index, search and provide real-time statistics about data.
In this tutorial, we’ll integrate with Node.js and use it to index and search data. After the data is imported, it is immediately available for search. Elasticsearch is schema-less, stores data in JSON documents, and automatically detects data structures and their types.
Elasticsearch is also fully API driven. This means that almost everything can be done using a simple RESTful API using JSON data over HTTP. It has client libraries for almost every programming language, including JavaScript. In this example, we will use the official client library.
The first step
Install Elasticsearch by following instructions in How to Install Elasticsearch on Linux, MacOS, and Windows. For Mac users, you can also install Elasticsearch using the following command:
$ brew install elasticsearch
Copy the codeElasticsearch can be run on a Mac as follows:
$ brew services start elasticsearch
Copy the codeLet’s get the Elasticsearch server up and running as described in that article.
The second step
We create a directory on our computer, for example:
$ mkdir node-elastic
$ cd node-elastic
Copy the codeIn this directory, we can create package.json files for our Nodejs project by using the following command.
$ npm init
Copy the codeI chose the default option
{" name ":" elastic ", "version" : "1.0.0", "description" : ""," main ":" index. Js ", "scripts" : {" start ": "node index.js" }, "author": "", "license": "ISC", "dependencies": { }Copy the codeHere, I made a small change to the scripts section. So we can run our index.js file with NPM start.
As shown above, let’s create a file called index.js. It reads as follows:
// index.js const express = require('express'); const elasticsearch = require('elasticsearch'); const fs = require('fs'); const app = express(); const PORT = 5000; Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); app.listen(PORT, function() { console.log('Server is running on PORT:',PORT); });Copy the codeIn the above code we need the express and ElasticSearch modules. We need to install through the following method:
$ npm install express --save
$ npm install elasticsearch --save
Copy the codeWith the –save option, we save the modules we need in our current directory.
We can run our application as follows:
$ node index.js
Copy the codeOr:
$ npm start
Copy the codeIf our index.js runs successfully, it will be at PORT:5000. Our Elasticsearch server is running on PORT:9200. The following information is displayed:
Server is running on PORT: 5000
Copy the code
Create a sample data
We need to create a sample data. This data will be read in and stored in Elasticsearch. Once the data is entered into Elasticsearch, we can search for them. We create a file called data.json in the root directory of the project and enter it with the following content:
[{" id ":" 2 ", "user" : "ShuangYuShu - zhang SAN", "message" : "today the weather is good, walk to", "uid" : 2, "age" : 20, "city" : "Beijing", "province" : "Beijing", "country" : "Chinese", "address" : "haidian district in Beijing, China", "location" : {" lat ":" 39.970718 ", "says lon" : "116.325747"}}, {" id ": "3", "user" : "the dongcheng district - liu", "message" : "in yunnan, the next stop!", "uid" : 3, "age" : 30, "city" : "Beijing", "province", "Beijing", "country" : "China" and "address", "China Beijing dongcheng district stylobate factory three 3", "location" : {" lat ":" 39.904313 ", "says lon" : "116.412754"}}, {" id ": "4", "user" : "the dongcheng district - li si", "message" : "happy birthday!", "uid" : 4, "age" : 30, "city" : "Beijing", "province" : "Beijing", "country" : "Chinese", "address" : "China Beijing dongcheng district", "location" : {" lat ":" 39.893801 ", "says lon" : "116.408986"}}, {" id ": "7", "user" : "hong qiao-laowu ", "message" :" happy birthday!", "uid" : 7, "age" : 90, "city" : "Shanghai", "province", "Shanghai", "country" : "Chinese", "address" : "China Shanghai minhang district", "location" : {" lat ":" 31.175927 ", "says lon" : "121.383328"}}]Copy the codeIt is an array of four elements. So when we index the data, if we index it correctly, then the length of the data is 4, and we’re going to prove that.
In real applications, these data may come from various sources such as real-time data or data.
Add sample data for indexing
Indexing is required for all data added to Elasticsearch. We continue to modify our index.js file as follows:
const express = require('express'); var elasticsearch = require('elasticsearch'); const fs = require('fs'); const app = express(); const PORT = 5000; Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); client.ping({ requestTimeout: 30000 }, function(error) { if (error) { console.error('elasticsearch cluster is down! '); } else { console.log('Everything is ok'); }}); const bulkIndex = function bulkIndex(index, type, data) { let bulkBody = []; data.forEach(item => { bulkBody.push({ index: { _index: index, _type: type, _id: item.id } }); bulkBody.push(item); }); client.bulk({body: bulkBody}) .then(response => { let errorCount = 0; response.items.forEach(item => { if (item.index && item.index.error) { console.log(++errorCount, item.index.error); }}); console.log( `Successfully indexed ${data.length - errorCount} out of ${data.length} items` ); }) .catch(console.err); }; async function indexData() { const twittersRaw = await fs.readFileSync('./data.json'); const twitters = JSON.parse(twittersRaw); console.log(`${twitters.length} items parsed from data file`); bulkIndex('twitter2', '_doc', twitters); }; indexData(); app.listen(PORT, function() { console.log('Server is running on PORT:',PORT); });Copy the codeHere, we can see that in the indexData() method, we first read in data from our data.json using readFileSync. Parse converts data to Javascript objects using json.parse. Finally, we use bulkIndex to reference elasticSearch API to complete data entry. As we mentioned earlier, this data may come from other kinds of data sources in practical applications. If you want to know what the BULK API is, check out my tutorial “Getting Started with Elasticsearch (1)”. It can quickly put many documents together through a single REST interface call and complete indexing of data.
Verify the entered data
In the previous section, we entered our data into the system through the BULK API interface. Now let’s see how to read the data we entered using elasticSearch. We create a new file verify.js in the current directory.
// verify.js const elasticsearch = require('elasticsearch'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); function indices() { return client.cat.indices({v: true}) .then(console.log) .catch(err => console.error(`Error connecting to the es client: ${err}`)); } module.exports = function verify() { console.log(`elasticsearch indices information:`); indices(); }Copy the codeHere, we use the client.cat.indices interface to display all current indexes. To use the verify.js file we just created, we need to further modify our previous index.js:
const express = require('express'); var elasticsearch = require('elasticsearch'); const fs = require('fs'); const app = express(); const PORT = 5000; const verify = require('./verify'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); client.ping({ requestTimeout: 30000 }, function(error) { if (error) { console.error('elasticsearch cluster is down! '); } else { console.log('Everything is ok'); }}); const bulkIndex = function bulkIndex(index, type, data) { let bulkBody = []; data.forEach(item => { bulkBody.push({ index: { _index: index, _type: type, _id: item.id } }); bulkBody.push(item); }); client.bulk({body: bulkBody}) .then(response => { let errorCount = 0; response.items.forEach(item => { if (item.index && item.index.error) { console.log(++errorCount, item.index.error); }}); console.log( `Successfully indexed ${data.length - errorCount} out of ${data.length} items` ); }) .catch(console.err); }; async function indexData() { const twittersRaw = await fs.readFileSync('./data.json'); const twitters = JSON.parse(twittersRaw); console.log(`${twitters.length} items parsed from data file`); bulkIndex('twitter2', '_doc', twitters); }; indexData(); verify(); app.listen(PORT, function() { console.log('Server is running on PORT:',PORT); });Copy the codeNotice the front
const verify = require('./verify');
Copy the codeAnd the verify() call that follows
. indexData(); verify(); .Copy the codeLet’s rerun our index.js. The following information is displayed:
We can see that four documents have been imported into Elasticsearch and an index called Twitter2 has been created. This means that we have successfully entered the data into Elasticsearch. In the following sections, we will search our data.
Search for data in the Elasticsearch index
First, let’s search through all the Twitter documents. We create a new file in the project directory called search.js. It reads as follows:
// search.js const elasticsearch = require('elasticsearch'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); const search = function search(index, body) { return client.search({index: index, body: body}); }; module.exports = function searchData() { let body = { size: 4, from: 0, query: { match_all: {} } }; search('twitter2', body) .then(results => { console.log(`found ${results.hits.total} items in ${results.took}ms`); console.log(`returned twitters:`); results.hits.hits.forEach( (hit, index) => console.log( hit._source ) ) }) .catch(console.error); };Copy the codeThis is a brand new module. Since we have indexed our data, I just need to read the data out of Elasticsearch. We pass two arguments to the search() method:
- The index name
- The DSL body of the query
If you are not familiar with DSL, please refer to my previous tutorial “Getting Started with Elasticsearch (2)”. The body here contains a query statement that we need to return the size we need for the query.
To make our search.js usable, we further modify our index.js file:
const express = require('express'); var elasticsearch = require('elasticsearch'); const fs = require('fs'); const app = express(); const PORT = 5000; const verify = require('./verify'); const searchData = require('./search'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); client.ping({ requestTimeout: 30000 }, function(error) { if (error) { console.error('elasticsearch cluster is down! '); } else { console.log('Everything is ok'); }}); const bulkIndex = function bulkIndex(index, type, data) { let bulkBody = []; data.forEach(item => { bulkBody.push({ index: { _index: index, _type: type, _id: item.id } }); bulkBody.push(item); }); client.bulk({body: bulkBody}) .then(response => { let errorCount = 0; response.items.forEach(item => { if (item.index && item.index.error) { console.log(++errorCount, item.index.error); }}); console.log( `Successfully indexed ${data.length - errorCount} out of ${data.length} items` ); }) .catch(console.err); }; async function indexData() { const twittersRaw = await fs.readFileSync('./data.json'); const twitters = JSON.parse(twittersRaw); console.log(`${twitters.length} items parsed from data file`); bulkIndex('twitter2', '_doc', twitters); }; // indexData(); // verify(); searchData(); app.listen(PORT, function() { console.log('Server is running on PORT:',PORT); });Copy the codeNote the beginning of the file:
const searchData = require('./search');
Copy the codeAnd the modification of the following parts:
. // indexData(); // verify(); searchData(); .Copy the codeThis time, we will only do search, so comment out the index and verify code. Save our code and rerun our application:
Returned twitters: {id: '2', user: '2yushu-Zhang SAN ', message:' Nice weather today, go for a walk ', uid: 2, age: 20, city: 'Beijing ', province: Lat: '39.970718', LON: '116.325747'}} {id: '3', user: 'Dongcheng District - Lao Liu ', message:' Set out, next stop Yunnan! ', uid: 3, age: 30, city: 'Beijing ', province:' Beijing ', country: 'China ', address:' Beijing ', location: {lat: '39.904313', LON: '116.412754'}} {id: '4', user: 'dong cheng cheng - Li Si ', message:' Happy birthday! ', uid: 4, age: 30, city: 'Beijing ', province:' Beijing ', country: 'China ', address:' Dongcheng district ', location: {lat: '39.893801', lon: '116.408986'}} {id: '7', user: 'rainbow - laowu ', message:' happy birthday! ', uid: 7, age: 90, city: 'Shanghai ', province:' Shanghai ', country: 'China ', address:' Minhang ', location: {lat: '31.175927', lon: '121.383328'}}Copy the codeSearch for specific terms
Create a new file search_term.js in the root directory of the project and add the following code:
// search_term.js const elasticsearch = require('elasticsearch'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); const search = function search(index, body) { return client.search({index: index, body: body}); }; Module. Exports = function searchTerm() {let body = {size: 4, from: 0, query: {match: {city: {query: 'Beijing'}}}}; console.log(`retrieving documents whose twitter matches '${body.query.match.city.query}' (displaying ${body.size} items at a time)... `); search('twitter2', body) .then(results => { console.log(`found ${results.hits.total} items in ${results.took}ms`); if (results.hits.total > 0) console.log(`returned twitters:`); results.hits.hits.forEach(hit => console.log(hit._source)); }) .catch(console.error); };Copy the codeHere, we use the following query:
Query: {match: {city: {query: 'Beijing'}}Copy the codeI look for all the documents where the city is “Beijing”.
Also, we need to change our index.js accordingly:
const express = require('express'); var elasticsearch = require('elasticsearch'); const fs = require('fs'); const app = express(); const PORT = 5000; const verify = require('./verify'); const searchData = require('./search'); const searctTerm = require('./search_term'); Const client = new ElasticSearch.client ({host: '127.0.0.1:9200', log: 'error'}); client.ping({ requestTimeout: 30000 }, function(error) { if (error) { console.error('elasticsearch cluster is down! '); } else { console.log('Everything is ok'); }}); const bulkIndex = function bulkIndex(index, type, data) { let bulkBody = []; data.forEach(item => { bulkBody.push({ index: { _index: index, _type: type, _id: item.id } }); bulkBody.push(item); }); client.bulk({body: bulkBody}) .then(response => { let errorCount = 0; response.items.forEach(item => { if (item.index && item.index.error) { console.log(++errorCount, item.index.error); }}); console.log( `Successfully indexed ${data.length - errorCount} out of ${data.length} items` ); }) .catch(console.err); }; async function indexData() { const twittersRaw = await fs.readFileSync('./data.json'); const twitters = JSON.parse(twittersRaw); console.log(`${twitters.length} items parsed from data file`); bulkIndex('twitter2', '_doc', twitters); }; // indexData(); // verify(); // searchData(); searctTerm(); app.listen(PORT, function() { console.log('Server is running on PORT:',PORT); });Copy the codeNotice this line of code in front of index.js:
const searctTerm = require('./search_term');
Copy the codeAnd the following part:
. // indexData(); // verify(); // searchData(); searctTerm(); .Copy the codeRunning our application, we can see the following results:
Found [object object] items in 3ms {id: '2', user: '2', message: 'Nice weather today, go out for a walk ', uid: 2, age: 20, city: Lat: '39.970718', LON: '116.325747'}} {id: '3', user: 'Dongcheng district - Lao Liu ', message:' Go, next stop Yunnan! ', uid: 3, age: 30, city: 'Beijing ', province:' Beijing ', country: 'China ', address:' Beijing ', location: {lat: '39.904313', LON: '116.412754'}} {id: '4', user: 'dong cheng cheng - Li Si ', message:' Happy birthday! ', uid: 4, age: 30, city: 'Beijing ', province:' Beijing ', country: 'China ', address:' Dongcheng district ', location: {lat: '39.893801', lon: '116.408986'}}Copy the codeHere we see all the documents of City for Beijing. One of the documents in Shanghai is not here.
At this point, we have completed all of this tutorial. If you want to learn more knowledge about the Elastic Stack, please refer to our website: www.elastic.co/guide/index…
The source code for this tutorial can be found at github.com/liu-xiao-gu…