Go language to move brick ElasticSearch – API call

This is the 17th day of my participation in the August More text Challenge. For details, see: August More Text Challenge

Introduction to the

Elasticsearch is a distributed, RESTful search and data analysis engine that is fast, can carry petabytes of data, and is highly reliable. A single cluster can run hundreds of nodes simultaneously

What can be done?

What can ElasticSearch do now? Officials have given the following complete list of solutions

In actual business, it is mostly used as log server and search service

task

Here, we mainly use the GO language to call SDK to realize the addition, deletion, change and search of ES, and we are familiar with the search operation of ES API

Installation Tools

In order to write code quickly, we use Docker to install elasticSearch. In addition to installing ElasticSearch, we also install monitoring tools and query tools to help with efficient development

Centos quick docker portal installation: juejin.cn/post/684490…

Install the elasticsearch

Create a network with containers within the same network, Docker run -d --name elasticSearch docker run -d --name ElasticSearch  --net somenetwork -p9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.62.
Copy the code

Install the ES query tool dejavu

docker run -p 1358:1358 -d --name dejavu appbaseio/dejavu
Copy the code

Install the ES monitoring tool

docker run -d --name cerebro -p 9000:9000 lmenezes/cerebro
Copy the code

Install the Kibana query tool

docker run -d --name kibana --net somenetwork -p 5601:5601 kibana:7.62.
Copy the code

API example

In fact, there are two GO language ES clients. One is the official one, and the other is the olivere one. Currently, olivere is more popular than the official one

Olivere Client address: github.com/olivere/ela…

Olivere document address: olivere. Making. IO/elastic /

Es official client address: github.com/elastic/go-…

Since we are using the ES7 version, specify this when introducing the client

import "github.com/olivere/elastic/v7"
Copy the code

The global variable

Set global variables directly for objects used repeatedly in subsequent methods to reduce code redundancy

var (
   client *elastic.Client
   url    = "http://es-ip:9200"
   ctx    = context.Background()
)
Copy the code

Creating a client

SetSniff this should be turned off since we are using Docker to start es and there is no such problem if it is installed by RPM

Write an init function to execute client creation and status detection at project start. Then you can concentrate on writing

func init() {
   client, _ = elastic.NewClient(elastic.SetSniff(false), elastic.SetURL(url))
   // Test the state of the ES connection
   _, _, err := client.Ping(url).Do(ctx)
   iferr ! = nil { log.Println("Connection to ES failed", err)
   }
   // Check the current version of es
   version, err := client.ElasticsearchVersion(url)
   iferr ! = nil { log.Println("Error querying es version", err)
   }
   log.Println("Elasticsearch version: ", version)
}
Copy the code

Create indexes and structures

The index mapping for ES is specified at creation time and cannot be updated later. It is best to specify it at design time

The mapping structure This structure has the mappings in the official document. Tweet. The properties directly remove the tweet here. Because ES7 can’t specify a type

const mapping = ` { "settings":{ "number_of_shards": 1, "number_of_replicas": 0 }, "mappings":{ "properties":{ "user":{ "type":"keyword" }, "message":{ "type":"text", "store": true, "fielddata": true }, "image":{ "type":"keyword" }, "created":{ "type":"date" }, "tags":{ "type":"keyword" }, "location":{ "type":"geo_point" }, "suggest_field":{ "type":"completion" } } } }`
Copy the code

Create index function

Es7 version does not have type, default is _doc, no need to create methods now, query, etc. Type method, or call the following method directly

func indexCreate(action string) {
   // Check whether the index exists
   exists, err := client.IndexExists("twitter").Do(ctx)
   iferr ! = nil { log.Fatal(err) }// The index does not exist
   if! exists { indexCreate,err := client.CreateIndex("twitter").BodyString(mapping).Do(ctx)
      iferr ! = nil { log.Println("Failed to create index", err)
      }
      if! indexCreate.Acknowledged { log.Println("Failed to create")}}}Copy the code

Write data

Before we can write data, we need a structure

type Tweet struct {
   User     string                `json:"user"`
   Message  string                `json:"message"`
   Retweets int                   `json:"retweets"`
   Image    string                `json:"image,omitempty"`
   Created  time.Time             `json:"created,omitempty"`
   Tags     []string              `json:"tags,omitempty"`
   Location string                `json:"location,omitempty"`
   Suggest  *elastic.SuggestField `json:"suggest_field,omitempty"`
}
Copy the code

For testing convenience, directly loop through 100 pieces of data, or more, not every field will write data, depending on the case

func createTweet() {
   // Define a dead piece of data
   tweet := Tweet{User: "olivere".Message: "Take Five".Retweets: 0.Image: "icon".Tags: []string{"music"."book"}}
   res, err := client.Index().Index("twitter").Id("1").BodyJson(tweet).Do(ctx)
   iferr ! = nil { log.Println("Data write failed", err)
   }
   fmt.Printf("Indexed tweet %s to index s%s, type %s\n", res.Id, res.Index, res.Type)
   // Loop 100 pieces of data
   for i := 1; i < 100; i++ {
      tweet := Tweet{User: "olivere".Message: "Take Five".Retweets: i, Image: "icon".Created: time.Now(), Tags: []string{"music"."book"}}
      _, _ = client.Index().Index("twitter").Id(strconv.Itoa(i)).BodyJson(tweet).Do(context.Background())
   }
   / / brush plate
   _, _ = client.Flush().Index("twitter").Do(ctx)
}
Copy the code

To get the data

Specify the data ID, get a piece of data, data exists in the Source, the data is decoded into a structure output

func getTweet() {
   res, _ := client.Get().Index("twitter").Id("3").Do(ctx)
   if res.Found {
      fmt.Printf("Got document %s in version %d from index %s, type %s\n", res.Id, res.Version, res.Index, res.Type)
      var tweet Tweet
      _ = json.Unmarshal(res.Source, &tweet)
      fmt.Println("Data content:",tweet)
   }
}
Copy the code

Delete the data

To delete a piece of data, specify the data ID

func deleteTweet() {
   result, _ := client.Delete().Index("twitter").Id("3").Do(ctx)
   fmt.Println(result.Result)
}
Copy the code

Update the data

Update the data of a certain field, specify the ID, update one

func updateTweet() {
   result, _ := client.Update().Index("twitter").Id("99").Doc(map[string]interface{}{"image": "image"}).Do(ctx)
   fmt.Println(result.Result)
}
Copy the code

A few simple CURD examples are done here, but ES is a search server, so let’s look at the SEARCH API

Search API

In the same demo here, to avoid code redundancy, do not return the err value directly, remember to add production environment, error handling is a necessary part of the process

Common methods

Because when you call the search method, the values often have to be processed and extracted to write a function to make the code a little cleaner

(struct (item.(struct))); (struct (item.(struct))

func searchValue(res *elastic.SearchResult) {
   var tweet Tweet 
   // Use reflection to get Struct field information
   for _, item := range res.Each(reflect.TypeOf(tweet)) {
      // Force the interface type to be struct
      t := item.(Tweet)
      fmt.Printf("%#v\n", t)
   }
}
Copy the code

Search all

Note that res.hits. Hits returns only 10 results by default. If you want to display more results, you need to call the search method: search (“index”).size(100).do

func searchAll() {
   res, _ := client.Search("twitter").Do(ctx)
   searchValue(res)
}
Copy the code

Field search

Searches for a value of a specified field

func searchField()  {
   stringQuery := elastic.NewQueryStringQuery("image:icon")
   res, _ := client.Search("twitter").Query(stringQuery).Do(ctx)
   searchValue(res)
}
Copy the code

Conditions of the search

Two conditions were set for search, one was the field value image/icon, the other was retweets greater than 33

func searchCondition() {
   boolQuery := elastic.NewBoolQuery()
   boolQuery.Must(elastic.NewMatchQuery("image"."icon"))
   boolQuery.Filter(elastic.NewRangeQuery("retweets").Gt(33))
   res, _ := client.Search("twitter").Query(boolQuery).Do(ctx)
   searchValue(res)
}
Copy the code

A phrase search

Phrase search is to search for a single word or more words connected together in a large section of language

func searchPhrase()  {
   phraseQuery := elastic.NewMatchPhraseQuery("message"."Take")
   res, _ := client.Search("twitter").Query(phraseQuery).Do(ctx)
   searchValue(res)
}
Copy the code

Combination of search

Combinatorial searches are common in production environments, because the user is typing one word or a set of words, and all we need to do is find what the user wants from multiple fields

func searchGroup() {
   phraseGroup := elastic.NewMatchPhraseQuery("message"."Five")
   stringGroup := elastic.NewQueryStringQuery("image:icon")
   res, _ := client.Search("twitter").Query(phraseGroup).Query(stringGroup).Do(ctx)
   searchValue(res)
}
Copy the code

conclusion

The above examples run down, can feel olivere client is still very powerful, basically can meet our daily use. And in the data structure mapping method is also implemented, we do not need to use a field one by one

The olivere website lists the apis that have been implemented, very multi-portal: github.com/olivere/ela…