ElasticSearch Application – Search enhancements

ElasticSearch has the ability to search for synonyms based on the user’s search time, and to split words based on semantics to return the most appropriate results.

A,ES Search with Analyzer

1.ES Search process

2.Analyzer (Parser)

1)An Analyzer

2)Differences between Analyzer and Search_Analyzer

There are two main situations in ES where analyzer is used: one is when inserting documents, the field of text type is segmented and then the inverted index is inserted; the other is when querying, the input of text type to be queried is segmented first and then the inverted index is searched. The rules used are as follows: when creating an index, just look at whether the field defines an Analyzer. If it does, use the defined one. If it does not, use the ES default one. When querying, see if the search_Analyzer is defined. If not, see if the analyzer is defined and use the ES default.

2.CharFilter

1)HTML Strip Char Filter

You can remove HTML tags, such as <p>I ‘<br>Love</br> cat</p> to: I Love cat.

Sample code:

{ "tokenizer":"keyword", "char_filter":[ "html_strip" ], "text":"< p> I' < br> Love< /br> cat< /p>" }Copy the code

The result is:

{
    "tokenizer":"keyword",

    "char_filter":[

        "html_strip"

    ],

    "text":"I Love cat"
}
Copy the code

2)Mapping Char Filter

{ "settings":{ "analysis":{ "analyzer":{ "my_analyzer":{ "tokenizer":"keyword", "char_filter":[ "my_char_filter" ] } }, "Char_filter" : {" my_char_filter ": {" type" : "mapping", "the mappings" : [" trump = & gt; TTT "]}}}}}Copy the code

3)Pattern Replace Char Filter

You can use regular expressions to match and replace characters in strings. See regular expressions for specific capabilities, and omit specific examples.

3.tokenizer

1)Standard Tokenizer

The standard type of Tokenizer, which is very friendly to European languages like English, supports Unicode. Properties:

2)NGram Tokenizer

3)Edge NGram Tokenizer

Very similar to NGramTokenizer, but with auto-completion.

Second,Chinese word segmentation

1.scenario

Assume that the ik_demo index has the following records, as shown in the table below

According to this example, “guo” is different in different contexts, and the second sentence “China” is a single word, which should be returned in a separate search for “guo”.

2.elasticsearch-analysis-ik

3.Set up the mapping

curl -XPUT 'http://localhost:9200/ik_demo/_mapping? pretty' -H 'Content-Type: application/json' -d ' { "properties" : { "message" : { "type" : "text", "analyzer" : "ik_max_word", "search_analyzer" : "ik_smart", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256}}}}}'Copy the code

4.Add data

curl -XPUT 'http://localhost:9200/ik_demo/_doc/1? pretty' -H 'Content-Type: application/json' -d ' { "message": "I love the People's Republic of China "}' curl -XPUT 'http://localhost:9200/ik_demo/_doc/2? Pretty '-h' Content-type: Application /json' -d '{"message": "I am from China "}'Copy the code

5.search

In the word segmentation process, China will be used as a word, so when searching “country” alone, the record “I come from Zaoyang, Hubei, China” will not be found.

curl -XGET 'http://localhost:9200/ik_demo/_search? Pretty = true '-h' the content-type: application/json '3-d' {" query ": {" match" : {" message ":" countries "}}} 'Copy the code

Return result:

{ "took":3, "timed_out":false, "_shards":{ "total":5, "successful":5, "skipped":0, "failed":0 }, "Hits" : {" total ": {" value" : 1, the "base" : "eq"}, "max_score" : 0.2876821, "hits" : [{" _index ":" ik_demo ", "_type" : "_doc." "_id" : "1", "_score" : 0.2876821, "_source" : {" message ":" I love the People's Republic of China "}}}}]Copy the code

Here is only a brief introduction to the use of the specific word segmentation content can refer to: github.com/medcl/elast…

Three,Synonyms, synonyms search

1.scenario

Assume the following entries in the index synonym_demo, as shown in the table below

Cat and cat actually mean the same thing. If you want to search for a cat, you should search for [1, 2] records. How to deal with it?

2.elasticsearch-dynamic-synonym

3.Create indexes

In ES_HOME/config/analysis/synonyms. TXT add synonyms: the cat, the cat

Create index:

curl -XPUT 'http://localhost:9200/synonym_demo? pretty' -H 'Content-Type: application/json' -d ' { "settings": { "analysis": { "analyzer": { "my_dynamic_synonym": { "type": "custom", "tokenizer": "whitespace", "filter": ["my_synonym"] } }, "filter": { "my_synonym" : { "type" : "synonym", "expand": true, "ignore_case": true, "synonyms_path" : "analysis/synonyms.txt" } } } }, "mappings":{ "properties" : { "message" : { "type" : "text", "analyzer" : "ik_max_word", "search_analyzer" : "my_dynamic_synonym", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } }'Copy the code

4.Add data

curl -XPUT 'http://localhost:9200/synonym_demo/_doc/1? pretty' -H 'Content-Type: application/json' -d '{ "message": "I like cats"} 'curl - XPUT' http://localhost:9200/synonym_demo/_doc/2? pretty' -H 'Content-Type: application/json' -d '{ "message": "I like the cat"} 'curl - XPUT' http://localhost:9200/synonym_demo/_doc/3? Pretty '-h' Content-type: application/json' -d '{"message": "I like dogs "}'Copy the code

5.search

Since cat and cat are synonyms, a search for cat should return cat as well.

curl -XGET 'http://localhost:9200/synonym_demo/_search? pretty=true' -H 'Content-Type: application/json' -d ' {"query":{"match":{"message":"cat"}}}'Copy the code

Return result:

{ "took" : 3, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : {" total ": {" value" : 2, the "base" : "eq"}, "max_score" : 1.2039728, "hits" : [{" _index ": "Synonym_demo _type", "" :" _doc ", "_id" : "1", "_score" : 1.2039728, "_source" : {" message ":" I like cats "}}, {" _index ": "Synonym_demo _type", "" :" _doc ", "_id" : "2", "_score" : 1.2039728, "_source" : {" message ":" I like the cat "}}}}]Copy the code

Four,Reference documentation