With the continuous development of Internet technology and the continuous iteration and update of technology, there will be many new products emerging, and we also need to keep learning to master these products.

I don’t know if any of your projects already use Elasticsearch, but I’m sure you’ll hear “Search uses ES”, “logs are put in ES via message-oriented middleware”…

For those of you who don’t know about Elasticsearch, you will be surprised to see that the es is not yet available.

Elasticsearch is one of the most common parts of Elasticsearch. I hope it will help you.

Elasticsearch

What is Elasticsearch?

Elasticsearch is a jSON-based distributed search and analysis engine based on the restful Web interface. Elasticsearch is an open source project based on the Apache protocol. It is the most popular enterprise search engine.

What is ELK?

ELK is an acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analysis engine. Logstash is a server-side data processing pipeline that captures data from multiple sources at the same time, transforms it, and sends it to a “repository” such as Elasticsearch. Kibana allows users to visualize data using graphs and charts in Elasticsearch.

Basic concepts of Elasticsearch

Cluster – Cluster

The Elasticsearch cluster consists of one or more nodes, including a primary node that can be elected, and primary and secondary nodes that are internal to the cluster.

The concept of Elasticsearch is decentralized, which literally means no central node. This is for the outside of the cluster because the Elasticsearch cluster is logically integrated from the outside. You can communicate with any node in the cluster as well as the entire Elasticsearch cluster.

Shards – shard

Elasticsearch provides the concept of subdividing an index into shards to address the problem of exceeding the hardware limits of a single node when storing large amounts of data. Sharding is divided into master sharding and copy sharding.

Primary shard: Represents the primary shard of an index. An index can be divided into multiple primary shards.

Replica Shard: indicates the replica of the main index shard.

The Index – Index

Similar to database in mysql database. It stores several Document data with similar structure. Such as: customer index, order index, commodity index and so on.

Type – the Type

Similar to mysql database table, es can create type (table) in Index.

Note: In ElasticSearch 5.x and later, you can have multiple types in an Index. The concept of type has been weakened since elasticSearch6.x. Only one type can be found in an index. After 7.x, delete the type definition.

Document

Smallest unit of data in Elasticsearch. A Document is a piece of data, typically represented by a JSON data structure.

Field

Multiple columns in a document in ES correspond to each column in the mysql database

Query DSL

SQL statement similar to mysql, but in ES is the use of JSON format query statement, the technical term is called: QueryDSL

GET/PUT/POST/DELETE

The select of similar to mysql/update/delete… It can be called directly in a RESTFUL manner

The installation

Through the installation package

The official download address: www.elastic.co/cn/download…

Install Chinese word segmentation plug-in

In Windows, open the CMD window, and run the following command in the ElasticSearch – version \bin directory:

elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.2/elasticsearch-analysis-ik- 6.2.2.zip
Copy the code

Using Docker

The official provides the mirror library, from which you can directly download the desired version.

Downloading the Image Package

$ docker pull elasticsearch:6.4.3
# or
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:6.4.3
Copy the code

Modify virtual memory (or not in the test environment)

$ grep vm.max_map_count /etc/sysctl.conf
vm.max_map_count=262144
Copy the code

Start with the Docker command (cluster parameters can also be added to the formal environment)

$ docker run -p 9200:9200 -p 9300:9300 --name elasticsearch \
-e "discovery.type=single-node" \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-d elasticsearch:6.4.3
Copy the code

Install a word divider

Install the tokenizer to run inside the Docker container
$ docker exec -it elasticsearch /bin/bash
$ elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.4.0/elasticsearch-analysis-ik- 6.4.3.zip
Need a reboot
$ docker restart elasticsearch
Copy the code

access

The browser can be directly accessed through http:// machine IP address :9200/

Install Kibana

Version information must be consistent with Elasticsearch
docker pull kibana:6.4.3
Start with the docker command
docker run --name kibana \
-p 5601:5601 \
--link elasticsearch:es \
-e "elasticsearch.hosts=http://es:9200" \
-d kibana:6.4.3
Http://machine IP:5601
Copy the code

Common operations for Elasticsearch

Operations related to cluster status

  • Check the cluster health status

    GET /_cat/health? v

Status: green, yellow, red

Green: The primary shard and Replica shard in each index are active

Yellow: All the primary shards in each index are active, but some Replica shards are not active

Red: Not all index primary shards are active.

  • Check the node health status

    GET /_cat/nodes? v

  • View all indexes

    GET /_cat/indices? v

  • Viewing Fragment Information

    GET _cat/shards? v

Index related operations

Create indexes

Syntax: PUT index name {index configuration parameter}

The index name must be lowercase and cannot start with an underscore ‘_’, ‘-‘, or ‘+’.

Example:

PUT test_index
Copy the code

Modify the index

Command syntax: PUT index name /_settings{index configuration parameters}

Once an index is created, the number of primary shards cannot be changed, but the number of replica shards can be changed.

Example:

PUT test_index/_settings
{
  "number_of_replicas" : 2
}
Copy the code

Remove the index

Syntax: DELETE index name 1[, index name 2…

Example:

DELETE test_index
Copy the code

Document-related operations

The new document

Add documents to the index. Elasticsearch has automatic identification. If the index corresponding to the added document does not exist, the index is automatically created. If index exists but type does not, type is automatically created. If both index and type exist, the existing index and type are used.

  • PUT the grammar

    You can specify your own document ID

    Syntax: PUT Index name/Type name/Unique ID{field name: field value}

    Example:

    PUT test_index/test_type/1
{
   "name":"test_put",
   "remark":"test",
   "order_no":1
}
Copy the code

    Syntax: PUT index name/type name/unique ID/_create{field name: field value}

    _create: Forcibly added. If the Document ID already exists in Elasticsearch, an error will be reported using the force-add syntax. (Version conflict, document already exists)

  • POST the grammar

    Elasticsearch automatically generates the ID

    Command syntax: POST Index name/Type name {field name: field value}

    Example:

    POST test_index/test_type
{
   "name":"tes_post",
   "remark":"test",
   "order_no":4
}
Copy the code

Query the document

  • Query a single piece of data by ID

    Syntax of the command: GET Index name/type name/unique ID

    Example:

    GET test_index/test_type/1
Copy the code

  • Batch query

    You can use the metadata “_mget” to do more.

    Example:

    # the first GET _mget {" docs ": [{" _index" : "index name", "_type" : "type name", "_id" : "The only ID value"}, {}, {}} # second GET index name / _mget {" docs ": [{" _type" : "type name", "_id" : "The only ID value"}, {}, {}} # third GET index name/type name / _mget {" docs ": [{" _id" : "the only ID value"}, {" _id ":" the only ID value "}]}Copy the code

Update the document

Update within Elasticsearch does not actually modify the Document, but instead marks the original Document as deleted and creates a new Document to store the data. There are two ways to update:

  • Full quantity update

    Syntax: PUT Index name/Type name/Unique ID{Field name: Field value} The syntax of the PUT command is the same as that of the new PUT command.

  • Part of the update

    Syntax: POST Index name/Type name/unique ID/_update{doc:{field name: field value}}

    Example:

    POST test_index/test_type/1/_update
{
   "doc":{
      "name":" test_post_for_update"
   }
}
Copy the code

Delete the document

Note When Elasticsearch is deleted, Document is marked as deleted rather than physically deleted. The physical deletion operation is performed only when the Elasticsearch storage space is insufficient or free. Data marked as deleted will not be searched by the query.

Syntax: DELETE Index name/type name/unique ID

Example:

DELETE test_index/test_type/1
Copy the code

Introduction to common metadata

Metadata usually begins with an underscore ‘_’. When using the query document, the return result includes not only the data we saved, but also the metadata.

  • _index

    Represents the index name to which document data is stored.

  • _type

    It represents the document and the data is stored in the category in the index.

  • _id

    Represents the unique identity of the document. Use index, type, and ID to locate a unique document.

  • _source

    Represents JSON string information for the document.

  • _version

    Represents the version of the document. Each time the data changes, the version changes.

4. Data Search (DSL)

Elasticsearch provides a full JSON-based query syntax: DSL–Domain Specific Language where the query parameter is a piece of JSON.

Syntax format:

GET index name /_search {"command":{"parameter_name" : "parameter_value"}}Copy the code

Query all data

GET index name /_search {"match_all":{}}Copy the code

Match query -match query

Search terms after the word and any matching data is the search results

GET the index name / _search {" query ": {" match" : {" field name ":" search criteria "}}}Copy the code

Phrase query -match_phrase query

The query criteria are broken down, but the target data must contain all terms to be the result.

GET the index name / _search {" query ": {" match_phrase" : {" field name ":" search criteria "}}}Copy the code

Phrase query-term query

Query conditions are not segmented and are taken directly to match the target data.

# GET a single condition index name/type name / _search {" query ": {" term" : {" field name ":" search criteria "}}} # multiple conditions GET index name/type name / _search {" query ": {"terms" : {"terms" : [" search criteria 1", "search criteria 2"]}}}Copy the code

Range query -range query

Range search, similar to mysql between… The and…

Gte: greater than or equal to GT: greater than or equal to LTE: less than or equal to lt: less than or equal to

GET Index name /_search {"query" : {"range" : {" field name ": {"gt" : search condition 1," LTE ": search condition 2}}}}Copy the code

Multi-conditional compound query -bool

Multiple conditional queries, combined with bool, have several types in bool:

  • must: indicates that multiple conditions must be met.
  • must_not: indicates that multiple conditions must not be met.
  • should: indicates that one of multiple conditions can be met.
GET index name/type name /_search {"query": {"bool": {"must": [# multiple conditions in array must meet {"range": {" field name ": {"lt": }}}] condition, "must_not" : [# arrays of multiple conditions must not meet {" match ": {" field name" : "conditions"}}, {" range ": {" field name" : {" gte ": "Search criteria"}}}] "should" : [# arrays of multiple conditions in any one meet. {" match ": {" field name" : "conditions"}}, {" range ": {" field name" : {" gte ": "Search criteria"}}}]}}Copy the code

Aggregate query -aggs query

Es also provides rich aggregated queries, similar to groups, statistics, and so on in MySQL. Specific can view the official documentation: www.elastic.co/guide/en/el…

Five, the last

The basic concept of Elasticsearch is pretty much getting started, but there are still a lot of things to be aware of and some bugs to be found

In the follow-up, I will summarize some articles on the use of ES. If you are interested, you can follow it. You can also follow the public account “Hugh’s Whiteboard” and contact me to learn and discuss together. Come on!