preface

Welcome to our GitHub repository Star: github.com/bin39232820… The best time to plant a tree was ten years ago, followed by now

Tips

The interview guide series, which in many cases does not delve into the details, is a way for students to review their knowledge in the role of the interviewee, so I assume that most of the things you, as the interviewer, know.

www.processon.com/view/link/6…

This is the brain map address

Where t

Let’s take a look at Elasticsearch today and here’s a summary of the previous articles

• 2021-Java Backend Engineer Interview Guide (Introduction)
• 2021-Java Backend Engineer Interview Guide
• 2021-Java Backend Engineer Interview Guide -(Concurrency – Multithreading)
• 2021-Java Backend Engineer Interview Guide -(JVM)
• 2021-Java Backend Engineer Interview Guide -(MySQL)
• 2021-Java Backend Engineer Interview Guide -(Redis)

In fact, Es is used a lot, and if the volume is large, it is basically needed to use it, so it is necessary to master it, so let’s have a look

What is Elasticsearch

• Elasticsearch, based on Lucene. Distributed Restful real-time search and analysis engine
• Distributed real-time file storage where each field is indexed and searchable
• High scalability, can be extended to hundreds of servers, processing PB level of structured or unstructured data
• Elasticsearch for full text search, structured search, analysis/merge use

Elasticsearch features:

• Elasticsearch has no typical transaction meaning (non-transactional)
• Elasticsearch is a document-oriented database
• Elasticsearch does not provide authorization or authentication features

What is full text search and Lucene?

Full text search, inverted index

Full-text retrieval refers to the computer indexing program through scanning every word in the article, to establish an index for each word, indicating the number and position of the word in the article, when the user queries, the retrieval program will be based on the index established in advance to search, and the search results feedback to the user retrieval method. This process is similar to looking up words through the search word table in a dictionary. Full text search data in a search engine database.

lucene

Lucene is a JAR package that contains all the code for building inverted indexes and searching, including algorithms. We use Java development, the introduction of Lucene JAR, and then based on Lucene API to carry out development can be.

Talk about the core concepts of Elasticsearch, the ones we use a lot.

Near real time

Near real time, two meanings, with a small delay (about 1 second) from the time the data is written to the time it can be searched; Performing search and analysis based on ES can be done in seconds.

Cluster (Cluster)

A cluster contains multiple nodes, and which cluster each node belongs to is determined by a configuration (cluster name, default is ElasticSearch). For small to medium sized applications, starting with one node per cluster is normal

Node (Node)

A node in a cluster also has a name (randomly assigned by default). The node name is very important (when performing o&M operations). The default node is added to a cluster named “ElasticSearch”. Of course, a node can also form an ElasticSearch cluster.

Index (Index – database)

An index contains a bunch of document data with similar structures, such as a customer index, a category index, an order index, and an index with a name. An index contains many documents, and an index represents a class of similar or identical documents. For example, if you create a product index, a product index, you might have all of the product data in it, all of the product documents.

Type (Type – table)

Each index can have one or more types, type is a logical data classification in index, document under a type, all have the same field, such as blog system, there is an index, can define user data type, blog data type, comment data type.

Document (Document – line)

A document is the smallest data unit in ES. A document can be a customer data, a commodity classification data, and an order data, usually represented by JSON data structure. Multiple Documents can be stored in type under each index.

Field (Field – column)

Field is the smallest unit of Elasticsearch. A document contains multiple fields, each of which is a data field.

shard

A single machine cannot store a large amount of data. Es can shard the data in an index into multiple shards and store them on multiple servers. With the Shard, you can scale horizontally, store more data, distribute search and analysis operations across multiple servers, and improve throughput and performance. Each shard is a Lucene index.

replica

Any server may fail or go down at any time and shard may be lost. Therefore, multiple replica copies can be created for each shard. Replica can provide backup service when shard fails to ensure data is not lost. Multiple replicas can improve the throughput and performance of search operations. By default, each index has 10 shards, 5 primary shards, 5 replica shards, minimum high availability configuration. It’s two servers.

Elasticsearch optimistic concurrency control

Elasticsearch is distributed. When a document is created, updated, or deleted, the new version of the document is copied to other nodes in the cluster. Elasticsearch is both synchronous and asynchronous, meaning that these replication requests are sent in parallel and out of sequence to their destination. This requires a way to ensure that older versions of documents never overwrite new versions. When we mentioned the index, GET, and DELETE requests above, we indicated that each document has a _version number, which is incremented by one when the document is changed. Elasticsearch uses this _version to ensure that all changes are sorted correctly. When an old version comes after a new version, it is simply ignored. We take advantage of _version to ensure that data is not lost due to change conflicts. We can specify the version of the document to make the desired changes. If that version number is not the current one, our request has failed. Use version to ensure sequential consistency of concurrency

Let’s talk about the differences between text and keyword

• Text: When a field is to be searched in full text, such as Email content or product description, use text. After setting the text type, the field content is parsed, and the string is parsed into terms before generating an inverted index. Fields of type text are not used for sorting and are rarely used for aggregation.
• Keyword: The keyword type applies to indexed structured fields, such as email addresses, host names, status codes, and labels. If fields need to be filtered (for example, looking for published posts with the status attribute), sorted, aggregated. Fields of the keyword type can only be searched by exact value.

What are the main things that the query API returns

hits

The most important part of the response is hits, which contains the total field to indicate the total number of documents that were matched, and the HITS array also contains the first 10 data that were matched. Each result in the HITS array contains the _index, _type, and the _id of the document, which is added to the _source field, meaning that we will be able to use the entire document directly in our search results. Unlike other search engines, which only return document IDS, you need to retrieve the document separately. Each node has a _score field, which is the Relevance Score, which measures how well documents match queries. By default, the document with the most relevance is returned first; This means that it is sorted in descending _score order. In this case, we don’t specify any queries, so all documents have the same correlation, so all results have a median _score of 1. Max_score refers to the maximum _score of all documents matching queries.

took

Took tells us how many milliseconds the entire search request took.

shards

The _shards node tells us the total number of shards participating in the query and how many are successful and how many are failed. Normally we don’t want sharding to fail, but it can happen. If we suffer some major failure that causes both the master shard and the replication shard to fail, the shard’s data will not be able to respond to search requests. In this case, Elasticsearch will report sharding failed, but will continue to return results on the remaining shards.

timeout

The time_out value tells us whether the query timed out or not. Generally, search requests do not time out. If response speed is more important than complete results, you can define the timeout parameter as either 10 or 10ms (10 milliseconds), or 1s (1 second).

Talk about the shard&replica mechanism

• Index contains multiple shards
• Each shard is a minimal unit of work, carrying partial data, Lucene instances, complete indexing and request processing capabilities
• When nodes are added or removed, the SHard automatically balances load among nodes
• Primary shard and Replica Shard, each document must only exist in one primary shard and its corresponding replica shard, and cannot exist in multiple primary shards
• Replica Shard is a copy of the Primary Shard. It is responsible for fault tolerance and load of read requests
• The number of primary shards is fixed when the replica shard is created, and the number of replica shards can be changed at any time
• The default number of primary shards is 5, and the default replica is 1. By default, there are 10 shards, 5 primary shards and 5 replica shards
• The primary shard cannot be placed on the same node as the replica shard of one’s own replica shard (otherwise, the node breaks down and both the primary shard and the replica shard are lost, which cannot be fault-tolerant). However, the primary shard can be placed on the same node as the replica shard of another Primary shard

How does ES implement master elections?

Prerequisites:

• Only nodes that are candidate master nodes (master: true) can become master nodes.
• The purpose of the minimum number of primary nodes (MIN_master_nodes) is to prevent brain splitting.

For Elasticsearch, the ZenDiscovery module is responsible for Ping (the RPC used by nodes to find each other) and Unicast (the Unicast module contains a host list to control which nodes need to be pinged). The core entry for obtaining the Master node is findMaster. If the Master node is selected successfully, return the corresponding Master. Otherwise, return NULL.

The election process is roughly described as follows:

• Step 1: Ensure the number of candidate primary nodes is up to the specified value of elasticSearch. yml: discovery.zen.minimum_master_nodes;
• Step 2: Sort all the candidate primary nodes according to the nodeId dictionary. In each election, each node is ranked in the order of its known nodes, and then select the first node (the 0th bit), which is considered to be the master node.
• Step 3: If the number of votes for a node reaches a certain value (n/2+1) and the node elects itself, then the node is master. Otherwise, a new election will be held until the above conditions are met.

How to solve the brain split problem of ES cluster

Split cluster brain is a situation where 10 of the 20 Elasticsearch nodes select one master and the other 10 select another master.

When the cluster has at least three master candidates, the split brain problem can be solved by setting the minimum number of votes (discovery.zen.minimum_master_nodes) to exceed half of all candidate nodes. When the number of candidates is two, only the master candidate can be changed, and the other candidates can be used as data nodes to avoid the brain-splitting problem.

Let’s talk about the es write process

Elasticsearch uses the multi-shard mode. Routing rules are configured to divide data into multiple data subsets. Each data subset provides independent indexes and search functions. When a document is written, the document is sent to a specific Shard for indexing according to routing rules. That’s how you get distributed.

Each Index consists of multiple shards (5 by default). Each Shard has a master node and multiple replicas. The number of replicas is configurable. For each write, the write Request will first select which Shard to send according to the _routing rule. The Filed value can be set as the routing parameter in the Index Request. If no field value is set, the configuration in the Mapping will be used. The Hash value of _routing is used to select the Shard (in the OperationRouting class), and the Primary node of the Shard is found from the Meta of the cluster.

The request is then sent to the Primary Shard. After successful execution on the Primary Shard, the request is then sent to multiple Replica shards from the Primary Shard. After the write request is successfully executed on multiple Replica shards and returned to the Primary Shard, the write request is successfully executed and the result is returned to the client.

Why don’t you tell me how to write to a shard

In each Shard, the writing process is split into two parts, starting with Lucene and then TransLog.

When the write request reaches the Shard, write the Lucene file first, create the index, which is still in memory, then write TransLog, write TransLog data to disk, write disk success, return the request to the user. Here are a few key points:

CommitLog (select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database, select * from database. To avoid a large number of invalid records in TransLog, reduce the complexity and speed of recover, write Lucene first.

The Lucene memory was not searchable, but the object was transformed into a complete Segment by Refresh, and the scene was reopen but was searched again. This time was usually set to 1 second. So Elasticsearch is a Near Real Time (NRT) system for searching.

Every so often, after a long period of time, such as 30 minutes, Lucene will flush the new Segment generated in memory to disk. After the refresh, the index file will be persisted and the old TransLog will no longer be used.

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * If the machine goes down during this time, the data in memory is lost. If this happens, the data in memory can be recovered from TransLog, which by default refreshes the disk every 5 seconds. This still does not guarantee data security, as it is still possible to lose up to five seconds of TransLog data. This can be configured to increase the frequency of the TransLog flush disk to increase data reliability, up to a minimum of 100ms, but this is not recommended because it can have a significant impact on performance. Elasticsearch usually solves this problem through the replica mechanism. Even if the node where the master shard is located is down and data is lost for 5 seconds, it can still be recovered through a copy.

To summarize, the data is written to the buffer, and then refreshed to the OS cache every 1s. When the OS cache is refreshed, the data can be searched. Write data to a Translog file every 5 seconds (so that if the machine is down and there is no data in memory, at most 5 seconds of data will be lost). If the translog is large enough, or every 30 minutes by default, the commit operation will be triggered. Flush all buffer data into segment file.

Talk about the es update process

Lucene does not support Update of some fields, so you need to implement this function in Elasticsearch as follows:

• After the Update request, read the full Doc from the Segment or TransLog with the same id.
• Merge the full Doc of version V1 and partial field Doc in the request into a full Doc, and update the in-memory VersionMap. After the full Doc is retrieved, the Update request becomes an Index request.
• Lock.
• Read the maximum version number of this ID V2 from versionMap. If not, read it from Segment or TransLog, which is usually obtained from versionMap.
• Check whether the versions conflict (V1==V2). If yes, go back to the Update doc phase and run the Update doc command again. If there are no conflicts, the latest Add request is executed.
• In Index Doc stage, first add Version + 1 to get V3, and then add Doc to Lucene. In Lucene, the existing Doc iD under the same ID will be deleted first, and then add new Doc. After writing Lucene successfully, update the current V3 to versionMap.
• Release the lock and part of the update process ends

Describe in detail the process of ES search?

The search is performed as a two-phase process, Query Then Fetch;

The Query phase:

The query is broadcast to each shard copy (master shard or replica shard) in the index. Each shard performs a search locally and builds a priority queue matching documents of size from + size. PS: Filesystem Cache is queried during search, but some data is still stored in Memory Buffer, so the search is performed in near real time. Each shard returns the IDS and sorted values of all documents in its respective priority queue to the coordinator node, which merges these values into its own priority queue to produce a globally sorted list of results.

The Fetch phase:

The coordination node identifies which documents need to be retrieved and submits multiple GET requests to the related shard. Each shard loads and enriches the document, then returns the document to the coordination node if necessary. Once all documents have been retrieved, the coordination node returns the result to the client.

Talk about write consistency in ES

We send any add/delete operation, such as put /index/type/ ID, with a consistency parameter indicating what consistency we want. put /index/type/id? consistency=quorum

• One: The write operation can be performed as long as one primary shard is active
• All: The write operation can be performed only when all primary and Replica shards are active
• Quorum: The default value that requires a majority of all shards to be active and available before a write operation can be performed

Talk about elasticSearch deep paging and Scroll searching

The depth of the page

Deep pagination is the depth of the search, for example, page 1, page 2, page 10, page 20, is shallow; Page 10,000. Page 20,000 is pretty deep. Searching too deeply can cause performance problems, consuming memory and CPU. And for performance, ES does not support paging queries that exceed 10,000 data. So how to solve the problem caused by deep paging, we should avoid the deep paging operation (limit the number of paging pages), for example, only 100 pages of display can be provided at most, starting from page 101, after all, users will not search so deep, we usually search Taobao or Jingdong will only see 10 pages at most.

Scroll to search

Querying 10,000 + data at a time often affects performance because there is too much data. In this case, you can use scroll search, or scroll. A scroll search can first query some data, and then follow the query down. There is a scroll ID on the first query, which is equivalent to an anchor tag, and a subsequent scroll requires the scroll ID of the previous search, based on which the next search request is made. Each search is based on a historical snapshot of the data. If there is a change in the data during the query, it has nothing to do with the search.

How does ES improve performance in the case of large data volumes

filesystem

Es The maximum query speed is achieved when the fileSystem cache is accessed each time. Therefore, set the capacity of each queried data to 50% = the capacity of the fileSystem cache.

Data preheating

Data preheating means that hot data is manually queried in the background at regular intervals and flushed to fileSystem Cache

Hot and cold separation

This is similar to MySQL, where the hot data is indexed separately and three machines are assigned to keep the hot data indexed and the other machines to keep the cold data indexed, but the problem is that you have to know in advance which data is hot and which data is cold

Deep paging is not possible

Tell the product manager that your system does not allow you to turn pages that deep, and the deeper you turn pages by default, the worse your performance will be.

So it’s kind of like if you’re in an app and you’re constantly pulling down pages and pages and pages and pages and pages and pages, you can use the Scroll API

The end of the

I probably use es a little bit myself, just to do some searches, not BI, so what? It’s not that deep, but hopefully it’s helpful, so let’s review the queue

Daily for praise

Ok, everybody, that’s all for this article, you can see people here, they are real fans.

Creation is not easy, your support and recognition, is the biggest motivation for my creation, we will see in the next article

Wechat search “six pulse Excalibur program life” reply 888 I find a lot of information to you