What is the ElasticSearch

todo

The deployment of

Compressed package deployment

Copy the es file from elasticSearch. yml to elasticsearch.yml

Docker – compose deployment

version: '3'
services:
  elasticsearch:
    ##image:tag image is the image name on hub.docker.com. Tag is the version number
    image: Elasticsearch: 7.9.3
    container_name: es
    ## Loaded environment variables
    environment: 
         ## Cluster name
      - cluster.name=my-application
      - node.name=node
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "cluster.initial_master_nodes=node,node1,node2" The cluster initializes the master node
      - "discovery.seed_hosts=elasticsearch,elasticsearch1,elasticsearch2" 
    ulimits:
      memlock:
        soft: - 1
        hard: - 1
    Port mapping between host and container, ports need to be exposed
    ports:
      - 9200: 9200 
      - 9300: 9300
    networks:
      - esnet
    ## Data volume mapping can be multiple mappings es_data alias on the left and intra-container address on the right
    Docker volume inspect ${user}_es_data
    Docker volume inspect root_es_data
    volumes: 
      - es_data:/usr/share/elasticsearch/data
      - es_plugin:/usr/share/elasticsearch/plugins
  elasticsearch1:
    image: Elasticsearch: 7.9.3
    container_name: es1
    environment:
      - cluster.name=my-application
      - node.name=node1
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "cluster.initial_master_nodes=node,node1,node2"
      - "discovery.seed_hosts=elasticsearch,elasticsearch1,elasticsearch2"
    ulimits:
      memlock:
        soft: - 1
        hard: - 1
    ports:
      - 9201: 9200
      - 9301: 9300
    networks:
      - esnet
    volumes:
      - es1_data:/usr/share/elasticsearch/data    
      - es_plugin:/usr/share/elasticsearch/plugins
  elasticsearch2:
    image: Elasticsearch: 7.9.3
    container_name: es2
    environment:
      - cluster.name=my-application
      - node.name=node2
      - bootstrap.memory_lock=true
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "cluster.initial_master_nodes=node,node1,node2"
      - "discovery.seed_hosts=elasticsearch,elasticsearch1,elasticsearch2"
    ulimits:
      memlock:
        soft: - 1
        hard: - 1
    ports:
      - 9202: 9200
      - 9302: 9300
    networks:
      - esnet
    volumes:
      - es2_data:/usr/share/elasticsearch/data
      - es_plugin:/usr/share/elasticsearch/plugins
  kibana:
    image: Kibana: 7.9.3
    container_name: kibana
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - "5601:5601"
    networks:
      - esnet
volumes:
  es_data:
    driver: local
  es1_data:
    driver: local
  es2_data:
    driver: local
  es_plugin:
    driver: local
networks: ## The concept of a LAN for communication between containers
  esnet:
Copy the code

ElasticSearch architecture principles

The basic concept

The index

• A collection that brings together the same kind of data is the equivalent of a library in a traditional relational database

type

• 7.0 deprecated default type _doc version 8.0 officially excludes tables equivalent to traditional relational databases

Document/doc

• That is, one row of data is a DOC

Field/field

• A field in doc

shard

The index is divided into several primary shards, that is, the data is distributed equally among different nodes. Each primary shard has several copies, namely replica shard. The primary shard and replica shard do not exist in the same node. To meet the fault tolerance of partitions, which shard data is stored according to hash(routing) % number_of_primary_shards Routing default is _id. The number of shards cannot be changed after being set due to consistent hashCopy the code

Reading and writing process

Write requests:

• Request any node (coordination node) The coordination node routes to the specified Primary shard based on the ID -> Primary Shard distributed to the replica shard-> Replica Shard Sends information back to the primary shard Shard ->primary Shard notifies the coordination node when all write signals are received

Read requests:

• Requests to any node (coordination node) without ID can be routed to all shards. The data returned by all shards is put together (the result set only contains docId and all sorted field values) and the docId to be returned to the client is determined And then go to the shard that corresponds to the docId and get the real data back

Inverted index

Divide a sentence into several words or words with a word divider, in which doc each word is distributed

A forward index is a document whose ID is key. The number of occurrences of each keyword is recorded in the table. The word information of each document in the table is scanned until all documents containing the keyword are found

Docb 内容 提 要 : docA_id 内容 提 要 : docA_id 内容 提 要 :... DocB_id Big data 1 Center 1 Search for key word intelligence cloud, query all doc docA,docBCopy the code

DocA_id,docB_id,docB_id, docA_id,docB_id,docB_id... DocA_id is returned for keyword intelligence cloud retrieval. DocA_id is returned for keyword big data retrieval. DocB_id is returned for keyword center retrievalCopy the code

Word segmentation is

Ik_max_word (fine-grained) for indexing and IK_smart (coarse-grained) for searching

Ik_max_word will split the text in the most fine-grained way. For example, it will split “Great Hall of the People of the People’s Republic of China” into words like “People’s Republic of China, People’s Republic of China, China, Chinese, People’s Republic, People’s Republic, Great Hall, general Assembly, hall and so on.

Ik_smart will do the coarsest splitting, such as splitting the “Great Hall of the People of the People’s Republic of China” into the People’s Republic of China and the Great Hall of the People.

pinyin

Keep_first_letter When this option is enabled, for example, Andy Lau > LDH, default: true Keep_separate_first_letter When this option is enabled, keep the first letter separately, for example, Andy Lau > L, D, H, default: False limit_first_LETTER_length Specifies the maximum length of the first_letter result. Default value: 16 keep_full_pinyin When this option is enabled, for example, Andy Lau > [liu, de, Hua], the default value is: True keep_joinED_full_pinyin When this option is enabled, for example, Liudehua > [liudehua], default value: false keep_none_Chinese Keep non-Chinese letters or numbers in the result. Default value: Keep_none_chinese_together Keep_none_chinese_together. Default: true, e.g. DJ musician - > DJ, Yin, Yue, jia, when set to false, e.g. DJ musician - > D, J, Yin, Yue, jia, note: Keep_none_chinese must be enabled to keep the first letter of keep_none_chinese_in_first_letter non-Chinese, for example: Andy Lau AT2016- > ldhat2016. Default value: Keep_none_chinese_in_joined_full_pinyin keep_none_chinese_in_joined_full_pinyin keep_none_chinese_in_joined_full_pinyin False none_chinese_pinyin_tokenize Breaks non-Chinese letters into separate pinyin entries if they pinyin, default: true, e.g. Liudehuaalibaba13zhuanghan - >, liu DE hua, a, li, ba, ba, 13, zhuang, han, note: Keep_none_chinese and keep_none_chinese_together should be enabled first keep_original When this option is enabled, the original input is kept. Default value: false lowercase non-Chinese letters. Default value: True TRIM_WHITespace Default: true remove_DUPLicated_term Duplicates will be removed to save the index when this option is enabled, for example: > de of de, default: false, note: location-related queries may be affectedCopy the code

setting

{
    "settings": {"analysis": {"char_filter": {},// Character filter before word segmentation
            "tokenizer": {},// Create a custom tokenizer to modify the properties of existing tokenizers
            "filter": {},// After word segmentation, word segmentation filter
            "analyzer": {}// Customize the parser, combine the above three to assemble your own segmentation}}}Copy the code

{
	"settings": {
		"index.max_ngram_diff": 20."number_of_shards": "5"."number_of_replicas": "2"."analysis": {
			"analyzer": {
				"pinyin_analyzer": {
					"tokenizer": "standard_pinyin"
				},
				"first_py_letter_analyzer": {
					"tokenizer": "first_py_letter"
				},
				"full_pinyin_letter_analyzer": {
					"tokenizer": "full_pinyin_letter"}},"tokenizer": {
				"standard_pinyin": {
					"keep_joined_full_pinyin": "true"."keep_first_letter": "true"."keep_separate_first_letter": "false"."lowercase": "true"."type": "pinyin"."limit_first_letter_length": "16"."keep_original": "true"."keep_full_pinyin": "true"."keep_none_chinese_in_joined_full_pinyin": "true"
				},
				"first_py_letter": {
					"type": "pinyin"."keep_first_letter": true."keep_full_pinyin": false."keep_original": false."limit_first_letter_length": 16."lowercase": true."trim_whitespace": true."keep_none_chinese_in_first_letter": false."none_chinese_pinyin_tokenize": false."keep_none_chinese": true."keep_none_chinese_in_joined_full_pinyin": true
				},
				"full_pinyin_letter": {
					"type": "pinyin"."keep_separate_first_letter": false."keep_full_pinyin": false."keep_original": false."limit_first_letter_length": 16."lowercase": true."keep_first_letter": false."keep_none_chinese_in_first_letter": false."none_chinese_pinyin_tokenize": false."keep_none_chinese": true."keep_joined_full_pinyin": true."keep_none_chinese_in_joined_full_pinyin": true
				}
			}
		}
	}
}
Copy the code

Index Static configuration △△△index. number_of_SHards Number of index fragments. At the ES level, you can set the maximum number of index fragments using the es.index.max_number_of_shards attribute. The default value is 1024. The default value of index.number_of_shards is math.min (es.index.max_number_of_shards,5), so the default value is usually 5. Index. shard. Check_on_startup Whether a fragment should be checked for damage before being opened. When corruption is detected, it prevents shards from being opened. Optional value: false: no detection. Checksum: checks only physical structures. True: checks for physical and logical damage, which consumes CPU. Fix: similar to false, deprecated after version 7.0. Default value: false. Index. Codec Compression algorithm for data storage. The default value is LZ4, and the optional value is best_COMPRESSION, which provides a better compression ratio than LZ4 (that is, takes up less disk space, but has lower storage performance than LZ4). Index. routing_partition_size Number of route partitions. If this parameter is set, the routing algorithm is as follows: (Hash (_routing) + hash(_id) % index. routing_PARTtion_size) % number_of_shards. If this value is not set, the routing algorithm is hash(_routing) % number_of_shardings, and the default value of _routing is _id. Index dynamic configuration △△△△index.number_of_replicas Number of index replicas. The default value is 1. The value must be greater than or equal to 0 and can be changed after the index is created. Index. auto_expand_replicas Indicates whether the number of replicas automatically expands. The value can be set to 0-5 or 0-all. Index. refresh_interval How often the refresh operation is performed to make the latest changes to the index visible to the search. The default value is 1s. You can set it to -1 to disable the refresh. Max_result_window Controls the total number of pages searched. The value of from + size cannot exceed this value. The default value is 10000. Max_inner_result_window Maximum value from from+ size, used to control top Aggregations. The default value is 100. Internal and top hit aggregations take up heap memory, and the time is proportional to from + size, which limits memory. Index. max_REScore_window The maximum window_size of rescore requests in rescore searches. Max_docvalue_fields_search Maximum number of enabled DOC_VALUES fields in a query. The default value is 100. Index. max_script_fields Maximum number of script_fields allowed in a query. The default value is 32. Max_ngram_diff Maximum allowable difference between min_gram and max_gram of NGramTokenizer and NGramTokenFilter. The default value is 1. Max_shingle_diff Maximum allowable difference between max_SHingLE_SIZE and min_SHingLE_size for ShingleTokenFilter. The default value is 3. Read_only Indicates whether the index data and index metadata are read-only. If this parameter is set to true, the index data and index metadata cannot be modified. Read_only_allow_delete is similar to index.blocks. Read_only except that deletion is allowed. Set index.blocks. Read to true to disable reading of index data. Set index.blocks. Write to true to disable writes to index data. (for index data, not index metadata) Index. blocks. Metadata is set to true, meaning that read and write is not allowed on index metadata. Max_refresh_listeners Maximum number of listeners available for index refresh per shard. These listeners are used to implement REFRESH = WAIT_for. Index. highlight.max_analyzed_offset Highlights the maximum number of characters requested for analysis. This setting only applies to text fields without offsets or TERM Vectors. By default, this setting is not set in 6. X. The default value is -1. Index. max_terms_count The maximum number of terms that can be used in terms queries. Default value 65536. Index. The routing. The allocation. The enable allocation mechanism, the main solution is how to put the index on the ES in the cluster in which nodes distribution shard (for example in the shard Node1 is created, on the other nodes to create replication fragmentation). For example, if a new node is added to the cluster, the cluster has 4 nodes instead of 3: all All types of shards can be reassigned, by default. Primaries allow allocation of Primary sections only. New_primaries allow allocation only of newly created primary sections. None All shards are not allowed to be allocated. The index. The routing. Rebalance. Enable indexing shard to balance mechanism. Possible values are as follows: All The default value, allowing rebalancing for all fragments. Primaries allow rebalancing of primary sections only. Replicas only allows you to rebalance replicated shards. None Does not allow rebalancing of any shards index.gc_deletes Deletes the period during which documents can survive. The default is 60s. Index. max_regex_length Specifies the length of an ongoing expression for an ongoing expression query (regex query). The default value is 1000. Index.default_pipeline The default pipeline aggregator.Copy the code

mapping

{
	"mappings": {
		"properties": {
			"field1": {
				"type": "keyword"
			},
			"field2": {
				"type": "text"
			},
			"field3": {
				"type": "text"."analyzer": "ik_max_word"."search_analyzer": "ik_smart"}}}}Copy the code

The election

Use zK elections

Zk creates a path for a node that is qualified to be master. If the node succeeds in being master and fails, watch the path

Own election

Split brain: it is possible that a single node elects multiple valid masters, causing the cluster to have multiple masters. The following elects a single node to select its own master

ElasticSearch. Class = > Command. The class: the main method to start the Command. The class = > EnvironmentAwareCommand. Class: execute() EnvironmentAwareCommand.class=>ElasticSearch.class: execute() ElasticSearch.class=>Bootstrap.class: init() Bootstrap.class=>Node.class: start()Copy the code

// Node.class
public Node start(a) throws NodeValidationException {... . discovery.start(); . . discovery.startInitialJoin(); . .Copy the code

discovery.start()

protected void doStart(a) {
       DiscoveryNode localNode = transportService.getLocalNode();
       assertlocalNode ! =null;
       synchronized (stateMutex) {
           // set initial state
           assert committedState.get() == null;
           assertlocalNode ! =null;
           ClusterState.Builder builder = ClusterState.builder(clusterName);
           ClusterState initialState = builder
               .blocks(ClusterBlocks.builder()
                   .addGlobalBlock(STATE_NOT_RECOVERED_BLOCK)
                   .addGlobalBlock(noMasterBlockService.getNoMasterBlock()))
               .nodes(DiscoveryNodes.builder().add(localNode).localNodeId(localNode.getId()))
               .build();
           committedState.set(initialState);
           clusterApplier.setInitialState(initialState);
           nodesFD.setLocalNode(localNode);
           joinThreadControl.start();//**running.set(true)**
       }
       zenPing.start();
   }
Copy the code

discovery.startInitialJoin();

//ZenDiscovery.class
public void startNewThreadIfNotRunning(a) {
           assert Thread.holdsLock(stateMutex);
           if (joinThreadActive()) {
               return;
           }
           threadPool.generic().execute(new Runnable() {
               @Override
               public void run(a) {
                   Thread currentThread = Thread.currentThread();
                   if(! currentJoinThread.compareAndSet(null, currentThread)) {
                       return;
                   }
                   // Spin until the above is set to true
                   while (running.get() && joinThreadActive(currentThread)) {
                       try {
                           innerJoinCluster();
                           return;
                       } catch (Exception e) {
                           logger.error("unexpected error while joining cluster, trying again", e);
                           // Because we catch any exception here, we want to know in
                           // tests if an uncaught exception got to this point and the test infra uncaught exception
                           // leak detection can catch this. In practise no uncaught exception should leak
                           assertExceptionsHelper.reThrowIfNotNull(e); }}// cleaning the current thread from currentJoinThread is done by explicit calls.}}); }Copy the code

innerJoinCluster();

private void innerJoinCluster(a) {
        DiscoveryNode masterNode = null;
        final Thread currentThread = Thread.currentThread();
        nodeJoinController.startElectionContext();
        while (masterNode == null && joinThreadControl.joinThreadActive(currentThread)) {
            masterNode = findMaster();
        }

        if(! joinThreadControl.joinThreadActive(currentThread)) { logger.trace("thread is no longer in currentJoinThread. Stopping.");
            return;
        }
        // If the election result is oneself
        if (transportService.getLocalNode().equals(masterNode)) {
        // More than half of the results become a cluster master requiredJoins are required for a few more node joins
            final int requiredJoins = Math.max(0, electMaster.minimumMasterNodes() - 1); // we count as one
            logger.debug("elected as master, waiting for incoming joins ([{}] needed)", requiredJoins);
            // Wait for other nodes to return election results
            nodeJoinController.waitToBeElectedAsMaster(requiredJoins, masterElectionWaitForJoinsTimeout,
                    new NodeJoinController.ElectionCallback() {
                        @Override
                        public void onElectedAsMaster(ClusterState state) {
                            synchronized(stateMutex) { joinThreadControl.markThreadAsDone(currentThread); }}@Override
                        public void onFailure(Throwable t) {
                            logger.trace("failed while waiting for nodes to join, rejoining", t);
                            synchronized(stateMutex) { joinThreadControl.markThreadAsDoneAndStartNew(currentThread); }}}); }else {// The elected master sends join message to the elected node when it is not itself
            // process any incoming joins (they will fail because we are not the master)
            nodeJoinController.stopElectionContext(masterNode + " elected");

            // send join request
            final boolean success = joinElectedMaster(masterNode);

            synchronized (stateMutex) {
                if (success) {
                    DiscoveryNode currentMasterNode = this.clusterState().getNodes().getMasterNode();
                    if (currentMasterNode == null) {
                        Post 1.3.0, the master should publish a new cluster state before acking our join request. We now should have
                        // a valid master.
                        logger.debug("no master node is set, despite of join request completing. retrying pings.");
                        joinThreadControl.markThreadAsDoneAndStartNew(currentThread);
                    } else if (currentMasterNode.equals(masterNode) == false) {
                        // update cluster state
                        joinThreadControl.stopRunningThreadAndRejoin("master_switched_while_finalizing_join");
                    }
                    joinThreadControl.markThreadAsDone(currentThread);
                } else {
                    // Select another node as master if the selected node does not elect itself as master.
                    // reelect masterjoinThreadControl.markThreadAsDoneAndStartNew(currentThread); }}}}Copy the code

findMaster();

//ZenDiscovery.class
private DiscoveryNode findMaster(a) {
        // Ping other nodes to determine the number of nodes that can be connected to this node. Yml discovery.seed_hosts to obtain the ping list
        List<ZenPing.PingResponse> fullPingResponses = pingAndWait(pingTimeout).toList();
        if (fullPingResponses == null) {
            return null;
        }
        // The current node
        final DiscoveryNode localNode = transportService.getLocalNode();
        fullPingResponses.add(new ZenPing.PingResponse(localNode, null.this.clusterState()));
        / / filter according to the configuration is rejecting the master node configuration file true | false
        final List<ZenPing.PingResponse> pingResponses = filterPingResponses(fullPingResponses, masterElectionIgnoreNonMasters, logger);
        // Remove the existing master node from activeMasters because ping itself is meaningless
        List<DiscoveryNode> activeMasters = new ArrayList<>();
        for (ZenPing.PingResponse pingResponse : pingResponses) {
            if(pingResponse.master() ! =null&&! localNode.equals(pingResponse.master())) { activeMasters.add(pingResponse.master()); }}// Set Node. matser=true to qualify as master
        List<ElectMasterService.MasterCandidate> masterCandidates = new ArrayList<>();
        for (ZenPing.PingResponse pingResponse : pingResponses) {
            if (pingResponse.node().isMasterNode()) {
                masterCandidates.add(newElectMasterService.MasterCandidate(pingResponse.node(), pingResponse.getClusterStateVersion())); }}// If the master node is empty, let the standby node run
        if (activeMasters.isEmpty()) {
        	// Determine whether to meet the minimum number of masters configured for min_master_nodes
            if (electMaster.hasEnoughCandidates(masterCandidates)) {
                final ElectMasterService.MasterCandidate winner = electMaster.electMaster(masterCandidates);
                // The master can also be itself because it is the master of another node
                return winner.getNode();
            } else {
                return null; }}else {
        // Select activeMasters with the smallest nodeID, as the master cannot have its own
            returnelectMaster.tieBreakActiveMasters(activeMasters); }}Copy the code

Compared with traditional relational data

todo

Usage scenarios

todo

How to use

sql

limitationsOnly data is returned, like Score and Heightlight, which are not returned as true relational data

dsl

• Match will query the search terms in terms
• Match_phrase Searches for words in slop order
• match_phrase_prefix
• Operator concatenates the word segmentation condition with and and defaults to OR
• Boost retrieval weights are related to scores
• Minmum_should_match Search term relevancy
• For example, if ab slop=1, a@b can also be returned

todo

spring-data-elasticsearch

todo

highRestClient

todo