“This is the 19th day of my participation in the Gwen Challenge in November. Check out the details: The Last Gwen Challenge in 2021”
I. Failover
When only one node is running in a cluster, that means there is a single point of failure — no redundancy. Fortunately, we only need to start one more node to prevent data loss. When you start a second node on the same machine, as long as it has the same cluster.name configuration as the first node, it will automatically discover the cluster and join it. However, when starting nodes on different machines, you need to configure a list of connected unicast hosts in order to join the same cluster. Unicast discovery is configured to prevent nodes from inadvertently joining the cluster. Only nodes running on the same machine automatically form a cluster.
If the second node is started, our cluster will have a two-node cluster: all primary and replica shards have been allocated
Use the ElasticSearch-head plug-in to check the cluster status
Node-1: The three main shards are normal. The thick box is the main shard
Node-2: When the second node is added to the cluster, three replica shards are assigned to the node — one for each master shard. This means that when any node in the cluster fails, our data is intact. All newly indexed documents are stored on the master shard and then copied in parallel to the corresponding replica shard. This ensures that documents can be retrieved from both master and replica shards.
Two, horizontal expansion
How do we scale up our growing applications on demand? When the third node is started, our cluster will have a three-node cluster: shards are redistributed to spread the load
Use the ElasticSearch-head plug-in to check the cluster status
Cluster health value: Green (6 of 6) : indicates that all six shards (including three primary shards and three replica shards) are running properly.
One shard on Node 1 and one shard on Node 2 has been migrated to the new Node 3, so there are now two shards on each Node instead of three. This means that the hardware resources (CPU, RAM, I/O) of each node will be shared by fewer shards and the performance of each shard will be improved. Sharding is a full-featured search engine that has the ability to use all resources on a node. Our index with six shards (3 master shards and 3 replica shards) can be expanded to a maximum of six nodes, with one shard on each node, and each shard has all the resources of its node.
3. What if you want to expand more than 6 nodes?
The number of master shards is determined when the index is created. In effect, this number defines the maximum amount of data that the index can store. (The actual size depends on your data, hardware, and usage scenario.) However, read operations — searching and returning data — can be handled by both master and replica shards, so the more replica shards you have, the higher throughput you will have.
The number of replica shards can be dynamically adjusted on a running cluster, and we can scale the cluster on demand. Let’s increase the number of copies from the default of 1 to 2
{
“number_of_replicas” : 2
}
Use the ElasticSearch-head plug-in to check the cluster status
Of course, simply adding more replica shards to a cluster with the same number of nodes will not improve performance because each shard will get fewer resources from the nodes. You need to add more hardware resources to improve throughput.
But the larger number of replica fragments increases data redundancy: with the node configuration above, we can lose two nodes without losing any data.