Elasticsearch Data migration

This is the 13th day of my participation in the August More Text Challenge. For details, see:August is more challenging

Today, we need to migrate data from ES database to new cluster. We need to migrate data from ES database to new cluster.

1, install,

Officially, there are two ways to install, one is NPM, the other is Docker. We use one of these NPM installations here.

Install NPM # #
yum install npm
Install Elasticdump globally
npm install elasticdump -g
Copy the code

Why install globally? If you do not want to install elasticdump globally, remove the -g parameter. If you want to use Elasticdump, you need to go to the installation path. A global installation is recommended here.

2. Verify the installation

elasticdump --version
Copy the code

If the correct version information is displayed, congratulations on your installation. But if you get an error message like this:

SYNTAXERROR: UNEXPECTED IDENTIFIER at createScript (vm.js:56:10) at Object.runInThisContext (vm.js:97:10) at Module._compile (module.js:549:28) at Object.Module._extensions.. js (module.js:586:10) at Module.load (module.js:494:32) at tryModuleLoad (module.js:453:12) at Function.Module._load (module.js:445:3) at Module.require (module.js:504:17) at require (internal/module.js:20:19) at Object.<anonymous> (/opt/node_modules/elasticdump/elasticdump.js:3:28)Copy the code

This error is caused by an early nodeJS version, so we need to upgrade the nodeJS version.

3. Upgrade NodeJS

If your Elasticdump failed, skip this step and go to the next step.

npm install -g n
n latest
Copy the code

After the upgrade is complete, try again

elasticdump --version
Copy the code

If the version information is present, then the next step is a big one.

4. Copy the mapping

In the last section of how Docker customizes host files, we mentioned that our databases are deployed on the Intranet, so the original ES and the new ES cannot be directly connected. So how do you migrate?

• The IP address of the company can access the public network of the two clusters (the public network can permit a specific IP address).

  You can install Elasticdump on your company’s servers by executing the following command

  /db_customer --output= http://new ES IP:9200/db_customer --type=mapping
  Copy the code

• Only the Intranet is accessible, and the public network is not open at all

  In this case, you have to export the original ES data into a file, and then copy the file to the new ES cluster, and then perform the import operation.

  # Original es server/db_customer --output=/data/db_customer.json --type=mapping
  Copy the code

  Copy the /data/ my_index_Mapping. json file to the new ES

  # New ES server/db_customer.json --output= http://new ES IP:9200 --type=mapping
  Copy the code

  This approach is relatively cumbersome, and it is best to write scripts in combination with commands. If your index is more than that it is not tired to death.

• An ES cluster with a password

  Take the first way.

  • Create authentication file auth.ini. The content is as follows

    user=elastic
    password=password
    Copy the code

  • Execute the command

    Ini --input= http://original ES IP:9200/db_customer --output= http://new ES IP:9200/db_customer --type=mapping
    Copy the code

  Some people say that you can enter the user name and password directly in the address bar, just like the URL to connect to the database, which I tested in the browser is ok. You can try it out.

  Dump --input=http://elastic:password @original ES IP:9200/db_customer --output=http://elastic:password @new ES IP:9200/db_customer --type=mapping
  Copy the code

5. Copy data

Change the type to data.

Ini --input= http://original ES IP:9200/db_customer --output= http://new ES IP:9200/db_customer --type=data
Copy the code

6. Other types

Write in the last

According to visual observation, 1s can generate 100 documents. My documents are relatively small and there is not much data, because I am the public network import of two servers, which may also be limited by bandwidth. If the amount of data is large, I do not know how the effect is. I will have a chance to see the test in the future.

Here we are. Give us a thumbs-up before we go!

Pay attention to WangScaler, wish you promotion, pay rise, no bucket!