Recently there were some changes, and an old project was entrusted to our group for maintenance. My heart was broken when I encountered such things, but I still had to force a smile and pat my chest and say there was no problem. What’s more sad is that mongo is also used in this project, which was built by myself instead of being managed by DBA. I have no choice but to learn Mongo by myself.

About mongo

There are three main ways of mongodb cluster construction, namely, the master-slave mode, Replica set mode and Sharding mode. Each of the three modes has its advantages and disadvantages and is suitable for different occasions. Replica set is the most widely used, the master-slave mode is less used now, and the Sharding mode is the most complete. However, configuration and maintenance are complicated.

So far, the project I have taken over uses Replica Set, so I mainly understand this model. The official website can be found here

It is necessary to know the following three types of roles in Replica Set mode:

The master node (Primary)

Receive all write requests and synchronize changes to all Secondary. A Replica Set can only have one Primary node. When the Primary fails, other Secondary or Arbiter nodes will re-elect a Primary node. The default read request is also sent to the Primary node for processing, and the client needs to modify the connection configuration to forward the read request to Secondary.

Copy node [Secondary]

Keep the same data set as the primary node. When the primary node fails, participate in the primary election.

Referee [Arbiter]

You don’t keep data, you don’t vote, you vote. Using Arbiter can reduce the hardware needs of data storage, Arbiter runs up almost no big hardware resource needs, but the important point is that in the production environment it and other data nodes do not deploy on the same machine.

Note that the number of nodes in the Replica Set of an automatic failover must be an odd number, so that there should be a majority in the primary voting to make the primary voting decision.

Set up the cluster

After understanding the basic concept, I started to try to build a cluster. In order to better understand, I specially found three test machines to deploy.

preparation

First, prepare three test machines:

10.100.1.101 Primary node (Master) 10.100.1.102 Secondary node (slave) 10.100.1.103 Arbitration point (arbiter)

Copy the code

Then there is the mongo installation package (since the online version is 3.4.2, so keep the same).

The curl - O https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-3.4.2.tgz

Copy the code

Install the mongo

/usr/local/mongodb is installed.

First unzip and rename:

Tar -zxvf mongodb-linux-x86_64-3.0.6. TGZ mv mongodb-linux-x86_64-3.4.2/ /usr/local/mongodb

Copy the code

Then create a few new files under /mongodb:

# mkdir -p conf # mkdir -p data # mkdir -p data

Copy the code

Note that the file path configured in the configuration file must exist, otherwise mongo will fail when starting, and Mongo will not be automatically generated when starting.

Next allocate create config file:

Mongodb_master.conf main node: mongodb_master.conf

#master.conf
dbpath=/usr/local/mongodb/data
logpath=/usr/local/mongodb/logs/mongodb.log
pidfilepath=/usr/local/mongodb/master.pid
directoryperdb=true
logappend=true
replSet=testdb
port=27017
oplogSize=100
fork=true
noprealloc=true

Copy the code

Backup node: vi mongodb_slave.conf

#slave.conf
dbpath=/usr/local/mongodb/data
logpath=/usr/local/mongodb/logs/mongodb.log
pidfilepath=/usr/local/mongodb/master.pid
directoryperdb=true
logappend=true
replSet=testdb
port=27017
oplogSize=100
fork=true
noprealloc=true

Copy the code

Vi Mongodb_arbiter.conf: mongodb_arbiter.conf

#arbiter.conf
dbpath=/usr/local/mongodb/data
logpath=/usr/local/mongodb/logs/mongodb.log
pidfilepath=/usr/local/mongodb/master.pid
directoryperdb=true
logappend=true
replSet=testdb
port=27017
oplogSize=100
fork=true
noprealloc=true

Copy the code

In actual scenarios, you can configure the parameters based on service requirements. Other parameters are for reference:

--quiet --port arg # specifies the service port number. The default port number is 27017. --bind_ip arg # specifies the service IP address. -- logPath arg # specify MongoDB log files -- logAppend # append -- pidFilepath arg # PID full path to File, If not set, there is no PID file --keyFile arg # cluster private key full path, Only valid for Replica Set --unixSocketPrefix arg # UNIX domain socket replacement directory (default: / TMP) --fork # run MongoDB as a daemon, Create server process --auth # enable validation -- CPU # display CPU utilization and IOwait periodically -- dbPath arg # specify database path --diaglog arg # diaglog option 0=off 1=W 2=R 3=both 7=W+some reads --directoryperdb # set each database to be saved in a separate directory --journal # enable the logging option, --journalOptions arg # enable log diagnostics --ipv6 # enable ipv6 -- jSONp # Allow JSONP access over HTTP (security implications) --maxConns arg # Default maximum number of simultaneous connections 2000 --noauth # disable authentication -- nohttpInterface # disable HTTP interface Disable port 27018 access by default -- NoPrealloc # Disable data file preallocation (often affecting performance) -- Noscripting # Disable scripting engine -- NotablesCAN # Do not allow table scanning -- Nounixsocket # Disable Unix socket listening --nssize arg (=16) # set message database.ns file size (MB) -- objCheck # Check the validity of receiving client data, --profile arg # file parameter 0=off 1=slow, 2=all --quotaFiles arg # number of files allower per db, Repairpath requires -- Quota -- REST # Start the simple REST API --repair # Run repair on all DBS -- RepairPath ARG # Dbpath --slowms arg (=100) # value of slow for profile and console log --smallfiles # use smaller default files -- Syncdelay arg (=60) # Number of seconds before data is written to disk (0=never, not recommended) --sysinfo # Print some diagnostic system information --upgrade # If database needs to be upgraded --fastsync # Enable the slave database replication service from a DBPATH that is a snapshot of the master database and can be used to quickly enable synchronization --autoresync --oplogSize arg # Set the size of oplog (MB) --master # master database mode --slave # slave database mode --source arg # slave database port number --only arg # specify a single database replication ReplSet arg # Set replica set name -- configSvr # Declare this is a config service for a cluster, default port 27019, Default directory /data/configdb -- shardSvr # declares this to be a sharding of a cluster, default port 27018 --noMoveParanoia # closes for moveChunk data saving

Copy the code

After the node is configured, you can start mongo, CD to the bin directory:

./mongod -f /etc/mongodb_master.conf
./mongod -f /etc/mongodb_slave.conf
./mongod -f /etc/mongodb_arbiter.conf

Copy the code

Node is configured

Finally, you need to configure the primary, secondary, and quorum nodes. First we choose a server to connect to:

. / mongo 10.100.1.101:27017 > use the admin

Copy the code

Then configure:

CFG = {_id: "testdb", members: [{_id: 0, host: '10.100.1.101:27017' priority: 2}, {_id: 1, host: '10.100.1.102:27017' priority: 1}, {_id: 2, host: '10.100.1.103:27017, arbiterOnly: true}}; Rs.initiate (CFG) # Initiate configuration to take effect

Copy the code

If the configuration works normally, it is almost complete. You can view the related information by running the rs.status() command.

At this point, you can log in to the database to test the results, see the normal database operation, whether the master and slave synchronization. I don’t want to talk more about testing here.

Data backup and restoration

After simply building the cluster, the original test environment data needs to be migrated, so it involves the backup and restoration of Mongo.

It is relatively easy to implement with Mongodump and Mongorestore:

/bin/mongodump -h 10.100.1.101 -d testdb -o. # mongodump -h dbhost -d dbname -o dbdirectory # -h: address of the server where MongDB resides, for example: 127.0.0.1, 127.0.0.1:27017 # -d: specifies the database instance to be backed up, for example: test # -o: /bin/mongorestore -h 10.100.1.102 -d testdb testdb # mongorestore -h <hostname><:port> -d dbname <path> # --host <:port>, -h <:port> : MongoDB server address, default: localhost:27017 # --db, -d: database instance to be restored # --drop: --dir: Specify the directory to be backed up. You cannot specify <path> and --dir at the same time.

Copy the code

conclusion

Here, I have a certain understanding and understanding of Mongo, and also have a basic grasp of the construction and migration process. In the face of the three nods (no development, no documentation, no notes) of the old project, I have a little confidence, the rest of the time will be spent in the side of the code while making fun of the day, the imagination is tired…