Service Application Scenarios

  • High concurrency, high storage, high scalability and availability
  • NoSql, loose database
  • Social scenarios, storage of user information, geographical location; Game scene, easy to query, efficient storage and access; Logistics scenario, order status is constantly updated; In the Internet of Things scenario, information about all connected smart devices and logs are stored. Live video, interactive information and user information
  • Large amount of data, frequent write operations, low value, low transactional requirements
  • Transactions and complex JSON support are not required; TB is required to set the PB level data store. The stored data is not lost; A large number of geographical location queries, text queries
  • No schema, no concrete chain, similar to JSON’s Bson structure (binary JSON); A document database is a non-relational database that most resembles a relational database

mac

Start the

  • inbash_profileAdding environment variables
vim ~/.bash_profile
export PATH=/usr/local/mongodb/bin:$PATH
source ~/.bash_profile
Copy the code
  • The service start
Sudo mongod --dbpath= /data/db/Copy the code
  • Client startupmongo
  • The client connects to the specified portMongo, host = 127.0.0.1 port = 27017

configuration

  • Creating a folder
sudo mkdir -p single/log
sudo mkdir -p single/data/db
Copy the code
  • Creating a Configuration File
sudo vi single/mongod.conf
Copy the code
  • The configuration file
SystemLog: // Send all logs to the destination specified as the file destination: file // Mongod or Mongos send logs to the destination file path: / usr/local/mongo/use/log/mongod log / / instance restarts, new entries will be attached to the existing at the end of the log logAppend: true storage: / / storage stranger dbPath: / usr/local/mongo/use/data/db journal: / / start or disable persistence log to ensure that the data files remain valid and recoverable enabled: true processManagement: // Start daemon mode to run the server in background fork:true net: // Access IP bindIp: localhost // Port port:27017Copy the code
  • Configuration file start, reported1 or 48The configuration file is written incorrectlyymlFormat, space indent
sudo mongod -f mongod.conf
Copy the code
  • Or to set permissions for files
sudo chown apple /usr/local/mongodb/replica-sets/myrs_27019/log/mongod.log
sudo chown apple /usr/local/mongodb/replica-sets/myrs_27019/data/db/
Copy the code
  • Check the service
Ps aux | grep mongod pid / / check processCopy the code

Shut down

  • Client Shutdown
use admin
db.shutdownServer()
Copy the code
  • Kill off
Ps aux | grep mongod / / view the process pid sudo kill 2 pidCopy the code

Basic Common Commands

Database operations

  • Viewing all databasesshow dbs
  • Creating a Databaseuse articledbWhen created, it only exists in memory and is not persisted. It is persisted when there is a collection
  • Viewing the Current Databasedb
  • adminStorage user information permissions;localThis data is never replicated and stored in arbitrary collections on a single server locally;configUsed for Mongo sharding setup, internal use, save sharding related information
  • Deleting a Databasedb.dropDatabase()

Set operations

  • Explicitly createdb.createCollection("article")
  • See the collectionshow collections
  • Delete the collectiondb.article.drop()

The document operation

Insert and query

  • A single insertDb.com ment. Insert ({" articleid ":" 10000 ", "content" : "so tired", "userid" : "102"})
  • The querydb.comment.find()
  • Multiple insert
[{db.com ment. InsertMany (" _id ":" 1 ", "articleid" : "10000", "content" : "so tired", "userid" : "102"}, {" _id ":" 2 ", "Articleid" : "10001", "content" : "not tired", "userid" : "105"}, {" _id ":" 3 ", "articleid" : "10002", "content" : "very tired", "Userid", "104"}, {" _id ":" 4 ", "articleid" : "10003", "content" : "bad tired", "userid" : "103"}));Copy the code
  • Conditions of the querydb.comment.find({"_id":"1"})
  • A conditional query returns a single piece of datadb.comment.findOne({articleid:"10000"})
  • Display partial fieldsdb.comment.find({articleid:"10000"},{articleid:1, content:1, _id:0}) _idDefault display, exclude need to set 0; Set the fields to 1
  • An exception occurs when inserting multiple threads. Proceduretry... catchcapture
Try {// insert statement} catch(e) {print(e); }Copy the code

update

  • Cover changeDb.com ment update({_id:"1"},{content:" not at all tired "})The second parameter overwrites the entire data. By default, the first qualifying record is modified. If it does not exist, it is automatically added
  • Local modificationDb.com ment. Update ({_id: "2"}, {$set: {content: "is not a bit tired," userid: "200"}})
  • Bulk changesDb.com ment. Update ({userid: "104"}, {$set: {content: "tired not tired"}}, {multi: true}), set the third parameter
  • The column value growthdb.comment.update({"userid":"104"},{$inc:{likenum:NumberInt(32)}}), an increase of 32

delete

  • Conditions to deletedb.comment.remove({_id:ObjectId("6142ab897f0555879ed562a7")})
  • Delete alldb.comment.remove({})

The query

  • Statistical query
Db.count ({userid:"104"})Copy the code
  • Query the first few itemsdb.comment.find().limit(2)
  • Skip the first fewdb.comment.find().limit(2).skip(2)
  • Sorting querydb.comment.find({}, {userid:1}).sort({userid:1})1 is in ascending order and -1 is in descending order
  • Regular expression queryDb.com ment. Find ({the content: / /}), using regular expressions to match content
  • Comparison of the query
Db.ment. Find ({userid:{$gt:"102"}}) lt less than GTE greater than or equal to LTE Less than or equal to Ne Not equal toCopy the code
  • Contains the querydb.comment.find({_id: {$in:["2","3"]}})
  • Conditions of the query
//id>1 userid<105
db.comment.find({$and:[{_id:{$gt:"1"}}, {userid:{$lt:"105"}}]})
//id>2 or id>=4
db.comment.find({$or:[{_id:{$lt:"2"}},{_id:{$gte:"4"}}]})
//id<2 || (likenum==1000 && id==ObjectId("6142ae7e7f0555879ed562a8"))
db.comment.find({$or:[{_id:{$lt:"2"}},{$and:[{likenum:NumberInt(1000)},{_id:ObjectId("6142ae7e7f0555879ed562a8")}]}]})
Copy the code

The index

  • Improve search order,B tree; Mysql index is B+ tree

The index management

  • View the current index
Db.com ment. GetIndexes () "v" : 2, indexing engine version "key" : {" _id ": 1 / / name ascending}," name ":" _id_ index name "ns" : "Articledb.com ment" collectionCopy the code
  • Create indexesdb.comment.createIndex({userid:1})
  • Creating a composite indexdb.comment.createIndex({userid:1, articleid:-1})
  • Remove the index
Db.ment. DropIndex ({userid:1}) userid_1_articleid_-1 Db.ment. DropIndex ("userid_1_articleid_-1") // Delete all indexes db.ment. DropIndex ()Copy the code

Index usage

  • Execute the plan and query performance.db.comment.find({userid:"104"}).explain()To see if this condition is obtained by index queryIXSCANIndex collection, again throughFETCHFind the target field

Cover the query

  • The content of the query is the index, and the content that needs to be returned is the index, you can directly find in the index, do not need to continue to find in the collection through the index
db.comment.find({userid:"104"}, {userid:1,_id:0})
Copy the code

Spring integration mongo

configuration

  • pom.xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Copy the code
  • application.yml
Data: mongodb: # host: 127.0.0.1 # database: articledb # # 27017 # can also use the uri connection uri: mongo: / / 192.168.40.134:27017 / articledbCopy the code

Entity class

  • Annotated entity class
@document (collection="comment");Copy the code
  • With the primary key
@id private String ID; @id private String id; @id private String id; @id private String id; / / the primary keyCopy the code
  • Annotation fields
@field ("content") private String content; @field ("content") private String content; // Make fun of the contentCopy the code
  • index
// Added a single field index @indexed Private String userID; // Publisher IDCopy the code
  • The annotation conforms to the index, the annotation is on the class. You are advised to create an index on the command console, because indexes that will be created in the future may not have corresponding attribute values
Def = "{'userid': 1, 'nickname': -1}")Copy the code

Add and delete

  • defineDao layerinterface
public interface CommentRepository extends MongoRepository<Comment, String> {
}
Copy the code
  • defineService business layerfunction
@Service public class CommentService { @Autowired private CommentRepository commentRepository; /** * save a comment * @param comment */ public void saveComment(comment comment){// If you want to customize the primary key, specify the primary key here; If you do not specify a primary key, MongoDB will automatically generate a primary key // set some default initial values... // Call dao Commentrepository.save (comment); } /** * updateComment * @param comment */ public void updateComment(comment comment){commentrepository.save (comment); } / delete comments by id * * * * * @ param id/public void deleteCommentById (String id) {commentRepository. DeleteById (id); } / queries all Comment * * * * @ return * / public List < Comment > findCommentList () {return commentRepository. The.findall (); } /** * Query comments by ID * @param ID * @return */ public Comment findCommentById(String id){return commentRepository.findById(id).get(); }}Copy the code

Case – Paging query

  • daolayer
Page<Comment> findByParentid(String paraentid, Pageable Pageable);Copy the code
  • servicelayer
Public Page < Comment > findCommentListByParentid (String parentid, int Page, int size) {/ / query Page starting from 0, The size refers to a page of a few return commentRepository. FindByParentid (parentid, PageRequest) of (page 1, the size)); }Copy the code
  • call
/ / id for 3 documents, each Page two, take the fourth Page Page < Comment > CBP. = commentService findCommentListByParentid (" 3 ", 4, 2); / / the number System. Out. Println (CBP) getTotalElements ()); For (Comment Comment: cbp.getContent()) {system.out.println (Comment); }Copy the code

MongoTemplate- Implement comment likes

  • servicelayer
@Autowired private MongoTemplate mongoTemplate; Public void updateCommentLikenum(String ID) {// Query Query = Query.query(Criteria.where("_id").is(id)).addCriteria(Criteria.where("likenum").lt(889)); Update Update = new Update(); update.inc("likenum"); mongoTemplate.updateFirst(query,update,Comment.class); }Copy the code

A copy of the set

  • A set of services that maintain the same data set provides redundancy and high availability
  • Master slave replication, with a fixed master and fixed slave; Replica set, no fixed master node, the whole cluster through voting mechanism to elect a master node, there is always an active point (master) and one or more backup nodes (slave)
  • Two types. Master node, the primary node for data operations, read and write; Secondary nodes, redundant data backup nodes, can be read or elected
  • Three roles. Primary member, which accepts all write operations; Copy member, from the master node through the replication operation to maintain the same database, cannot write operation, can read operation; An arbiter, which does not hold a copy of any data and only serves as a vote, can be both an arbiter and a slave node type

Set up

  • Increase the configuration
replication: 
 replSetName: myrs
Copy the code
  • The primary node is initialized. Procedurers.initiate()
  • Check the configurationrs.conf()
  • Check the statusrs.status()
  • Adding slave NodesRs. The add (" 127.0.0.1:27018)"
  • Adding a Quorum NodeRs. AddArb (127.0.0.1: "27019")/rs. The add (" 127.0.0.1:27019 ", true)
  • Change the IP address of the primary node
var config = rs.config(); The config. Members [0]. Host = "180.76.159.126:27017"; rs.reconfig(config)Copy the code

Data read and write operation

  • The primary node can read and write properly
Db.com ment. Insert ({"articleid":"100000","content":" It's nice today, "," userID ":"1001","nickname":"Rose"," createDateTime ":new Date()}) db.ment.find()Copy the code
  • Slave node Settings can only be readrs.slaveOk()
  • The read permission is cancelled from the noders.slaveOk(false)
  • The slave node can only readdb.comment.find()

Arbitration node

  • No service data is stored. You can log in to view it
  • Set up thers.slaveOk()
  • There’s only one libraryThe local 0.000 GBOnly configuration information

Voting principle

  • Trigger an election. The primary node is faulty. The network of the primary node cannot be large (the default heartbeat information is 10 seconds). Human interventionrs.stepDown(600)
  • Election rules. The highest number of votes, supported by a majority of members; If the votes are tied, the newer node wins and passes the operation logoplogcontrast
  • You can set the priority of a server. A higher priority indicates that the server is more likely to become the active node. throughrs.conf()You can view"priority" : 1,, the priority of the quorum node is 0, and the quorum node cannot become the primary node
  • Changing the Priority
CFG =rs.conf() // Set the priority of the corresponding member cfg.members[1]. Priority =2 // Reload configuration rs.reconfig(CFG)Copy the code

The fault test

  • When the replica node is down, data on the primary node is not affected. After the replica node is reconnected, data is automatically synchronized
  • The primary node is down, the selection is triggered, the replica set is 3, the quorum node also votes for the secondary node, the secondary node is elected as the primary node
  • The quorum node and the master node are down, leaving the slave node itself, but there is no majority of members, only itself, but when reconnected, will become the master node; When there are more than three slave nodes left, a new master node is immediately selected
  • Both the quorum node and the secondary node are down. After 10 seconds, the service is degraded and the secondary node is downgraded. After reconnection, the status of the primary node is restored

Spring connection replica set

  • Uri format
mongodb://host1,host2,host3/articledb? Connect =replicaSet&slaveOk=true&replicaSet= name of the replica setCopy the code
  • configurationyml
Spring: # data source configuration data: mongo: uri connection uri: mongo: / / 127.0.0.1:27017127.00 0.1:27018127.00 0.1:27019 / articledb? connect=replicaSet&slaveOk=true&replicaSet=myrsCopy the code

Shard cluster

  • Replica sets, where all data is synchronized and can only exist on one server
  • Sharding, breaking up data, spreading it across different machines
  • Solutions to system growth. Vertical expansion, plus hard disk; Horizontal expansion, plus machines
  • Components. Shard, which stores data. Each corner shard can be deployed as a replica set. Mongos, routing, provides an excuse between client applications and sharded clusters; Config Servers, scheduling configuration, metadata and configuration Settings for configuring server storage clusters, must be deployed as replica sets

  • Users access the configuration set through routes to find the fragment in which data resides and then search for the target data in the fragment

Configure fragments and configuration files

  • Shard profile
Replication: // Replica set name replSetName: mySharDRs01 Sharding: // Sharding Role clusterRole: shardSvrCopy the code
  • Configure the node configuration file
replication:
 replSetName: myconfigrs
sharding:
 clusterRole: configsvr
Copy the code
  • Initialize the shard replica set, as in the replica set operation above
  • Initialization configuration shards, no quorum nodes, are replica nodes

Configure the routing

  • Routing node configuration file, do not need data directory, only need log directory
systemLog: destination: file path: /usr/local/mongodb/sharded_cluster/mymongos_27017/log/mongod.log logAppend: True processManagement: fork: true net: bindIp: localhost port: 27017 sharding: // Specify configDB: Myconfigrs / 127.0.0.1:27019127.00 0.1:27119127.00 0.1:27219Copy the code
  • Configure the connection between routing nodes and fragments
Sh. AddShard (" myshardrs01 / localhost: 27018127.00 0.1:27118127.00 0.1:27218 ") Sh. AddShard (" myshardrs02 / localhost: 27318127.00 0.1:27418127.00 0.1:27518 ")Copy the code
  • Route Viewing Statussh.status()
  • Enable the fragmentation function for route and open the corresponding databasesh.enableSharding("articledb")
  • Sharding a collection, specifying the collection and the sharding key. Sharding rule, hashing strategy; Scope strategy.A collection can only be sharded according to one policy. Once a collection is sharded, the sharding key and sharding value cannot be changed. For example, you cannot select different sharding keys for the collection or update the values of the sharding keys.
Sh.shardcollection ("articledb.comment", {"nickname":"hashed"}) Sh.shardcollection ("articledb.author", {"age":1})Copy the code
  • Add routing nodes, same configuration, different end; After the connection is normal, it can be used directly without further configuration

use

  • Through the routing client, the loop inserts 1000 pieces of data to test the hash strategy
for(var i=1; i<=1000; i++) {db.comment.insert({_id:i+"",nickname:"BoBo"+i})}Copy the code
  • Test scope rules because the default block size is 64M and needs to be reduced for testing purposes
Use config db.settings.save({_id:"chunksize", value: 1}) i<=20000; i++) {db.author.save({"name":"BoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoBoB Obobobobobobobo "+ I,"age":NumberInt(I %120)})} db.settings.save({_id:"chunksize", value: 64})Copy the code
  • Effect of range strategy
{ "age" : { "$minKey" : 1 } } -->> { "age" : 0 } on : myshardrs01 Timestamp(2, 0) 
{ "age" : 0 } -->> { "age" : 46 } on : myshardrs01 Timestamp(3, 0) 
{ "age" : 46 } -->> { "age" : 92 } on : myshardrs02 Timestamp(3, 1) 
{ "age" : 92 } -->> { "age" : 111 } on : myshardrs02 Timestamp(1, 6) 
{ "age" : 111 } -->> { "age" : { "$maxKey" : 1 } } on : myshardrs02 Timestamp(1, 3)
Copy the code

Spring links to sharded clusters

  • configurationyml
Spring: # data source configuration data: mongo: uri: mongo: / / 127.0.0.1:27017127.00 0.1:27117 / articledbCopy the code

Safety certification

  • Roles grant users operation permissions on database resources. The permissions of each role can be explicitly specified or inherited from other roles
  • View all built-in permissionsdb.runCommand({rolesInfo:1, showBuiltinRoles:true})

Single-instance environment

  • Example Create a super administratordb.createUser({user:"myroot", pwd: "12345", roles:["root"]})And only inadminUnder the
  • Create a dedicated admin userdb.createUser({user:"myadmin",pwd:"123456",roles: [{role:"userAdminAnyDatabase",db:"admin"}]})
  • View information about created usersdb.system.users.find()
  • Delete userdb.dropUser("myadmin")
  • Change the passworddb.changeUserPassword("myroot", "111111")
  • Verify the userdb.auth("myroot", "111111")Before login, switch to the corresponding database
  • An errortoo many users are authenticated, it is best to authenticate one account at a time, do not authenticate a client multiple times
  • Creating a Common Userdb.createUser({user: "manager", pwd: "123456", roles: [{ role: "readWrite", db: "articledb" }]})
  • Enable security authentication on the server
// Add mongod -f mongod. Conf --auth // Modify the configuration file securit: authorization: enabledCopy the code

Spring Secure Connection

  • Yml configuration
Data: mongodb: # host: 127.0.0.1 # database: articledb # 27017 # if user name and password string username: "manager" password: "123456" # uri: mongo: / / manager: @127.0.0.1:123456 27017 / articledbCopy the code

Replica set environment

  • Create a super administrator under admindb.createUser({user:"myroot",pwd:"123456",roles:["root"]})
  • Generating key files
/ / create, Sudo openssl genrsa -out mongo.keyfile 1024 // Modify permission only for current user to read sudo chmod 400 mongo.keyfile // Copy the file to each member folder sudo cp mongo.keyfile myrs_27019Copy the code
  • Modifying a Configuration File
security:
 keyFile: /usr/local/mongodb/replica-sets/myrs_27017/mongo.keyfile
 authorization: enabled
Copy the code
  • Other operations are consistent with the single instance
  • Creating a Regular Accountdb.createUser({user:"zhangsan",pwd:"123456",roles:["readWrite"]})
  • Spring visit,ymlconfigurationUri: mongo: / / zhangsan: 123456 @127.0.0.1:27017127.00 0.1:27018127.00 0.1:27019 / articledb? connect=replicaSet&slaveOk=true&replicaSet=myrs

Shard cluster

  • Put one on every serverkeyfile, the operation is the same as above