Scenarios applicable to MongoDB

  • Web site data: Mongo is ideal for real-time insertion, update, and query, with the replication and high scalability required for real-time web site data storage.
  • Caching: Because of its high performance, Mongo is also suitable as a caching layer for information infrastructure. After the system restarts, the persistent cache layer built by Mongo avoids overloading the underlying data source.
  • Large, low-value data: Before using traditional relational databases to store large, low-value data, programmers often used traditional files for storage.
  • Highly scalable scenarios: Mongo is well suited for databases consisting of dozens or hundreds of servers, and Mongo’s roadmap already includes built-in support for the MapReduce engine and a solution for clustering high availability.
  • For object and JSON data storage: Mongo’s BSON data format is very suitable for storage and query in document format.

Specific application:

  • Game scene, use MongoDB to store game user information, user equipment, points and other directly embedded document form storage, convenient query, update.

  • In logistics scenario, MongoDB is used to store order information, and the order status will be updated continuously during the delivery process. The order status is stored in the form of MongoDB embedded array, and all changes of the order can be read out in a single query.

  • In social networking scenarios, MongoDB storage is used to store user information and the information published by users in the circle of friends, and the nearby people and places are realized through geographical location index.

  • In the Internet of Things scenario, MongoDB is used to store information about all connected smart devices and log information reported by the devices, and analyze the information in multiple dimensions.

  • Live broadcast, use MongoDB to store user information, gift information, etc.

Springboot access mongo

Add the dependent

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
Copy the code

Configure the connection

Spring. The data. The mongo. Host = 192.168.152.39spring.data.mongodb.port=27017
spring.data.mongodb.database=people
Copy the code

Using a MongoTemplate

public class UserController {

    @Autowired
    private MongoTemplate mongoTemplate;

  public void insertResume(User user) {  mongoTemplate.insert(user,"people_table");  }   public User findByName(String name) {  Query query = new Query();  query.addCriteria(Criteria.where("name").is(name));  List<User> datas = mongoTemplate.find(query,User.class,"people_table");  returndatas.isEmpty()? null:datas.get(0); }    public List<User> findList(String name) {  Query query = new Query();  query.addCriteria(Criteria.where("name").is(name));  List<User> datas = mongoTemplate.find(query,User.class,"people_table");  return datas;  }   public List<User> findListByNameAndExpectSalary(String name, double salary) {  Query query = new Query();  query.addCriteria(Criteria.where("name").is(name).andOperator(Criteria.where("salary").is(salary)));  return mongoTemplate.find(query,User.class,"people_table");  } } Copy the code

Use the Repository

Write the entity class and type @document on the entity class.

@Document("people_table")
public class User {

    private String id;
    private String name;
 private String city;  private String birthday;  private double salary;  private double gender;  } Copy the code

Compile Repository interface to inherit MongoRepository

public interface ResumeRepository extends MongoRepository<User,String> {
    List<User>  findByNameEquals(String name);
    List<User>  findByNameAndExpectSalary(String name,double expectSalary);
}
Copy the code

Methods for specific reference:

https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods.query-creation
Copy the code

Mongo architecture

image-20200923131706176

The architecture of MongoDB and MySQL is similar, with pluggable storage engines at the bottom to meet the different needs of users. Users can choose different storage engines based on the data characteristics of the program. In the latest version of MongoDB, WiredTiger is used as the default storage engine. WiredTiger provides different granularity concurrency control and compression mechanisms, providing the best performance and storage rate for different applications.

On the upper layer of the storage engine is the data model and query language of MongoDB. Because MongoDB stores data differently from RDBMS, it has created a set of different data model and query language.

MongoDB data model

embedded

Inline means keeping related data in the same document structure. MongoDB’s document structure allows a field or an array of values to be nested as a document.

1. There is inclusion relationship between data objects, generally one-to-many or one-to-one relationship between data objects.

2. Data that needs to be read together frequently.

3. Data required by Map-reduce /aggregation is aggregated, and these operations can only operate a single collection.

reference

Reference mode associates two different documents by storing data reference information that applications can parse to access related data.

1, when embedded data will lead to a lot of data duplication, and read performance advantages are not enough to cover the disadvantages of data duplication.

2. When complex many-to-many relationships need to be expressed.

3. Large hierarchical result data sets should not be nested too deeply.

MongoDB storage engine WiredTiger

A storage engine is the core component of MongoDB, managing how data is stored on hard disks and memory. MongoDB supports storage engines such as MMAPv1,WiredTiger and InMemory. The InMemory storage engine is used to store data only InMemory, and only a small amount of meta-data and Diagnostic logs are stored in Disk files. The required data can be obtained without Disk I/o operations. The InMemory storage engine significantly reduces Latency for data queries. Mongodb3.2 default storage engine is WiredTiger, before version 3.2 default storage engine is MMAPv1, mongodb4.x version no longer support MMAPv1 storage engine.

WiredTiger storage engine advantages

1. Document space allocation method. WiredTiger uses BTree storage MMAPV1 linear storage requires Padding

2. Concurrency level. WiredTiger document level locking The MMAPV1 engine uses table level locking

3. Data compression. Snappy (default) and Zlib, saving several times more space than MMAPV1(uncompressed).

4. Memory usage. WiredTiger can specify how much memory to use.

5. The Cache is used. The WT engine uses the WiredTiger Cache and File System Cache to ensure final consistency of data on Disk. MMAPv1 only has journal logs.

Implementation principle of WiredTiger storage engine

Write requests

WiredTiger writes data to the Cache by default and persists to WAL (Write Ahead Log). A checkpoint is performed every 60 seconds or when the Log file reaches 2G. WriteConcern {w:, j:, wtimeout:}) generates a snapshot file. During WiredTiger initialization, the system restores to the latest snapshot state and then restores data based on WAL to ensure data integrity

image-20200923134312667

The Cache is based on a BTree. The node is a page, the root page is the root node, the internal page is the intermediate index node, and the leaf page stores the data. The data is read and written in page units. WiredTiger uses Copy on Write to manage write operations (insert, update, delete). Write operations are first cached in the cache. When persistent, write operations will not be performed on the original Leaf page, but on the newly allocated page. Each checkpoint produces a new root page.

Checkpoint process

  1. A checkpoint is performed on all tables, and the metadata of the checkpoint of each table is updated to wiredtig.wt

  2. Check wiredtig. wt and update the metadata of the table checkpoint to the temporary file wiredtig.turtle.set

  3. Rename wiredtig.turtle. set to wiredtig.turtle.

  4. If the preceding processes fail, WiredTiger restores data to the latest snapshot state during the next connection initialization and then restores data based on WAL to ensure storage reliability.

Journaling

When the database is down, to ensure the persistence of data in MongoDB, MongoDB uses Write Ahead Logging to pre-write to journal files on disks. In addition to journal logs, MongoDB also uses checkpoint to ensure data consistency. When the database is down, checkpoint and Journal files are needed to restore data.

  1. Look for the identifier of the previous checkpoint in the data file

  2. Find the record corresponding to the identifier in the Journal file

  3. Redo all operations after the corresponding record

image-20200923134818766

Replica sets

image-20200923135448988

A replication set is a cluster of Mongod instances that have the same data set. A replication set is a cluster consisting of 2 or more servers, and the members of the replication set include the Primary Primary node,secondary secondary node, and voting node. Replication sets provide redundant backup of data and store data copies on multiple servers, improving data availability and ensuring data security.

advantage

1. High availability. Prevent device (server, network) failures. Provides automatic failover. Technology to ensure high availability

Disaster recovery. When a fault occurs, it can be restored from another node for backup.

3. Functional isolation. We can perform reads on the standby node to reduce the stress on the primary node, for example, for analysis, reports, data mining, system tasks, etc.

Principle of replication set cluster architecture

Read and write operations can be performed on the Primary node in a replication set, while the Secondary node can only be used for read operations. The Primary node needs to record all operations that change the state of the database, and these records are saved in oplog. The file is stored in the local database. Each Secondary node uses oplog to copy data and apply it locally to keep the local data consistent with that of the Primary node. Oplog is idempotent, meaning its results are consistent no matter how many times it is executed, making it more useful than mysql binary logs.

The structure of oplog

{
"ts" : Timestamp(1446011584, 2), 
"h" : NumberLong("1687359108795812092"), 
"v": 2."op" : "i"."ns" : "test.nosql"."o" : { "_id" : ObjectId("563062c0b085733f34ab4129"), "name" : "mongodb"."score" : "10"}}Copy the code

Description:

Ts: operation time, current timestamp + counter, counter is reset every secondH: globally unique identifier of an operationV: Oplog version informationOp: indicates the operation typeI: insert operationU: Update operationD: Delete operationC: Run commands such as createDatabase and dropDatabase.N: Empty operation, special useNs: the set for which the operation is performedO: Operation contentO2: The query condition is updated. This field is contained in the update operation onlyCopy the code

Data synchronization in a replication set can be divided into initial synchronization and Keep synchronization. Initial synchronization means that data is fully synchronized from the Primary node. If the Primary node has a large amount of data, the synchronization takes a long time. Keep replication refers to the real-time synchronization between nodes after the initial synchronization is generally incremental synchronization.

Initial synchronization can be triggered in the following two cases:

(1) Secondary joins for the first time.

(2) The amount of data behind Secondary exceeds the size of Oplog, so it will also be copied in full.

MongoDB Primary node elections are based on heartbeat triggers. Any two of N nodes in a replication set maintain the heartbeat, and each node maintains the state of the other N-1 nodes.

image-20200923142320193
Heartbeat detection: The whole cluster needs to maintain a certain amount of communication to know which nodes are alive and which nodes are down. The mongodb node sends pings packages every 2 seconds to other nodes in the replica set, and if the other nodes do not return within 10 seconds, they are marked as inaccessible. Each node maintains a state mapping table, indicating the current role of each node, log timestamp and other key information. If the master node finds itself unable to communicate with most of the nodes, it demotes itself to secondary read-only node.Copy the code

Trigger timing of primary node election:

Initializes a replication set for the first timeIf the weight of the Secondary node is higher than that of the Primary node, a replacement election is initiatedThe Secondary node initiates an election when it finds no Primary in the clusterA Primary node that cannot access a Majority of members actively degradesCopy the code

When an election is triggered, the Secondary node attempts to elect itself as Primary. The primary election is a two-stage process + majority agreement.

Stage 1:It checks whether it is eligible for election. If it is, it initiates FreshnessCheck to other nodes for whether the node is eligible for election, and carries out peer arbitration
Stage 2:The initiator sends an Elect request to the surviving nodes in the cluster. The arbiter performs a series of legitimacy checks on the nodes receiving the request. If the check passes, the arbiter (up to 7 of the 50 nodes in a replication set can vote) gives the initiator one vote. Pv0 prevents two votes in one election with a 30-second election lock. Pv1 uses terms(a monotonically increasing election counter) to prevent voting twice in one election. Majority agreement:If the initiator obtains more than half of the votes, it passes the election and becomes the Primary node. In addition to the common network problems, the reason for obtaining less than half of the votes is that the nodes of the same priority go through the peer arbitration of the first stage and enter the second stage at the same time. Therefore, when there are insufficient votes, sleep[0,1] seconds of random time, after which the election is attempted again.Copy the code

Shard Cluster

Sharding is a method used by MongoDB to split large sets horizontally among different servers (or replication sets). You don’t need a powerful mainframe computer to store more data and handle larger loads.

Why sharding

1. The required storage capacity exceeds the capacity of the single-node disk.

2. Active data sets exceed the memory capacity of the single machine, causing many requests to read data from disks, affecting performance.

3.IOPS exceeds the service capability of a single MongoDB node. As data increases, the bottleneck of single MongoDB instances becomes more and more obvious.

4. The replica set has a node limit.

Subdivision principle

image-20200923144016069

A sharding cluster consists of the following three services:

  • Shards Server: Each Shard consists of one or more Mongod processes for storing data.

  • Router Server: All requests to the database cluster are coordinated through the Router(Mongos), which is a request distribution center that forwards application requests to the corresponding Shard server, without adding a routing selector to the application program.

  • Config Server: configures the Server. Stores the configuration of all database meta-information (routes and fragments).

Shard key: To allocate documents within a data set, MongoDB uses the shard primary key to split the set.

Chunk: MongoDB divides data into chunks within a Shard server. Each chunk represents a portion of data within the Shard server. MongoDB splits shard data into blocks, and each block contains a range of left closed and right open intervals based on the shard primary key.

Shard strategy

The scope of fragmentation

image-20200923145354003

Range sharding is the partitioning of data based on the values of the shard primary key, and each block will be assigned a range.

Range sharding is suitable for searching within a certain range, such as searching for data with X value between [20,30). Mongo routing can be directly located to the Chunk of the specified shard according to the metadata stored in Confifig server.

Disadvantages: If the shard key increases (or decreases), the newly inserted documents will be distributed to the same chunk, and the write capability cannot be extended.

Hash shard

image-20200923151051854

Hash sharding calculates the Hash value of a sharding primary key. Each block is assigned a range of Hash values.

Hash sharding is complementary to range sharding. It can randomly distribute documents to each chunk, which fully expands the write capability and makes up for the deficiency of range sharding. The disadvantage is that it cannot efficiently serve range query.

Combined slice key A + B(hash thought cannot be direct hash)

If there is no suitable slice key in the database to choose from, or if the base of the slice key you intend to use is too small (that is, it changes as little as 7 days a week), you can choose another field to use the combination of the slice key, or you can even add redundant fields to the combination. It is generally coarse-grained + fine-grained combination.

This article is formatted using MDNICE