What is the mongo

MongoDB is a Document database with JSON as data model. The so-called “Document” is “JSON Document”, not PDF, Word, Excel documents that we generally understand.

An analogy to other database types:

  • Relational database management systems, such as MySQL, Oracle, SQL Server, Postgresql, etc
  • Key-value stores, such as Redis, MemCached
  • Document storage, which is MongoDB, and CouchDB, which I don’t know about, Couchbase
  • Big data storage system, HBASE, Google Bigtable
  • Hadoop based data analysis system, Hive, Spark
  • Text query systems such as Lucence, Solr, and the common Elasticsearch

To give you an intuition, MongoDB stores data that looks like this:

With so many data stores to choose from, why learn MongoDB?

  1. High performance: MongoDB provides high performance data persistence. In particular, support for embedded data models reduces I/O activity on database systems.
  2. High availability: Replica sets of MongoDB provide automatic failover and data redundancy.
  3. High scalability: MongoDB provides level scalability. Sharding distributes data across a cluster of machines. For example, massive data storage, service capabilities can be horizontally expanded.
  4. Rich query support: MongoDB supports rich query languages and supports read and write operations (CRUD), such as data aggregation, text search, and geospatial queries.

Quick learning

Single-node Installation

The production environment usually runs Linux servers. I also use Linux virtual machines to simulate the server environment and install MongoDB on Linux virtual machines.

The best way to learn something new is to go to MongoDB:

www.mongodb.com

But, the site was hard to open (I basically opened it once in 10 attempts…). It is easy to find the download address from the official website.

Installation is extremely simple with the following steps:

Mkdir -p /usr/local/mongodb/data mkdir -p /usr/local/mongodb/conf mkdir -p /usr/local/mongodb/logs Wget download mongo install zip file https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.4.5.tgz # extract tar ZXVF. - Mongodb-linux-x86_64-rhel70-4.4.5. TGZ # rename (this is a custom issue, Mv mongodb-linux-x86_64-rhel70-4.4.5 mongodb-4.4.5 # configure the environment variable vi /etc/profile # export at the end of the file MONGODB_HOME = / usr/local/mongo/mongo - 4.4.5 export PATH = $PATH: $MONGODB_HOME/bin # # is effective source configuration file/etc/profile [root@mongodb-standalone-99 mongodb]# mongo --version mongodb shell version v4.4.5 Build Info: {"version": "" gitVersion 4.4.5", ":" ff5cb77101b052fa02da43b8538093486cf9b3f7 ", "openSSLVersion" : "OpenSSL 1.0.1 E-fiPS 11 Feb 2013", "Modules ": [], "allocator": "tcmalloc", "environment": {"distmod": "rhel70", "distarch": "x86_64", "target_arch": "x86_64" } }Copy the code

MySQL has backup and restore related commands, MongoDB also has corresponding commands, but we need to install a tool. Installation & Usage:

  1. Wget fastdl.mongodb.org/tools/db/mo…
  2. Tar – ZXVF mongo – database – tools – rhel70 – x86_64-100.3.1. TGZ
  3. Mv mongo database – tools – rhel70 – x86_64-100.3.1 mongo – dabase – tools
  4. Configure the environment variable vi /etc/profile

  1. source /etc/profile
  2. Usage:
  • Mongorestore -h localhost:27017 -d collectionName –dir /mongodb/backupdata/
  • Mongodump -h localhost:27017 -d collectionName -o /mongodb/

Basic commands

Start the

You can use the command directly:

mongod --dbpath /usr/local/mongodb/data --port 27017 --logpath /usr/local/mongodb/logs/mongod.log --fork
Copy the code

You can also start with a configuration file. Generally, we start with a configuration file. The content of the configuration file is as follows:

systemLog:
  #MongoDB sends all log output to a file
  destination: file
  # The path to the log file to which mongod or Mongos should send all diagnostic logging information
  path: "/usr/local/mongodb/logs/mongo.log"
  # When the Mongos or Mongod instance is restarted, Mongos or Mongod appends the new entry to the end of the existing log file.
  logAppend: true
storage:
  dbPath: "/usr/local/mongodb/data"
  journal:
    # Enable or disable persistent logging to ensure that data files remain valid and recoverable.
    enabled: true
processManagement:
  Enable daemon mode to run mongos or Mongod processes in the background.
  fork: true
  pidFilePath: "/usr/local/mongodb/logs/mongod.pid"
net:
  Service instance bound IP address, 0.0.0.0, any IP can access
  bindIp: 0.0. 0. 0
  # Bound port
  port: 27017
Copy the code

Start command:

mongod -f /usr/local/mongodb/conf/mongod.conf
Copy the code

Note that the configuration file is in YML format, the format is very strict, sometimes, mongo startup is not successful is the configuration file problems, you can get the idea format.

The connection

Clients can connect to MongoDB using Shell connections or tools (typically MongoDB Compass).

Command:

mongo
Copy the code

or

Mongo, host = 127.0.0.1 port = 27017Copy the code

The mongo command connects to the local port 27017 by default. –host=127.0.0.1 –port=27017 can be configured.

The MongoDB javascript shell is a javascript based interpreter, so it supports JAVASCRIPT applications. For example, you could:

Directing a Compass connection

Go to the MongoDB Compass website and download the software.

The database

  1. Switch or create a database
> use dbname
Copy the code

If the database does not exist, it is automatically created, and if it does, it is switched to the DBNAME database.

TIP: The name of the database can be any UTF-8 string that meets the following criteria

1. Cannot be an empty string. 2. Cannot contain '(Spaces) and. And $and/and \ and \0 (null characters) 3. should be all lowercase 4. maximum 64 bytesCopy the code
  1. To view
> DB test > show databases admin 0.000GB config 0.000GB local 0.000GB > show DATABASES admin 0.000GB config 0.000GB local 0.000 GB >Copy the code

The TIP1: db command is used to check the current database. By default, MongoDB connects to the Test database. If no other database is selected, the collection is stored in the Test database by default.

Or take this example:

As you can see, although use noc creates the database NOC, it is not displayed.

TIP2: In MongoDB, the database/collection is not actually created until the content is inserted. In the example above, to display the NOC database, we need to insert some data first.

  1. Deleting a Database
> db.dropDatabase()
{ "dropped" : "noc"."ok": 1} > show DBS admin 0.000GB config 0.000GBlocal0.000 GBCopy the code

It is used to delete persistent databases.

  1. Repairing the database
mongod --repair --dbpath=/use/local/mongodb/data
Copy the code

Fix database overuse,

Use Mongod-repair only when there are no other options. This operation deletes and does not save any corrupted data during repair.Copy the code

Collection (a table corresponding to MySQL)

Set operation commands, as illustrated above,

Display to create a collection:

db.createCollection(name)
db.createCollection("user")
Copy the code

Implicitly create a collection:

Db.user. insert({name: "zhang3 "})Copy the code

This sentence creates the User collection and inserts a document into the collection.

Delete collection:

> db.user.drop()
Copy the code

View the collection:

> show collections 
Copy the code

Document (Document, corresponding to fields in MySQL table)

new
  1. A single document

Syntax format:

Db. < set >. Insert (< JSON object >) db. < set > insertOne (< JSON object >) db. < set > save (< JSON object >)Copy the code

e.g.

Db.shop. insert({name:" home ", price:4000})Copy the code
  1. Multiple documents

Syntax format:

Db. < set >. InsertMany ([< JSON 1 >, < 2 > JSON,... > < JSON n])Copy the code

e.g.

Db. Shop. InsertMany ([{name: "mobile phone", price: 3000}, {name: "computer", price: 6000}, {name: "daily provisions," price: 50}])Copy the code
The query
  1. General query

Syntax format:

Db.<collection>. Find (<query>,[projection]) - query: optional, query filter, JSON object - projection: optional, result field, JSON objectCopy the code

e.g.

/ / all the db queries. Shop. The find () / / queries all, ditto the shop. The find ({}) / / single condition query db. The shop. The find ({" name ":" mobile phone "}) / / query conditions, Db.shop. find({"name":" phone ","price":3000}); Ditto the db. Shop. The find ({$and: [{" name ":" mobile phone "}, {" price ": 3000}]}) / / conditions or query, Equivalent SQL or db. Shop. The find ({$or: [{" name ":" mobile phone "}, {" price ": 4000}]}) / / regular expression query db. The shop. The find ({" name" : / /} ^)Copy the code

Comparison table of query conditions:

SQL MongoDB
a = 1 {a : 1} Single attribute fields match exactly
a <> 1 {a : {$ne : 1}} $ne indicates absence or presence but not equal to
a > 1 {a : {$gt : 1}} $gt: exists and is greater than
a >= 1 {a : {$gte : 1}} $gte indicates presence and greater than or equal to
a < 1 {a : {$lt : 1}} $lt indicates presence and less than
a <= 1 {a : {$lte : 1}} $LTE is present and less than or equal to
a = 1 and b = 1 {a: 1, b: 1}

or

{$and: [{a: 1}, {b: 1}]} 		$and indicates that all conditions are matched
a = 1 or b = 1 {$or: [{a: 1}, {b: 1}]} $or indicates that a match matches one of two or more conditions
a is null {a: null}

or

{a: {$exists: null}} 		$or indicates that a match matches one of two or more conditions

TIP: find searches for the correct posture of subdocuments

MongoDB encourages embedded documents to implement associated queries. Db.shop. insert({name:" computer ",category:{name:" lenovo ", CPU :"i7"}}) Db. Shop. The find ({" category. The name ":" lenovo "}) don't do this check: db. Shop. The find ({" category ": {" name" : "lenovo"}})Copy the code

Find search array

Find support for the elements in the array to search the shop. The insert ([{name: "lenovo", CPU: [" i5 ", "i7"]}, {name: "dell," CPU: [" i3 ", "i7"]}]) the shop. The find ({CPU: "i5}") db.shop.find({$or:[{cpu:"i5"},{cpu:"i3"}]})Copy the code

Find searches for objects in an array

Db. Shop. Insert ({name: "mobile phone", brand: [{name: "huawei", price: 4000}, {name: "millet", price: 3000}, {name: "apple", price: 8000}]}) Db.shop.find ({"brand.name": "Huawei "})Copy the code

TIP: When querying an attribute in an embedded document, the query condition (field name) must be enclosed in double quotes, like {“brand.name”: “Huawei “}

Find Projection query

If you want the query result to return some fields, you need to use a projection query (not all fields, only the specified fields), similar to the use of the AS keyword in MySQL. Db.shop. Find ({},{"name":1}) db.shop. Find ({},{"name":1}) db.shop. Find ({},{"name":1}) db.shop. 1, _id: 0})Copy the code

delete
Db.shop. remove({name: "phone "}) // Delete the number of records db.shop.remove({price:" phone "}) // delete the number of records db.shop.remove({price: "phone "}) {$lte: 3000}}) / / delete all records the shop, the remove ({}) / / an error db. Shop. The remove ()Copy the code
update
Syntax format: Db.<collection>. Update (< query criteria >,< update fields >) Where conditions and fields are JSON objects db.shop.insert([{name:"iphone12",price:8000},{name:"p40",price:5000},{name:"p30"}]) db.shop.updateOne({name:"iphone12"},{$set:{price:7500}}) db.shop.updateOne({name:"p30"},{$set:{price:3500}})Copy the code

Matters needing attention:

  1. db.. The update () with the db.. UpdateOne () updates only the first record, no matter how many records the entered criteria match

  2. With the db.. UpdateMany (), as many as the input criteria match

  3. The update/updateOne/updateMany, update requirement, must have one of the following conditions, otherwise an error

conditions meaning
$push Add an object to the bottom of the array
$pushAll Add multiple objects to the bottom of the array
$pop Removes an object from the bottom of the array
$pull If the specified value is matched, the corresponding object is removed from the array
$pullAll If any match, the corresponding object is removed from the data
$addToSet If not, add one to the array
$set Modify object attribute values

e.g.

Db.shop. insert({name: "xiaomi", color: [1,2]}) // add db.shop.updateone ({name: "Xiaomi"}, {$push: {color: 3}}) / / delete the db. Shop. UpdateOne ({name: "xiaomi"}, {$pop: {color: Db.shop. updateOne({name: "iphone12"}, {price: 9000}) uncaught exception: Error: the update operation document must contain atomic operators : DBCollection.prototype.updateOne@src/mongo/shell/crud_api.js:565:19 @(shell):1:1Copy the code
The aggregation

Aggregate operations: Process data records and return computed results.

An aggregation operation groups values from multiple documents together and can perform various operations on the grouped data to return a result.

MongoDB provides aggregation:

  • Aggregation Pipeline

The aggregation framework of MongoDB is based on the concept of data processing pipeline. The document enters a multi-stage pipeline that transforms the document into an aggregated result. The aggregation framework of MongoDB is based on the concept of data processing pipeline. The document enters a multi-stage pipeline that transforms the document into an aggregated result.

For example, create a collection of Orders and insert multiple documents:

db.orders.insertMany([{cust_id:"A123",amount:500,status:"A"},{cust_id:"A123",amount:250,status:"A"},{cust_id:"B212",amou nt:200,status:"A"},{cust_id:"A123",amount:300,status:"B"}])Copy the code

Now it is required to query all documents whose status is A and calculate the sum of amount according to cust_ID grouping. The following is achieved by aggregate query:

db.orders.aggregate([{
    $match: {
        status: "A"
    }
}, {
    $group: {
        _id: "$cust_id",
        total: {
            $sum: "$amount"
        }
    }
}])
Copy the code

Query process:

Common stages (steps) for aggregate queries:

function MQL SQL
filter $match where
Projection (alias) $project as
The sorting $sort order by
grouping $group group by
What the results
s k i p / skip/ limit		 limit
The left outer join $lookup left join
An array of $unwind –
Map search $graphLookup –
Faceted search
f a c e t / facet/ bucket		 –
  • Single-purpose aggregation methods

Db.collection.count ()

Db.collection.distinct ()

> db.orders.count()
4
> db.orders.distinct("cust_id")
[ "A123", "B212" ]
> 
Copy the code

Operate MongoDB in Java

After understanding the shell command operation of MongoDB, the API and U corresponding to Java are quite simple. About the common Java project operation MongoDB, Spring operation MongoDB, I organized a small exercise project:

The Java connection

private static final String DATABASE_NAME = "test";

private static final String URL = "Mongo: / / 192.168.242.99:27017 / test";

/** * Get MongoDB connection *@param collectionName
 * @return* /
public MongoCollection<Document> getMongoDB(String collectionName) {
    ConnectionString conn = new ConnectionString(URL);
    MongoClientSettings settings = MongoClientSettings.builder()
            .applyConnectionString(conn)
            .retryWrites(true)
            .build();
    MongoClient mongoClient = MongoClients.create(settings);
    return mongoClient.getDatabase(DATABASE_NAME).getCollection(collectionName);
}
Copy the code

insert

@Test
public void testInsert(a) {
    MongoCollection<Document> collection = getMongoDB("shop");
    Document document = new Document();
    document.append("name"."iphone");
    document.append("price".6799);
    InsertOneResult insertOneResult = collection.insertOne(document);
    System.out.println(JSON.toJSONString(insertOneResult, true));
}
Copy the code

More operations can be seen in the project:

Gitee.com/xblzer/mong…

Replica set (Cluster)

The use of a single MongoDB node was introduced above, but the actual production environment is more likely to use its replica set, which is actually a cluster.

The MongoDB replica set is a set of Mongod processes that maintain the same data set. Replica sets provide high availability and are the foundation of all production deployments.

Replica sets guarantee multiple copies of data on different database servers, and replication provides a degree of fault tolerance against loss to a single database server.

In some cases, replica sets can provide higher read power because clients can send read operations to different servers.

TIP: Since writing data to the replica set is usually written to the primary node and then synchronized to each secondary node, there is a certain network overhead, so the replica set does not improve the write ability.

Maintaining copies of data in different data centers improves data location and availability for distributed applications.

It can also be used specifically for disaster recovery and backup.

That is, the MongoDB replica set has:

  • High availability
  • Data distribution
  • Read/Write Separation – Improved reading skills (remember, not improved, but decreased writing skills)
  • The disaster recovery

A member of a replica set

  • Primary node

The primary node receives all write operations.

A replica set can have only one Primary node, and a Primary logs all changes to its data set in its operation log, oplog (yes, similar to MySQL’s binlog).

The Oplog(Operations Log) is a special set of logs that record all operations related to database modifications (additions, modifications, and deletions). These logs are called Oplog.

The operations of the MongoDB database on the primary node are recorded on oplog, and these logs are asynchronously copied by other secondary nodes. All secondary nodes contain copies of the primary oplog node.

To facilitate replication, all replica set members send heartbeat (pings) to all other members. Any slave node can import Oplog logs from other members.

Oplog operations are idempotent, that is, oplog actions on the target database have the same effect whether they are performed once or multiple times.

Oplog logs have a size, 5% of the size of the physical disk by default.

You can run the following command to view the size of the oplog cluster:

rs.printSlaveReplicationInfo()

Normally, the oplog grows at the same rate as the primary node inserts new files. Once the threshold is exceeded, old logs will be overwritten. Therefore, the oplog log size should be sufficient to add new files within 24 hours, which is generally 72 hours.

If the primary oplog node cannot be synchronized from the secondary node, manually synchronize data. Mongodb provides two data synchronization strategies:

1- Full, the way new nodes are added

2- All replication synchronization after initialization is incomplete, ensuring that each Oplog is the same file

  • Secondary node

Slave nodes copy the master node’s Oplog and apply the operations to their data sets.

If the primary node is unavailable, the eligible secondary node elects a new primary node.

Build replica set

Set up copy is very simple, it is a physical work, dry.

1. Plan hosts

The IP address The host name role
192.168.242.103 mongodb-103 Primary
192.168.242.104 mongodb-104 Secondary
192.168.242.105 mongodb-105 Secondary

2. Set the host name for each node

Vi /etc/hosts # Add 192.168.242.103 mongodb-103 192.168.242.104 mongodb-104 192.168.242.105 mongodb-105 to the end of the fileCopy the code

3. Create the MongoDB configuration file mongod.conf on each node

systemLog:
  #MongoDB sends all log output to a file
  destination: file
  # The path to the log file to which mongod or Mongos should send all diagnostic logging information
  path: "/usr/local/mongodb/logs/mongo.log"
  # When the Mongos or Mongod instance is restarted, Mongos or Mongod appends the new entry to the end of the existing log file.
  logAppend: true
storage:
  dbPath: "/usr/local/mongodb/data"
  journal:
    # Enable or disable persistent logging to ensure that data files remain valid and recoverable.
    enabled: true
processManagement:
  Enable daemon mode to run mongos or Mongod processes in the background.
  fork: true
  pidFilePath: "/usr/local/mongodb/logs/mongod.pid"
net:
  Service instance bound IP address, 0.0.0.0, any IP can access
  bindIp: 0.0. 0. 0
  # Bound port
  port: 27017
replication:
  replSetName: mongoReplSet
Copy the code

Unlike the single-node configuration file, a replication configuration is added at the end.

4. Start MongoDB on each node

mongd -f /usr/local/mongodb/conf/mongd.conf
Copy the code

5. Configure the replica set

// Start the replica set on mongod-103
// First use the mongo command to enter the console
mongo
// Open the replica set
rs.initiate() 
// Add node rs.add(" machine name: port ")
rs.add("mongod-104:27017")
rs.add("mongod-105:27017")
Copy the code

A set of MongoDB replicas is built.

Usage scenarios of MongoDB

The previous brief introduction of MongoDB single node, replica set installation and use, some of its related API I also summarized on GitHub, and interested friends can have a look.

So where does MongoDB fit in? My current project uses MongoDB for vehicle GPS positioning. In fact, MongoDB can be used as a technology choice for data storage in many scenarios, such as:

  • Many large projects merchandise articles content reviews
  • A lot of iot systems, shared electricity/bikes
  • The bank’s financial data center
  • Location navigation service

Finally, here’s a picture:

MongoDB is very marketable!

The first public line 100 li ER, welcome the old iron attention reading correction. GitHub gitee.com/xblzer/Java…