What is the mongo

A database based on distributed file storage, intermediate between non-relational and relational databases.

Compare MongoDB with MySQL

MongoDB	MySQL
Database	Database
A Collection	Table
Document	Row
Field	Column

What are the features of MongoDB

(1) Set-oriented data storage, data storage in JSON format;

(2) the query function is powerful, almost can realize all relational database single table query function;

(3) Its own distributed file system facilitates cluster deployment;

(4) It provides support for the Map-Reduce computing framework.

Why MongoDB is fast

(1) No transaction constraint;

(2) the use of memory mapping technology, write data as long as completed in memory can be returned to the application program, write disk is asynchronous operation; Hot data into memory; Data is concentrated on disks, which reduces the time of magnetic head positioning.

(3) No join is used. The existence of no relation means that each data has a separate storage space, and then a clustered index to point to.

MongoDB Application Scenarios

(1) Real-time data processing of the website, such as PV, UV, collection times of statistical goods, adding times to shopping cart, exposure times, etc.

(2) Server log recording;

(3) Large amount of data, TB or even PB level data storage;

(4) Data models change frequently and need to support horizontal scaling

(5) For game scenes, MongoDB is used to store game user information, and users’ equipment and points are directly stored in the form of embedded documents for easy query and update

In logistics scenario, MongoDB is used to store order information, and the order status will be updated continuously during the delivery process. The order status is stored in the form of MongoDB embedded array, and all changes of the order can be read out in a single query.

In social networking scenarios, MongoDB storage is used to store user information and the information published by users in the circle of friends, and the nearby people and places are realized through geographical location index

In the Internet of Things scenario, MongoDB is used to store information about all connected smart devices and log information reported by the devices, and analyze the information in multiple dimensions

Live video, use MongoDB to store user information, gift information, etc.

MongoDB does not apply to scenarios

(1) Data modification requires strict thing control;

(2) The business relationship of functions is complex and involves multi-table association;

(3) Relatively important business data, such as orders, goods, users and other data.

Mongo collection

MongoDB sets can be analogous to MySQL tables, and fields can be analogous to MySQL columns. The difference is that the filed of the set is dynamically expandable, and there is no limit on the data type, unlike Mysql, which needs to define the column name and the field type of the column.

In addition, collections do not need to be defined in advance, as mysql tables do. The collection is generated by inserting a single piece of JSON data using the db.collectionName.insert() statement.

Collections consist of documents, each of which is a single piece of JSON data.

Inserted into the document

Inserting a document into a collection (analogous to inserting a piece of data into a table) :

`db.getCollection("b2b_theme_activity").insert( { "merchant_code": "1ed0528595197975fa72f5ec8f9e0c62", "dfx_type": NumberInt (" 1 "), "create_time" : ISODate (" the 2020-03-16 T22:36:01. 964 z "), "member_code" : "E69e61a77b1ccc7b90be7995c6a6021f", "user_name" : "li he," and "merchant_name" : "nanjing is good at home intelligent household co., LTD.", the source: NumberInt("10"), "visit_duration": "", type: NumberInt("11"), "member_name": "Test enterprise account_01 ", "theme_activity_page_title": "Test test01", "login_name": "17366180925"," user_ID ": "1161150535707852801", event: NumberInt("20"), "theme_activity_id": "1238383672212533250"});Copy the code

Modify the document

grammar

The syntax for modifying a document is:

db.collection.update(

  <query>,

  <update>,

  {

   upsert: <boolean>,

   multi: <boolean>,

   writeConcern: <document>

  }

)
Copy the code

Query: The query condition is similar to where in the SQL update query.
Update: An update object and some update operators (e.g.,,inc…) SQL update set ()
Upsert: Optional, this parameter means whether to insert objNew if no update record exists. True is insert, default is false, no insert.
Multi: Optional. By default, mongodb updates only the first found record. If this parameter is true, mongodb updates all found records.
WriteConcern: Optional, level of exception thrown.

example

T_test_log = /ops/ API /b2b/theme-activities; /ops/ API /b2b/theme-activities;

db.t_test_log.update({'requestUri':'/ops/api/b2b/theme-activities'},{$set:{'requestSource': '1'}})
Copy the code

{multi:true} if you want to modify all qualified document data, add: {multi:true}

db.t_test_log.update({'requestUri':'/ops/api/b2b/theme-activities'},{$set:{'requestSource': '2'}}, {multi:true})
Copy the code

Delete the document

grammar

db.collection.remove(
   <query>,
   <justOne>
)
Copy the code

Query: the query
JustOne: whether to delete only the first document that meets the conditions. True: yes, false: no Default false

example

Delete the first document that meets the criteria:

db.t_test_log.remove({'requestUri':'/ops/api/b2b/theme-activities/11111/22222/testPut'},1)
Copy the code

Delete all documents that match the criteria:

db.t_test_log.remove({'requestUri':'/ops/api/b2b/theme-activities/11111/22222/testPut'})
Copy the code

The query

A simple query

select * from  spu where  spu_name="test0123"

db.spu.find({ spu_name:"test0123" })
Copy the code

and

select * from  spu where  spu_name="test0123" and  type=3

db.spu.find({spu_name:"test0123", type:"3" });
Copy the code

like

select * from  user where  username like "%test0123%"

db.user.find({username:/test0123/});
Copy the code

Greater than or equal to or less than or equal to

$lt, $lte, $gt, $gte select * from user where age>=10 and age<=20 db.user.find({age:{$gte:10, $lte:20} });Copy the code

Is equal to null

select * from  user where  address is null

db.user.find({ address: null });
Copy the code

Not null

select * from  user where  address is not null

db.user.find({ address: {$ne:null} });
Copy the code

Not null and not “”

select * from user where address is not null and address ! ='' db.user.find({ address: {$nin:[null,"" ]} });Copy the code

in/not in

The db. The activity. The find ({event: {$in: [50,60,70]}}); The db. The activity. The find ({event: {$nin: [50,60,70]}});Copy the code

count

db.activity.find().count(); The db. The activity. The find ({event: {$in: [50,60,70]}}). The count ();Copy the code

distinct

db.activity.distinct("user_name");
Copy the code

sort

db.getCollection("activity").find({type:10}).sort({createTime:1});
Copy the code

Period of time

Query the data of a certain time range

The db. The activity. The find ({createTime: {$gte: ISODate (' 2020-03-01 T00:00:00. 000 z '), $lte: ISODate (' 2020-03-11 T00:00:00. 000 z ')}});Copy the code

The aggregation

Aggregation query

In MongoDB, aggregate is mainly used to process data (calculate parity value, sum, etc.) and return the calculated data results. Similar to groups in SQL.

$sum sum

Group by user_id to find the total order amount of each user

db.order.aggregate([{$group : {_id : "$user_id", total_order_money: {$sum : "$order_money"}}}])
Copy the code

$avg average

Group by user_id to find the average payment amount for each user order

db.order.aggregate([{$group : {_id : "$user_id", avg_order_money: {$avg : "$order_money"}}}])
Copy the code

$min minimizes

Group by user_id to find the minimum payment amount for each user order

db.order.aggregate([{$group : {_id : "$user_id", min_order_money: {$min : "$order_money"}}}])
Copy the code

$Max

Group by user_id to find the maximum payment amount for each user order

db.order.aggregate([{$group : {_id : "$user_id", max_order_money: {$max : "$order_money"}}}])
Copy the code

$push

Group by user_id to view all payment methods for each user (without reweighting)

db.order.aggregate([{$group : {_id : "$user_id", payMethods: {$push: "$pay_method"}}}])
Copy the code

$addToSet

View all payment methods for each user, grouped by user_id

db.order.aggregate([{$group : {_id : "$user_id", payMethods: {$addToSet : "$pay_method"}}}])
Copy the code

$first

Group by user_id to find the amount of each user’s first order

db.order.aggregate([{$group : {_id : "$user_id", first_order_money : {$first : "$order_money"}}}])
Copy the code

$last

Group by user_id to find the amount of each user’s last order

db.order.aggregate([{$group : {_id : "$user_id", last_order_money: {$last : "$order_money"}}}])
Copy the code

Polymerization pipe

Aggregate pipes are a new feature introduced in MongoDB 2.2. It consists of stages. After a document is processed in one Stage, the aggregation pipeline passes the results to the next Stage.

Polymerization pipeline functions:

Filter documents and query the documents that meet the requirements
Transform the document to change the output form of the document

The following is the flow chart of the aggregation pipeline on the official website:

In the figure above, match is the screening condition, and the data that meets the conditions is filtered out and transmitted to the next stage match is the screening condition, and the data that meets the conditions is filtered out and transmitted to the next stage Group for grouping sum calculation. Finally, return Results. Match, match, match, and group are all phase operators, and sum used in the group used in the phase group is the expression operator.

Common phase operators

Stage operator	describe
$project	Modify the structure of a document, which can be used to rename, add, or delete fields in the document.
$match	Filter conditions. $match appears at the front of the pipe as much as possible to filter out the required data, which can improve efficiency in subsequent phases.
$group	grouping
$sort	The sorting
$limit	Limit the number of results, which can be used to achieve paging queries
$skip	Skip the specified number of documents and return the remaining documents.
$unwind	Divide the array type fields in a document into multiple pieces, each containing a value from the array.

Example 1(Aggregate query)

Group by user_id and find the average value of each user’s order whose order status is 20

mysql:
select  user_id, avg(total_amount) from t_child_order 
where child_order_status=20
group by merchant_code;

mongo:
db.t_child_order.aggregate
( {$match:{child_order_status:20}}   ,       
{$group:{_id:"$user_id",  avgDur:{$avg:"$total_amount" }}  }

);
Copy the code

Match indicates the query condition. Match indicates the query condition. Match indicates the query condition. Group: groups by user_id and calculates the average data of visit_duration with the same user_id.

Example 2 (Pipe operation)

Requirement: count the activity ID 1239575334792146946 from 2020-03-01 to 2020-04-18, and the operation event is to view the activity details page (Event :10). The daily PV, average daily access duration, and daily user set.

Db.activity. aggregate([// condition, filter out the data, pass to the next stage {" activity_id" : {" activity_id" : '1239575334792146946', "create_time" : "{" $lte: ISODate (' 2020-04-18 T00:00:00. 000 z ')," $gte ": ISODate (' 2020-03-01 T00:00:00. 000 z ')}," event ": {"$project" : {"create_time" : {"$dateToString" : {"format" : "%Y-%m-%d", "date" : {"format" : "%Y-%m-%d", "$create_time"}}, "visit_duration" : 1, "user_id" : 1}}, and then pass to the next stage {"$group" : {"_id" : "$create_time", "ALIAS_PAGE_PV" : { "$sum" : 1 }, "ALIAS_AVG_VISIT_DURATION" : { "$avg" : "$visit_duration"}, "ALIAS_USER_ARRAY" : {" $addToSet ":" $user_id "}}}, / / set the result to show {$project ": {" _id" : 0, "create_time" : "$_id", "ALIAS_PAGE_PV" : 1, "ALIAS_AVG_VISIT_DURATION" : 1, "ALIAS_USER_ARRAY" : Sort {1}}, / / "$sort" : {" create_time ": 1}}, / / article limit results for {$limit" : 3}]);Copy the code

The query results are shown as follows:

Map Reduce

Map-reduce is a computing model that decomposes complex computations and then merges the results.

To use MapReduce, you need to implement the Map and Reduce functions.

The Map function invokes emit(key, value), traverses all records in the collection, and passes the key and value to the Reduce function for processing.

The Map function must call emit(key, value) to return the key-value pair.

db.collection.mapReduce( function() {emit(key,value); }, // Map function(key,values) {return reduceFunction}, //reduce function {out: collection, query: document, sort: document, limit: number } )Copy the code

Map: mapping function (generating a sequence of key-value pairs as arguments to the reduce function).
Reduce statistics function. The reduce function converts key-values into key-value, that is, an array of values into a single value.
Out Stores the collection of statistical results. (If this parameter is not specified, a temporary collection is used, which is automatically deleted after the client is disconnected.)
Query A filter condition in which the map function is called only for documents that meet the condition. (query. Limit, sort
The sort sort parameters combined with sort and limit, which also sort documents before sending them to the Map function, optimize the grouping mechanism
Limit The maximum number of documents sent to the map function (without a limit, sort alone is not useful)

The following figure shows the map-reduce workflow provided by MongoDB official website. From the order set, find the documents with status as A, group them according to CUST_id, and calculate the total order amount of each CUST_id.

Example Map-reduce

Db.activity. mapReduce(function() {emit(this.user_id, this.visit_duration); // Emit (this.user_id, this.visit_duration); // Emit (this.user_id, this.visit_duration); }, function(key, values) { return Array.sum(values) }, { query: { activity_id: "1236122613668032514", event:10 }, out:"temp_result" } ).find();Copy the code

Db.activity. mapReduce(function() {emit(this.user_id, this.visit_duration); }, function(key, values) { var total= Array.sum(values); var sizes= values.length; return total/sizes }, { query: { activity_id: "1236122613668032514", event:10 }, out:"temp_result" } ).find();Copy the code

Db.activity. mapReduce(function() {emit(this.user_id, 1); }, function(key, values) { return Array.sum(values)}, { query: { activity_id: "1236122613668032514", event:10 }, out:"temp_result" } ); db.temp_result.find();Copy the code

MongoDB dry goods summary

What is the mongo

Compare MongoDB with MySQL

What are the features of MongoDB

Why MongoDB is fast

MongoDB Application Scenarios

MongoDB does not apply to scenarios

Mongo collection

Inserted into the document

Modify the document

grammar

example

Delete the document

grammar

example

The query

A simple query

and

like

Greater than or equal to or less than or equal to

Is equal to null

Not null

Not null and not “”

in/not in

count

distinct

sort

Period of time

The aggregation

Aggregation query

$sum sum

$avg average

$min minimizes

$Max

$push

$addToSet

$first

$last

Polymerization pipe

Common phase operators

Example 1(Aggregate query)

Example 2 (Pipe operation)

Map Reduce

Related Posts

8 Tips for Efficient Data analysis in Python

Postgresql. conf Log file for parameter Settings

Migrate Elastic Search: Upgrade to Elastic Search 2.x-7.x Custom Analyzer