Scrapy crawls the Cat’s Eye movie and stores it in the MongoDB database

1. What is MongoDB database?

MongoDB is an open source database system based on distributed file storage written in C++ language.

A record in MongoDB is a document, a data structure consisting of fields and value pairs. MongoDB documents are similar to JSON objects. The value of the field may include other documents, arrays, and document arrays.

2. Installation and Environment

Create database, collection & Delete database, collection

Open the Mongo shell

use test

This creates a database called test, but the database is still empty

show dbs

You can see that there is no test database (lianjia database was created by me before, everyone ignore, haha!!)

The admin 0.000 GB


Lianjia 0.000 GB


The local 0.000 GB

Add a collection to the database

Db. CreateCollection (” movies “)


{” OK “: 1}

If you run show DBS, the test database will be displayed.

The admin 0.000 GB


Lianjia 0.000 GB


The local 0.000 GB


The test of 0.000 GB

Leave the collection of movies behind and execute the delete command

db.movies.drop()


true

Run the show collections command and collections will no longer appear

Then execute the command to delete the database

db.dropDatabase()

Shell returns {” dropped “:” test “, “OK” : 1} Check the database again

show dbs

The test database is gone

The admin 0.000 GB


Lianjia 0.000 GB


The local 0.000 GB

4. Insert the document

Db.createcollection ('movies') # create database show DBSCopy the code

5. Query data

Query with specific criteria

Db.movies. Find ({‘ language ‘:’ Chinese common law ‘}).pretty()

Let’s look at the data that’s returned

{


“_id” : ObjectId (” 5 a1ff19bbff1b3cbd8e8f811 “),


Title: Farewell my Concubine,


“Directed_by” : “Chen Kaige”


“The actor” :


Leslie Cheung,


“Gong li”


].


“Type” : “plot”,


Country: China,


Chinese common law


“Release_time” : “1993-01-01”,


“Time”, “171”,


Score: 9.5


}

AND conditional query

Db. Movies. Find ({” title “:” farewell my concubine “, “language”, “Chinese law”}). Pretty ()

Only movies that meet both criteria will be found

OR conditional query

Db. Movies. Find ({$or: [{” title “:” farewell my concubine “}, {‘} “language”, “English]}). Pretty ()

The parentheses here are a little bit complicated, so you have to be careful

The conditional greater than operator – $gt finds movies with ratings greater than 9

Db. Movies. Find ({‘ score: {$gt: 9}}). Pretty ()

The less than operator – $lt

Db. Movies. Find ({‘ score: {$lt: 10}}). Pretty ()

The greater than or equal operator – $gte, the less than or equal operator – $lte, and the unequal operator – $ne are not listed here

The limit and SKIP conditional operator limit() methods take a numeric parameter that specifies the number of records to read from MongoDB

db.movies.find().limit(1).pretty()

This returns a movie message

{


“_id” : ObjectId (” 5 a1feb04bff1b3cbd8e8f80e “),


Title: Shawshank Redemption,


Directed_by: “Frank”,


Actor: “Morgan. The man “,


“Type” : “plot”,


“Country” : “United States”,


Language: English


“Release_time” : “1994-09-10”,


“Time”, “142”,


Score: 9.6


}

The skip() method skips a specified amount of data. The skip method also takes a numeric parameter as the number of records to skip

db.movies.find().skip(1).pretty()

6. Delete the document

7. Update documents

Db. Movies. Update ({” title “:” the shawshank redemption “}, {$set: {” title “:” MongoDB_Test ‘}})

MongoDB_Test = value MongoDB_Test

{


“_id” : ObjectId (” 5 a200300bff1b3cbd8e8f813 “),


“Title” : “MongoDB_Test”,


Directed_by: “Frank”,


“The actor” :


“Morgan Morleyman.”


“Robins”


].


“Type” : “plot”,


“Country” : “United States”,


Language: English


“Release_time” : “1994-09-10”,


“Time”, “142”,


Score: 9.6


}

The $set operator updates a field in the document instead of replacing it all with {$set: {:,… }}

To modify multiple documents, set the multi parameter to true

Db. Movies. Update ({” title “:” the shawshank redemption “}, {$set: {” title “:” MongoDB_Test ‘}}, {multi: true})

The above two operations will change the original data. If you want to add a value to the original value, you need to add $push, such as adding a new actor to the actor

Db. Movies. Update ({” title “:” farewell my concubine “}, {$push: {‘ actors’ : ‘Treehl’}})

So the actor has three actors

The actor “:


Leslie Cheung,


“Gong”,


“Treehl”


]

8. Sorting and indexing

9. The aggregation

In MongoDB, aggregate is mainly used to process data (such as statistical average, sum, etc.) and return calculated data results

> db.movies.aggregate([{$group:{_id:'$title',num_movies:{$sum:1}}}])Copy the code

return

{” _id “:” Farewell my Concubine “, “num_movies” : 1}


MongoDB_Test: MongoDB_Test, num_movies: 1}

Understand the command

$group: groups the documents in the collection, which can be used for statistical results. $sum: calculates the sumCopy the code

This is a recommendation that you can refer to the aggregation of rookie tutorials

10. Python operates MongoDB

MongoClient = MongoClient(' mongohost ', Movies = db.movie_name_list = [] movie_name = {'title': 'Shawshank Redemption ',' Description ': 'Little Accomplished Young Banker Andy ',' Year ': '1991'} movie_name_list.append(movie_name) # Print out data for item in movie_name_list: Find (): print(item) for item in movies.find(): print(item)Copy the code

{'_id': ObjectId(' 5a20c4f3AA85C109e881C7C7 '), 'title': 'Shawshenk Redemption ',' Description ': 'Young Banker Andy ', 'year': '1991'}Copy the code