Writing in the front

Individual written school graduation design project, as the front end, also is the first time contact with node to write the backend interface and crawl data, the start was to take a page up data, but does not find many page unique id or class, lead to the difficulty coefficient is very high, crawl data just crawl data web interface is open, So I plan to change the way to get data —- climb interface (of course, you can also manually import the database, but because I am lazy, as well as the graduation data is more, thinking of the manual cost is relatively large, they also want to learn more knowledge, began to start a variety of online information……) Without further ado, let’s get to the point

Data format for the crawl

{
  IsSuccess: true,
  Data: [
    { os: 'android', count: 2102 },
    { os: 'ios', count: 276 },
    { os: 'win10', count: 42 },
    { os: 'win7', count: 29 },
    { os: 'mac', count: 15 },
    { os: 'Others', count: 14 }
  ]
}
Copy the code

🔗 Connecting to the database

  • First installation mongo database, online tutorial www.runoob.com/mongodb/mon…
  • Install dependenciesnpm install mongoose
  • Create a config folder (I have separated the database connection from the database address)

  • The config. Ts file

Since mongodb initialization does not require a password, I ran it locally and did not set a password. If I want to deploy it on a server, I need to set the account password

// database address: 'mongodb:// username: password @ip: port number/database '; Module. Exports = {mongo: 'mongo: / / 127.0.0.1:27017 / analyst'}Copy the code
  • mongoose.ts
// define schema and model const mongoose = require('mongoose'); const config = require('./config'); const mongooseData = () => { mongoose.connect(config.mongodb, { useNewUrlParser: true, useUnifiedTopology: true }, (err) => { if (err) { throw err; } else {console.log("MongoDB connection successful!" )}}); } export default mongooseData;Copy the code

Make sure the database is open and you can connect by simply executing the mongooseData() method

Create the model and schema for the database

model

  • TestModel.ts

Because the interface data to crawl is relatively simple

const mongoose = require('mongoose'); // @ts-ignore export const TestSchema = new Schema({OS: String, count: Number })Copy the code
  • TestModel.ts
import { TestSchema } from '.. /schemas/TestSchema'; const mongoose = require('mongoose'); // Create model where test corresponds to the conllection of test in mongodb database. export const Test = mongoose.model('test', TestSchema);Copy the code

Encapsulate the data creation method

Mongoose has its own model.create () method to insert data into the database, but in order to facilitate the writing interface of my later design, I packaged the addition, deletion, checking and modification into a separate method to avoid code redundancy

Import {Test} from '... /.. /db/models/TestModel'; Export const Create = (model, param,callback) => {model. Create (param, (err, err, data) => { if (err) throw err; If (data) {const req = {data: data} callback(req)} else {const req = {MSG: 'create failed ', error: '34000' } callback(req) } }) }Copy the code

Crawl the interface method

I also encapsulate a request method to make it easier to distinguish the requested method

const request = require("request"); /* * baseUrl -> method -> params -> request parameters except get */ export const crawlerPostFun = (baseUrl, method, params); CallBack) => {request({url: baseUrl,// you want to request the address method: method,// request method post get json: true, headers: {"content-type": "application/json; charset=utf-8", "Access-Control-Allow-Methods": 'POST,GET,OPTIONS', }, body: }, function (error, response, body) {if (! error && response.statusCode == 200) { callBack(body); }})}Copy the code

Crawl the core interface

Having said that, the core code is here, and now it’s time to crawl the interface data using various methods that are encapsulated

import mongooseData from '.. /db/config/mongoose'; import { Test } from '.. /db/models/TestModel'; import { Create } from './utils'; import { crawlerPostFun } from './utils/crawler'; mongooseData(); Const test = (value) => {console.log('----value',value) if(value){const test = (value) => {console rows = value.Data; rows.forEach((item) => { Create(Test, { os: item.os, count: Item. The count,}, (res) = > {/ / database callback console. The log (' -- -- -- -- - res, res); }) }) } } crawlerPostFun("https://backsite.hubing.online/Monitor/OsStatis","POST",{ "TimeQuantum": 4, "appKey": "5ea9a55e5b0dd76c634bed78", "eTime": "", "sTime": "" },test)Copy the code
  • After executing this file, the console prints the data

  • Now the database

conclusion

The body parameter of the request post request was not passed to the server. Later, it was found that there was no request method to allow post, so we must pay attention to the attributes in the request header head.