preface

Hello, friends, I am your PubDreamcc, this blog post is about my GitHub warehouse Node learning tutorial material, welcome friends to like and star, your like is my motivation to continue to update.

GitHub repository address: Node learning tutorial

Ok, without further ado, today we will play an interesting demo:

Node implements crawler to dig gold50Quality front-end articles

Take a look at the following effects:

The server will update the content every 10 minutes. Ok, here we go

The technology used

  • express

  • Superagent (server sending request module)

  • art-template

  • Nuggets official API

The first step

To find the interface

Let’s take a look at the back end API for getting front end articles on the nuggets front page.

Open the official website of Nuggets, (F12) open the developer mode to check the network option, we can see the following API to obtain the article interface:

Open the developer mode, we can easily find the interface to get articles, which is easy to do, to tell the truth, front-end development, as long as there is an interface, that is equal to everything, we can enjoy coding ~

The second step

Create server file app.js, and send request to get article data through superagent module.

App.js is our server-side code, and the data required by the crawler is saved by sending a request from the server-side.

// Define a function to get information about the front page of the nuggets article
function getInfo () {
  // Use the superagent module to send a request to get the first 20 articles. Note the request header Settings and the POST request body data (request parameters)
  superagent.post('https://web-api.juejin.im/query').send(params).set('X-Agent'.'Juejin/Web').end((err, res) = > {
    if (err) {
      return console.log(err)
    }
    // Save all article information
    const array1 = JSON.parse(res.text).data.articleFeed.items.edges
    const num = JSON.parse(res.text).data.articleFeed.items.pageInfo.endCursor
    // Select posts with more than 50 likes
    result = array1.filter(item= > {
      return item.node.likeCount > 50
    })
    params.variables.after = num.toString()
    // Send the request again for another 20 articles
    superagent.post('https://web-api.juejin.im/query').send(params).set('X-Agent'.'Juejin/Web').end((err, res) = > {
      if (err) {
        return console.log(err)
      }
      const array2 = JSON.parse(res.text).data.articleFeed.items.edges
      const result2 = array2.filter(item= > {
        return item.node.likeCount > 50
      })
      result2.forEach(item= > {
        result.push(item)
      })
    })
  })
}
// Call once to get data
getInfo()

// Set the timer to 10 minutes to update the data
setInterval((a)= > {
  getInfo()
}, 10*1000*60)
Copy the code

The x-agent attribute of the request header must be set when the superagent sends the POST request, otherwise there will be an error, and the fixed request parameter params can be written according to the official website of the nuggets.

The third step

The template engine renders the data and sends the results to the browser for rendering

This step involves rendering the HTML page with the help of a template engine, rendering the results from the second step into the page, and eventually returning them to the browser for rendering.

App. Js code:

// Listen for routes
app.get('/', (req, res, next) => {
  res.render('index.html', {
    result
  })
})

// Bind ports to start services
app.listen(3000, () = > {console.log('running... ')})Copy the code

Template index.html code:

<! With bootstrap style, notice the use of template syntax
<ul class="list-group">
  {{each result}}
  <li class="list-group-item">
    <a href="{{$value.node.originalUrl}}" target="_blank">{{$value.node.title}}</a>
    <img data-v-7bf5f1fe="" src="https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-assets/v3/static/img/zan.e9d7698.svg~tplv-t2oaga2asx-image.image">
    <span>{{$value.node.likeCount}}</span>
  </li>
  {{/each}}
</ul>
Copy the code

Write in the back

If you need the source of the project can be found in GitHub corresponding repository of Node learning Demo case folder, thank you!