Creeper, the next generation of open source crawler framework based on simple scripts

Create hacker_news.crs

page(@page=1) = "https://news.ycombinator.com/news?p={@page}"

news[]: page -> $("tr.athing")
    title: $(".title a.storylink").text
    site: $(".title span.sitestr").text
    link: $(".title a.storylink").href
Copy the code

Then, create main.go

package main

import "github.com/wspl/creeper"

func main() {
    c : = creeper.Open("./hacker_news.crs")
    c.Array("news").Each(func(c *creeper.Creeper) {
        println("title: ", c.String("title"))
        println("site: ", c.String("site"))
        println("link: ", c.String("link"))
        println("= = =")})}Copy the code

Build and run. Console will print something like:

title: Samsung chief Lee arrested as S.Korean corruption probe deepens site: reuters.com link: http://www.reuters.com/article/us-southkorea-politics-samsung-group-idUSKBN15V2RD === title: ReactOS 0.4.4 Released site: reactos.org link: https://reactos.org/project-news/reactos-044-released = = = title: FeFETs: How this new memory stacks up against existing non-volatile memory site: semiengineering.com link: http://semiengineering.com/what-are-fefets/Copy the code

Town is a lambda like expression for saving (in)mutable string. Most of the time, we used it to store url.

page(@page=1, ext) = "https://news.ycombinator.com/news?p={@page}&ext={ext}"
Copy the code

When you need town, use it as if you were calling a function:

news[]: page(ext="Hello World!" ) -> $("tr.athing")Copy the code

Hey, you might have noticed that the @page parameter is not used. Yeah, it is a special parameter.

Expression in town definition line like name="something", represents parameter name has a default value "something".

Incidentally, @page is a parameter that will automatically increasing when current page has no more content.

Node has name. title is a field name, represents a general string data. news[] is a array name, represents a parent structure with multiple sub-data.

If a node owned page and fun at the same time, page should on the left of ->, fun should on the right of ->. Which is page -> fun

Plutonist

Impl. MOE, making @ WSPL