Go climbs the Top250 data list of Douban movies

Node inside, you can use webmaster tools first
tool.oschina.net/regex/Test whether the data extracted from the written regular expression is correct:

The div node with class as PIC is the regex for the movie ranking ID number and the movie image and movie name information:

regExp := `<div class="item">[\s\S]*? <div class="pic">[\s\S]*? <em class="">(.*?) <\/em>[\s\S]*? <a href=".*?" >[\s\S]*? ! [] ((. *?) ) `Copy the code

The node contains the alias of the movie and other information, and the regular expression is:

regExp := `<div class="item">[\s\S]*? <div class="pic">[\s\S]*? <em class="">(.*?) <\/em>[\s\S]*? <a href=".*?" >[\s\S]*? ! [] ((. *?) )[\s\S]*? div class="info[\s\S]*? class="hd"[\s\S]*? class="title">(.*?) <\/span>[\s\S]*? class="other">(.*?) <\/span>`Copy the code

The tag contains the information of the director and the leading actor of the movie. The tag contains the information of the director and the actor of the movie. In the tag, the data of the director and the actor can be extracted by using line breaks.

regExp := `<div class="item">[\s\S]*? <div class="pic">[\s\S]*? <em class="">(.*?) <\/em>[\s\S]*? <a href=".*?" >[\s\S]*? ! [] ((. *?) )[\s\S]*? div class="info[\s\S]*? class="hd"[\s\S]*? class="title">(.*?) <\/span>[\s\S]*? class="other">(.*?) <\/span>[\s\S]*? <div class="bd">[\s\S]*? <p class=".*?" >([\s\S]*?) <br>([\s\S]*?) <\/p>`Copy the code

The tag contains the movie’s star rating and rating data. The rules for extracting stars and ratings and the types analyzed above, so the final regular expression is:

regExp := `<div class="item">[\s\S]*? <div class="pic">[\s\S]*? <em class="">(.*?) <\/em>[\s\S]*? <a href=".*?" >[\s\S]*? ! [] ((. *?) )[\s\S]*? div class="info[\s\S]*? class="hd"[\s\S]*? class="title">(.*?) <\/span>[\s\S]*? class="other">(.*?) <\/span>[\s\S]*? <div class="bd">[\s\S]*? <p class=".*?" >([\s\S]*?) <br>([\s\S]*?) <\/p>[\s\S]*? span class="rating_num".*? average">(.*?) <\/span>`Copy the code

The complete code to extract all movie data on each page is as follows:

RegExp := '<div class="item">[\s\ s]*? <div class="pic">[\s\S]*? <em class="">(.*?) <\/em>[\s\S]*? <a href=".*?" >[\s\S]*? ! [] ((. *?) )[\s\S]*? div class="info[\s\S]*? class="hd"[\s\S]*? class="title">(.*?) <\/span>[\s\S]*? class="other">(.*?) <\/span>[\s\S]*? <div class="bd">[\s\S]*? <p class=".*?" >([\s\S]*?) <br>([\s\S]*?) <\/p>[\s\S]*? span class="rating_num".*? average">(.*?) <\/span> '// Use the re to match find := regexp.MustCompile(regexp) // where result is the HTML data obtained in the previous step content := The find. FindAllStringSubmatch (result, 1) / / return detailed two-dimensional slices matchingCopy the code

The content data format is [][] STRING two-dimensional slice, so we can use the for statement to traverse the number group to obtain useful information.

4. Use analysis to obtain valid data

Create a MySQL table to store the saved data

CREATE TABLE `top250` (
  `id` int(20) NOT NULL AUTO_INCREMENT,
  `title` varchar(20) DEFAULT '',
  `image` varchar(100) DEFAULT '',
  `subtitle` varchar(255) DEFAULT '',
  `other` varchar(255) DEFAULT NULL,
  `personnel` varchar(255) DEFAULT '',
  `info` varchar(255) DEFAULT '',
  `score` varchar(10) DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4;
Copy the code

Using the database/SQL packages and use the MySQL driver: MySQL software including database tutorial click: Go tutorial SQL database view

Func insert(movies [][]string) { ,? ,...). Make ([]interface{}, 0, 0) // make([]interface{}, 0, 0) For _, val := range movies {// Placeholder valueStrings = append(valueStrings, "(? ,? ,? ,? ,? ,? ,? ,?) ") for i := 1; i < len(val); i++ { valueArgs = append(valueArgs, val[i]) } } db, err := sql.Open("mysql", "Root: a root @ TCP (127.0.0.1:3306)/test") if err! = nil {log.fatal (err)} defer db.close () // INSERT INTO 'top250' (id,title,image,subtitle,other,personnel,info,score) VALUES %s", strings.Join(valueStrings, ",")) fmt.Println(sqlName) // insert stmt, err := db.Prepare(sqlName) if err ! = nil { log.Fatal(err) } fmt.Println(valueArgs) res, err := stmt.Exec(valueArgs...) if err ! = nil { log.Fatal(err) } lastId, err := res.LastInsertId() if err ! = nil { log.Fatal(err) } rowCnt, err := res.RowsAffected() if err ! = nil { log.Fatal(err) } log.Printf("ID = %d, affected = %d\n", lastId, rowCnt) }Copy the code

We only need to create a chan channel in main() to call SpiderDouBan() concurrently:

channel := make(chan int) for i := start; i <= end; i++ { go SpiderDouBan(i, channel) } for i := start; i <= end; I++ {FMT. Println (" is the first "+ strconv. Itoa (< - channel) +" mission accomplished ")}Copy the code

Here, at the end of the SpiderDouBan() function, we send data to the chan channel:

func SpiderDouBan(index int, ch chan int) { // ... Omit code //... // if ch < -index}Copy the code

Which complete Go climb douban Top250 movie code please click: gitee.com/lisgroup/go… To view

SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶ SpiderDouBan() ¶

Original link: www.sdk.cn/details/djg…

SDK is a neutral communities, there are a variety of front knowledge, has the rich API, developers have a love of learning artificial intelligence, have humor developers take you learn python, there will be the next hot HongMeng, when various elements together, let us together to build professional, fun and imaginative valuable developer community, Help developers realize their self-worth!