I am learning go recently. I used to use Python crawlers. This time I will try to write with Go. Compared with more fire Java, PHP, javascript, python language, such as go the advantage of natural ability of asynchronous. Multithreaded programming can be implemented in a very concise and elegant way. Go’s restrictions on variables are a boon for some obsessive-compulsive patients. So we beginners can also try to write crawlers in this language.
Today I wrote a simple crawler with it as an exercise. The first site to be collected is today’s headlines
Package main import ("net/url" "net/ HTTP ""bytes" "FMT "" IO /ioutil") // ProxyServer (www.16yun.cn "t.16yun.cn:31111" type ProxyAuth struct { Username string Password string } func (p ProxyAuth) ProxyClient() http.Client { var proxyURL *url.URL if p.Username ! = ""&& p.Password! ="" { proxyURL, _ = url.Parse("http://" + p.Username + ":" + p.Password + "@" + ProxyServer) }else{ proxyURL, _ = url.Parse("http://" + ProxyServer) } return http.Client{Transport: &http.transport {Proxy: http.proxyURL (ProxyURL)}}} func main() {targetURI := "HTTPS :www.toutiao.com/" // Initializes Proxy HTTP client client := ProxyAuth{"username", "password"}.ProxyClient() request, _ := http.NewRequest("GET", targetURI, NewBuffer([] byte(' '))) // Set proxy-tunnel // rand.seed (time.now ().unixnano ()) // Tunnel := rand.intn (10000) // request.Header.Set("Proxy-Tunnel", strconv.Itoa(tunnel) ) response, err := client.Do(request) if err ! = nil { panic("failed to connect: " + err.Error()) } else { bodyByte, err := ioutil.ReadAll(response.Body) if err ! = nil {FMT.Println(" Error reading Body ", err) return } response.Body.Close() body := string(bodyByte) fmt.Println("Response Status:", response.Status) fmt.Println("Response Header:", response.Header) fmt.Println("Response Body:\n", body) } }Copy the code