Some websites will jump to different sites (PC and M) according to the difference of user-Agent, and also give different hints according to different versions, etc., while the change of User-Agent is the basic posture in crawler

Use Go to write web crawler or need to simulate the user-agent of the browser (Request Headers), do you feel very cumbersome, you have to find the user-agent of the Request Headers. I ran into this problem earlier, so I came up with fake-UserAgent to solve your and mine pain points

Project address: github.com/EDDYCJY/fak…

support

  • All User-Agent Random
  • Chrome
  • InternetExplorer (IE)
  • Firefox
  • Safari
  • Android
  • MacOSX
  • IOS
  • Linux
  • IPhone
  • IPad
  • Computer
  • Mobile

The installation

$ go get github.com/EDDYCJY/fake-useragent
Copy the code

usage

package main

import (
	"log"

	"github.com/EDDYCJY/fake-useragent"
)

func main(a) {
	// Recommended
	random := browser.Random()
	log.Printf("Random: %s", random)

	chrome := browser.Chrome()
	log.Printf("Chrome: %s", chrome)

	internetExplorer := browser.InternetExplorer()
	log.Printf("IE: %s", internetExplorer)

	firefox := browser.Firefox()
	log.Printf("Firefox: %s", firefox)

	safari := browser.Safari()
	log.Printf("Safari: %s", safari)

	android := browser.Android()
	log.Printf("Android: %s", android)

	macOSX := browser.MacOSX()
	log.Printf("MacOSX: %s", macOSX)

	ios := browser.IOS()
	log.Printf("IOS: %s", ios)

	linux := browser.Linux()
	log.Printf("Linux: %s", linux)

	iphone := browser.IPhone()
	log.Printf("IPhone: %s", iphone)

	ipad := browser.IPad()
	log.Printf("IPad: %s", ipad)

	computer := browser.Computer()
	log.Printf("Computer: %s", computer)

	mobile := browser.Mobile()
	log.Printf("Mobile: %s", mobile)
}
Copy the code

custom

You can adjust the maximum number of pages, time interval, and maximum timeout for fetching data sources. If no value is specified, the default value is used.

client := browser.Client{
	MaxPage: 3,
	Delay: 200 * time.Millisecond,
	Timeout: 10 * time.Second,
}
cache := browser.Cache{}
b := browser.NewBrowser(client, cache)

random := b.Random()
Copy the code

Update the browser header’s temporary file cache

client := browser.Client{}
cache := browser.Cache{
	UpdateFile: true,
}
b := browser.NewBrowser(client, cache)
Copy the code

Finally, the general usage is recommended, and the default parameters are sufficient for everyday use

The output

The Random: Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_13_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 Chrome: Mozilla / 5.0 (Windows NT 10.0; Win64; X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 IE: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) Mozilla: 5.0 (Windows NT 6.3; WOW64; Rv :41.0) Gecko/20100101 Firefox/41.0 Safari: Mozilla/5.0 (iPhone; CPU iPhone OS 11_2_5 like Mac OS X) AppleWebKit/604.5.6 (KHTML, Like Gecko) Version/11.0 Mobile/15D60 Safari/604.1 Android: Mozilla/5.0 (Linux; The Android 6.0. Mya-l22 Build/ huaweimya-L22) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36 MacOSX: Mozilla / 5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14 IOS: Mozilla / 5.0 (iPhone; CPU iPhone OS 10 like Mac OS X) AppleWebKit/602.2.14 (KHTML, Like Gecko) Version/10.0 Mobile/14B72 Safari/602.1 Linux: Mozilla/5.0 (X11; Linux x86_64; Rv :42.0) Gecko/20100101 Firefox/42.0 IPhone: Mozilla/5.0 (IPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1 IPad: Mozilla / 5.0 (the device; CPU OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3 Computer: Mozilla / 5.0 (Windows NT 10.0; WOW64; Rv :54.0) Gecko/20100101 Firefox/54.0 Mobile: Mozilla/5.0 (Linux; The Android 7.0. Redmi Note 4 Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.111 Mobile Safari/537.36Copy the code

Pay attention to

If used for the first time, fake-UserAgent will collect data and create a file in the temporary directory as a file cache, so be patient for a few seconds

The last

If you find any problems in your project, feel free to submit a PR or issue. Hope you can like this project, the fundamental purpose is to solve the pain point, welcome Star! 😁

Project address: github.com/EDDYCJY/fak…