This is the second day of my participation in Gwen Challenge

HttpGo tool (Golang package HttpClient)

Encapsulating HttpClient simplifies development and is easy to use, and other projects or tools like crawlers rely on the library

Dependent libraries

import . "github.com/NoBugBoy/httpgo/http"
Copy the code

Method of use

req := &Req{}
	body, err := req.Url("https://www.baidu.com").
		Method(http.MethodGet). // Request mode
		Header("user-agent"."Mozilla / 5.0..."). / / request header
		Header("content-type"."application/json"). // Multiple headers can be set
		Timeout(3). // Request timeout
		Retry(3). // Number of requests for error retries
		Chunk(). // Start Chunk will not automatically close Response IO. You need to manually read response body data and close IO. Refer to Test5 for block transmission
		Params(Query{ // Request parameters are common to all requests. If the get parameter carries? If id=1, the URL parameter is preferred
			"id": 1,
		}).
		ProxyUrl("192.168.1.1:8080"). // Set the proxy IP address to be used
		ImportProxy(). // Import the proxy IP from the configuration file and use it randomly
		Proxy(). // Enable proxy mode
		Build(). // Create request, usually do not need to call separately, use method refer to Test1 stress test
		Go(). // Initiate a request
		Body() // Gets the return value string
	iferr ! =nil {
		panic(err)
	}
	fmt.Println(body)
	request := req.Request // Keep the *http.Request object as needed
	fmt.Println(request)
	response := req.Response // Reserve the *http.Response object in case you need it
	fmt.Println(response)
	transport := req.TransportSetting() // Run Transport to adjust parameters
	fmt.Println(transport)

Copy the code

A handy stress test is to call the Build() method to create the request object ahead of time, then turn on a bunch of coroutines to initiate the request

var join sync.WaitGroup

Test1 stress test, pay attention to ulimit and maxfd tuning /**
func Test1(a) {
	arr := make([]*Req, 0)
	for i := 0; i < 1000; i++ {
		join.Add(1)
		req := &Req{}
		x := req.Url("http://localhost:8080/get/1").
			Method(http.MethodGet).
			Header("Connection"."Keep-Alive").
			Header("Content-Type"."application/json").
			Timeout(30).
			Build()
		arr = append(arr, x)
	}
	for _, req := range arr {
	    // Initiate a stress test request
		go runAndPrint(req)
	}
	join.Wait()
}

// Concurrent requests
func runAndPrint(r *Req) {
	defer join.Done()
	r.Go()
	//fmt.Println(.Body())
}
Copy the code

Chunk() mode use, you can control how to use Response, need to manually control IO to close oh, otherwise there will be overflow problem

req := &Req{}
	re := req.ImportProxy().
		Method(http.MethodGet).
		Header("Connection"."Keep-Alive").
		Header("Transfer-Encoding"."chunked").
		Url("http://localhost:8080/get").
		Chunk().
		Timeout(30). // The timeout will be closed
		Go()
	fmt.Println(re.Response.Header)
	data := make([]byte.1024)
	for {
		read, err := re.Response.Body.Read(data)
		fmt.Println("Byte length", read)
		if read > 0 {
			fmt.Print(string(data[:read]))
		}
		if err == io.EOF {
			break
		}
	}
	fmt.Println("Ok")
Copy the code

Req objects can be reused. The Go method will reset request parameters, URLS, headers, etc., without multiple instantiations

The above source code and tests are in github repository github.com/NoBugBoy/ht… Welcome to star for issues