What is GoReplay

The most frightening thing is that there is a problem in the production environment. It may be that the traffic is suddenly heavy, or it may be that strange parameters are received. In the case of incomplete monitoring and log, the mentality must be broken. That’s where GoReplay comes in.

(# 'O')Copy the code

GoReplay is a network traffic (HTTP) forwarding application written in Go without hacking into code or modifying existing configurations. The configuration is simple and can be deployed using a single command line.

By listening to the network adapter, you can directly record requests for traffic playback, pressure test, and monitoring.

Realize the principle of

The Capture NetoWRk portion of the figure is implemented based on PCAP, using BPF to set the filter expression for the specified port

Tcpdump is recommended. For complex traffic, tcpCopy is recommendedCopy the code

Common use

# 1. Simple HTTP traffic replication:Gor -- input-raw :80 -- output-http "http://example.com"# 2.HTTP traffic replication frequency controlGor - input - TCP: 28020 - output - HTTP "http://example.com | 10"# 3.HTTP traffic replication diminishes:Gor - input - raw: 80 - output - TCP "replay. Local: 28020 | 10%"# 4. Logging HTTP traffic to a local file:Gor -- input-raw :80 -- output-file requests. Gor# 5.HTTP traffic playback and pressure measurement:Gor - input - the file "requests. Gor | 200%" HTTP "example.com" - the output -# 6.HTTP traffic filtering replication:Gor -- input-raw :8080 -- output-http example.com -- output-http-url-regexp ^ WWW.# 7.HTTP specified interface traffic replication:
gor --input-raw :80 --http-allow-url '/api/v1' --output-stdout   # --output-stdout means output directly to the console
Copy the code

In what context did I use it

An online program has been running smoothly for a long time, and recently Ali Cloud monitoring has been alarming bandwidth. Because the old project was not connected to Prometheus and did not know what was actually going on, GoReplay was used to record line traffic and analyze the interface.

Be sure to save the online traffic first and run the command line using root

./gor --input-raw :7018 \ Set the listening port
    --input-raw-track-response \ # save response
    --output-file logs/requests-%Y%m%d.gor \ Output files by day
    --output-file-append Split files by default, we need to merge them in one file
Copy the code

reads

// The specific request is parsed outside through the callback function
func load(filename string, iter func(i int, req *http.Request, resp *http.Response) error) error {
	file, err := os.Open(filename)
	iferr ! =nil {
		log.Printf("Cannot open text file: %s, err: [%v]", filename, err)
		return err
	}
	defer file.Close()
        
        // Set a buffer to read, with a custom bufio.splitFunc
	buf := make([]byte.0.1024*1024)
	scanner := bufio.NewScanner(file)
	scanner.Buffer(buf, 10*1024*1024)
        
        // You read that correctly, it is separated by emoji
	scanner.Split(splitByWords([]byte("๐Ÿต ๐Ÿ™ˆ ๐Ÿ™‰")))
	var req *http.Request
	var resp *http.Response
	var i = 1
	for scanner.Scan() {
		text := scanner.Bytes()
		n := bytes.IndexByte(text, '\n')
                
                // Check request or response according to the first character
                // Request is read directly using native methods
		if text[0] = ='1' {
			req, _ = http.ReadRequest(bufio.NewReader(bytes.NewReader(text[n+1:)))}else if text[0] = ='2' {
			resp, _ = http.ReadResponse(bufio.NewReader(bytes.NewReader(text[n+1:])), req)
		}
		if i%2= =0 {
			if iter(i/2, req, resp) ! =nil {
				return err
			}
		}
		i += 1
	}

	iferr := scanner.Err(); err ! =nil {
		log.Printf("Cannot scanner text file: %s, err: [%v]", filename, err)
		return err
	}

	return nil
}
Copy the code

Implement bufio SplitFunc

// copy from bufio.ScanWords
func isSpace(r rune) bool {
	if r <= '\u00FF' {
		// Obvious ASCII ones: \t through \r plus space. Plus two Latin-1 oddballs.
		switch r {
		case ' '.'\t'.'\n'.'\v'.'\f'.'\r':
			return true
		case '\u0085'.'\u00A0':
			return true
		}
		return false
	}
	// High-valued ones.
	if '\u2000' <= r && r <= '\u200a' {
		return true
	}
	switch r {
	case '\u1680'.'\u2028'.'\u2029'.'\u202f'.'\u205f'.'\u3000':
		return true
	}
	return false
}

func splitByWords(words []byte) bufio.SplitFunc {
	return func(data []byte, atEOF bool) (advance int, token []byte, err error) {
		start := 0
		for width := 0; start < len(data); start += width {
			var r rune
			r, width = utf8.DecodeRune(data[start:])
			if! isSpace(r) {break}}for width, i := 0, start; i < len(data); i += width {
			_, width = utf8.DecodeRune(data[i:])
			if bytes.HasSuffix(data[start:i], words) {
				return i + width, data[start : i-len(words)], nil}}// If we're at EOF, we have a final, non-empty, non-terminated word. Return it.
		if atEOF && len(data) > start {
			return len(data), data[start:], nil
		}
		return start, nil.nil}}Copy the code

conclusion

The whole code is in Gist

The goods received for this actual combat

  • Get a deeper understanding of HTTP traffic. Statistical analysis shows that HTTP headers actually account for a significant portion of traffic
  • For interfaces with high request frequency, the traffic can be greatly reduced by merging reporting interfaces and prolonging the return time of rotation training
  • Call some of the less commonly used interfaceshttp.ReadRequest,http.ReadResponse
  • In order to optimize the read performance, their own implementationbufio.SplitFunc