Goreplay installation and primary use of traffic replay tool

“This is the 17th day of my participation in the August More Text Challenge.

First, Goreplay introduction

Goreplay is an HTTP real-time traffic replication tool written in Golang. You can zoom in and out of traffic, limit traffic frequency, record requests to files for playback and analysis, and integrate with ElasticSearch to save traffic to ES for real-time analysis.
Instead of a proxy, GoReplay listens for traffic on the network interface, without changing the production infrastructure, and instead runs the GoReplay daemon on the same machine as the service.
Features: Easy to use
Address: github.com/buger/gorep…
Stars: 14.1 k

It has a simpler architecture than TCPCopy, with only one GOR component, as follows:

You just need to start a GOR process on the production server, which does all the work including listening, filtering, and forwarding. Its design follows the Unix design philosophy: everything is made up of pipes, and various inputs reuse data into outputs.

Two, Golang environment installation

First, install Golang and its dependent environment. Download the installation package from the official website or some Chinese websites of Go

golang.org/dl/
studygolang.com/dl

I downloaded go1.15.5.linux-amd64.tar.gz

Decompress the package to the /usr/local directory

tar -C /usr/local- ZXVF go1.14.4. Linux - amd64. Tar. GzCopy the code

3. Configure environment variables

# open
vim /etc/profile

# add
export GOROOT=/usr/local/go
export PATH=$PATH:$GOROOT/bin

# execute
source /etc/profile
Copy the code

Four, validation,

go env
Copy the code

[root@vm ~]# go versionGo version go1.15.5 Linux/amd64Copy the code

Goreplay installation

fromGithub.com/buger/gor/r…Download the latest Gor binaries (precompiled binaries for Windows, Linux X64, and Mac OS are available) or make your owncompile.

Download binary package:

[root@vm-1 ~]https://github.com/buger/goreplay/releases/download/v1.3.1/gor_1.3.1_x64.tar.gz # curl - L - O% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 626 100 626 0 0 741 - : -- : -- -- - : -- -- -- : 10.5 M - $741-100 100 10.5 M 0 0 0 3258 k 0:00:03 0:00:03 - : -- : -- 5951 k/root @ vm - 1 ~# tar XVZF gor_1. 3.1 _x64. Tar. Gz
gor
Copy the code

After unpacking the package, you can run Gor from the current directory, or you may want to copy the binaries to your PATH (for Linux and Mac OS, it can be /usr/local/bin).

Examples of Goreplay

1. Prepare RESTful API environment

Interface design:

Request type	Request path	function
Get	/person	Query Owner
Post	/person/two	The transaction test

The packers are run on both servers.

Server 1:

[root@vm-1 ~]# Java - jar spirng - the boot - demo - 0.0.1 - the SNAPSHOT. The jar____ ____ _ /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \ / _ ` | \ \ \ \ \ \ / ___) | | _) | | | | | | | (_ | |))))' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8. RELEASE) the 2021-08-17 14:19:10. 467 WARN 17537 - [the main] O.S.B oot. StartupInfoLogger: InetAddress.getLocalHost().getHostName() took 10014 milliseconds to respond. Please verify your network configuration. The 2021-08-17 14:19:20. 17537-494 the INFO [main] com. Zuozewei. SpirngbootdemoApplication: Starting SpirngbootdemoApplication v0.0.1 - the SNAPSHOT on the vm - jmeter with PID 17537 2021-08-17 14:19:20.495 INFO 17537 -- [main] (/root/ spirng-boot-demo-0.0.1-snapshot. jar started by root in /root) com.zuozewei.SpirngbootdemoApplication : The following profiles are active: A 14:19:22 2021-08-17. 17537-225 the INFO [main] S.D.R.C.R epositoryConfigurationDelegate: Bootstrapping Spring Data JPA repositories in DEFAULT Mode. 2021-08-17 14:19:22.355 INFO 17537 -- [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data Repository scanning in 111ms. Found 1 JPA Repository interfaces. 2021-08-17 14:19:23.642 INFO 17537  --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8082 (HTTP) 14:19:23 2021-08-17. 17537-667 the INFO [main] o.a pache, catalina. Core. StandardService: Starting the service [Tomcat] 2021-08-17 14:19:23. 667 INFO 17537 - [the main] org. Apache. Catalina. Core. StandardEngine: Starting Servlet engine: [Apache Tomcat/9.0.41] 2021-08-17 14:19:23.804 INFO 17537 -- [main] O.A.C.C.C. [Tomcat].[/] : Initializing Spring Embedded WebApplicationContext 2021-08-17 14:19:23.805 INFO 17537 -- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: Initialization Completed in 3179 MS 2021-08-17 14:19:24.314 INFO 17537 -- [main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: Default] 2021-08-17 14:19:24.479 INFO 17537 -- [main] org.hibernate.Version: HHH000412: Hibernate ORM Core Version 5.4.27.Final 2021-08-17 14:19:25.002 INFO 17537 -- [main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.final} 2021-08-17 14:19:25.228 INFO 17537 -- [main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... The 2021-08-17 14:19:25. 17537-635 the INFO [main] com. Zaxxer. Hikari. HikariDataSource: HikariPool - 1 - Start completed. The 2021-08-17 14:19:25. 17537-706 the INFO [main] org. Hibernate. The dialect, the dialect: HHH000400: Using dialect: Org. Hibernate. The dialect. MySQL57Dialect 14:19:27 2021-08-17. 17537-086 the INFO [main] O.H.E.T.J.P.I.J taPlatformInitiator:  HHH000490: Using JtaPlatform implementation: [org. Hibernate. Engine. The transaction. The jta. Platform. The internal. NoJtaPlatform] 14:19:27 2021-08-17. 17537-112 the INFO [main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'2021-08-17 14:19:27.990 WARN 17537 -- [main] JpaBaseConfiguration$JpaWebConfiguration: spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this Warning 14:19:28 2021-08-17. 17537-261 the INFO [main] O.S.S.C oncurrent. ThreadPoolTaskExecutor: Initializing ExecutorService 'applicationTaskExecutor'17537-2021-08-17 14:19:28. 749 INFO [main] O.S.B.W.E mbedded. Tomcat. TomcatWebServer: tomcat is started on the port (s) : 8082 (http) with context path ''17537-2021-08-17 14:19:28. 770 INFO [main] com. Zuozewei. SpirngbootdemoApplication: Started SpirngbootdemoApplication in 39.553 seconds (JVM running for 40.431)Copy the code

Server 2:

[root@vm-2 ~]# Java - jar spirng - the boot - demo - 0.0.1 - the SNAPSHOT. The jar____ ____ _ /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \ / _ ` | \ \ \ \ \ \ / ___) | | _) | | | | | | | (_ | |))))' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8. RELEASE) the 2021-08-17 14:18:34. 814 INFO 12775 - [the main] com. Zuozewei. SpirngbootdemoApplication: Starting SpirngbootdemoApplication v0.0.1 - the SNAPSHOT on the vm - hsyjy - ZNKZB vbi - 1. Novalocal with PID 12775 (/root/spirng-boot-demo-0.0.1- snapshot. jar started by root in /root) 2021-08-17 14:18:34.820 INFO 12775 -- [main] com.zuozewei.SpirngbootdemoApplication : The following profiles are active: A 14:18:36 2021-08-17. 12775-293 the INFO [main] S.D.R.C.R epositoryConfigurationDelegate: Bootstrapping Spring Data JPA repositories in DEFAULT Mode. 2021-08-17 14:18:36.419 INFO 12775 -- [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data Repository scanning in 107ms. Found 1 JPA Repository interfaces. 2021-08-17 14:18:37.694 INFO 12775  --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8082 (HTTP) 14:18:37 2021-08-17. 12775-716 the INFO [main] o.a pache, catalina. Core. StandardService: Starting the service [Tomcat] 2021-08-17 14:18:37. 717 INFO 12775 - [the main] org. Apache. Catalina. Core. StandardEngine: Starting Servlet engine: [Apache Tomcat/9.0.41] 2021-08-17 14:18:37.841 INFO 12775 -- [main] O.A.C.C.C. [Tomcat].[/] : Initializing Spring Embedded WebApplicationContext 2021-08-17 14:18:37.841 INFO 12775 -- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: Initialization Completed in 2886 MS 2021-08-17 14:18:38.305 INFO 12775 -- [main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [name: Default] 2021-08-17 14:18:38.444 INFO 12775 -- [main] org.hibernate.Version: HHH000412: Hibernate ORM Core Version 5.4.27.Final 2021-08-17 14:18:38.864 INFO 12775 -- [main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.1.2.final} 2021-08-17 14:18:39.119 INFO 12775 -- [main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting... The 2021-08-17 14:18:39. 12775-556 the INFO [main] com. Zaxxer. Hikari. HikariDataSource: HikariPool - 1 - Start completed. The 2021-08-17 14:18:39. 12775-648 the INFO [main] org. Hibernate. The dialect, the dialect: HHH000400: Using dialect: Org. Hibernate. The dialect. MySQL57Dialect 14:18:41 2021-08-17. 12775-052 the INFO [main] O.H.E.T.J.P.I.J taPlatformInitiator:  HHH000490: Using JtaPlatform implementation: [org. Hibernate. Engine. The transaction. The jta. Platform. The internal. NoJtaPlatform] 14:18:41 2021-08-17. 12775-086 the INFO [main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'2021-08-17 14:18:42.042 WARN 12775 -- [main] JpaBaseConfiguration$JpaWebConfiguration: spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this Warning 14:18:42 2021-08-17. 12775-330 the INFO [main] O.S.S.C oncurrent. ThreadPoolTaskExecutor: Initializing ExecutorService 'applicationTaskExecutor'12775-2021-08-17 14:18:42. 768 INFO [main] O.S.B.W.E mbedded. Tomcat. TomcatWebServer: tomcat is started on the port (s) : 8082 (http) with context path ''12775-2021-08-17 14:18:42. 786 INFO [main] com. Zuozewei. SpirngbootdemoApplication: Started SpirngbootdemoApplication in 8.988 seconds (JVM running for 9.939)Copy the code

2. Capture server traffic and save it to a local file

Run this command on the terminal:

Save traffic on port 8082 to a local file
sudo ./gor --input-raw :8082 --output-file=requests.gor
Copy the code

[root@vm-1 ~]# sudo ./gor --input-raw :8082 --output-file=requests.gorInterface: eth0 . BPF Filter: DST ((TCP port 8082) and (DST host 172.16.106.149 or DST host fe80: : f599:6 e65:3 dc9: b84a)) Interface: lo, BPF Filter: ((TCP DST port 8082) and (DST host 127.0.0.1 or DST host ::1)) Interface: virbr0.bpf Filter: ((TCP DST port 8082) and (DST host 192.168.124.1)) 2021/08/17 14:26:41 [PPID 17654 and PID 17655] Version:1.3.0Copy the code

Note:

Use the –output-file-append option to save captured traffic as a separate file using sudo and asking for a password: to analyze the network, Gor needs permissions that only superusers can use. However, you can configure to run Gor for non-root users.

To send a request to the server on Postman:

We view the service 1 log:

Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_
Hibernate: select next_val as id_val from hibernate_sequence for update
Hibernate: update hibernate_sequence set next_val= ? where next_val=?
Hibernate: select next_val as id_val from hibernate_sequence for update
Hibernate: update hibernate_sequence set next_val= ? where next_val=?
Hibernate: insert into person (age, name, id) values (?, ?, ?)
Hibernate: insert into person (age, name, id) values (?, ?, ?)
Copy the code

You can see that the service is processed normally.

When we stop Gor running on server 1, we can see that the recorded request file has been generated:

[root@vm-1 ~]# ls -l | grep "requests"
-rw-r----- 1 root    root         300 8月  17 14:33 requests_0.gor
Copy the code

Let’s look at the contents of the request file:

[root@vm-1 ~]# cat requests_0.gor 1 04481f92AC106afb42ce55CA 1629182282437724065 0 GET /person HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: 1 04481f92AC106afb42ce55CA 16291822824724065 0 GET /person HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: */* Cache-Control: no-cache Postman-Token: ab405d20-ef93-4ffb-b2f6-a16cf1205e7c Host: 172.16.106.149:8082 Accept-encoding: gzip, deflate, BR Connection: keep-aliveCopy the code

3. Play back the traffic from the file to server 2

Now it’s time to play back the original traffic to service 2. If there are multiple requests and they are replayed in exactly the same order and time as the records.

–output-http Provides the URL for the second service

./gor --input-file requests_0.gor --output-http="http://172.16.106.237:8082"
Copy the code

[root@vm-1 ~]./gor --input-file requests_0.gor --output-http="http://172.16.106.237:8082"
2021/08/17 14:42:55 [PPID 12356 and PID 17914] Version:1.3.0
[DEBUG][elapsed 985.792µs]: [INPUT-FILE] FileInput: end of file 'requests_0.gor'
Copy the code

We see the log for service two:

[root@vm-2 ~]# Java - jar spirng - the boot - demo - 0.0.1 - the SNAPSHOT. The jar____ ____ _ /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \ / _ ` | \ \ \ \ \ \ / ___) | | _) | | | | | | | (_ | |))))' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.3.8. RELEASE) the 2021-08-17 14:18:34. 814 INFO 12775 - [the main] com. Zuozewei. SpirngbootdemoApplication: Starting SpirngbootdemoApplication v0.0.1 - the SNAPSHOT on the vm - hsyjy - ZNKZB vbi - 1. Novalocal with PID 12775 (/root/spirng-boot-demo-0.0.1- snapshot. jar started by root in /root)...... Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_Copy the code

The service responded normally.

4. Print traffic requests to the terminal in real time

By command:

sudo ./gor --input-raw :8082 --output-stdout
Copy the code

This command logs to standard console output. Note that GoReplay does not track responses by default. You can enable them using the — output-http-track-Response option.

[root@vm-1 ~]sudo ./gor --input-raw :8082 --output-stdout Interface: eth0 . BPF Filter: DST ((TCP port 8082) and (DST host 172.16.106.149 or DST host fe80: : f599:6 e65:3 dc9: b84a)) Interface: lo, BPF Filter: ((TCP DST port 8082) and (DST host 127.0.0.1 or DST host ::1)) Interface: virbr0.bpf Filter: ((TCP DST port 8082) and (DST host 192.168.124.1)) 2021/08/17 14:55:45 [PPID 18060 and PID 18061] Version:1.3.0 1 POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: */* Cache-Control: no-cache Postman-Token: 4922ce23-5706-42c5-8a54-ade5d5aee785 Host: 172.16.106.149:8082 Accept-Encoding: gzip, deflate, BR Connection: keep-alive Content-Length: 0Copy the code

5. The replication traffic is forwarded to server 2 in real time

Run the command:

sudo ./gor --input-raw :8082 --output-http="http://172.16.106.237:8082"
Copy the code

[root@vm-1 ~]# sudo. / gor - input - raw: 8082 - output - HTTP = "http://172.16.106.237:8082"Interface: eth0 . BPF Filter: DST ((TCP port 8082) and (DST host 172.16.106.149 or DST host fe80: : f599:6 e65:3 dc9: b84a)) Interface: lo, BPF Filter: ((TCP DST port 8082) and (DST host 127.0.0.1 or DST host ::1)) Interface: virbr0.bpf Filter: ((TCP DST port 8082) and (DST host 192.168.124.1)) 2021/08/17 14:59:49 [PPID 18118 and PID 18119] Version:1.3.0Copy the code

Check the log on server 2 and find that traffic has entered:

Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_
Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_
Hibernate: select person0_.id as id1_0_, person0_.age as age2_0_, person0_.name as name3_0_ from person person0_ where person0_.age=?
Copy the code

6. Stress tests

Goreplay supports reducing or amplifying replay of captured production actual request traffic for stress testing in test environments. Pressure testing is generally for Input flow reduction or amplification.

Recorded request file:

[root@vm-1 ~]# cat requests.gor POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: 1 040f1f92AC106afb7e8d5679 1629183084044026882 0 POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: */* Cache-Control: no-cache Postman-Token: a8600dd1-28c1-4825-b6a2-68c42659942f Host: 172.16.106.149:8082 Accept-Encoding: gzip, deflate, BR Connection: keep-alive Content-Length: 0  [root@vm-jmeter ~]# cat requests.gor POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: 1 040f1f92AC106afb7e8d5679 1629183084044026882 0 POST /person/two HTTP/1.1 user-agent: PostmanRuntime/7.28.2 Accept: */* Cache-Control: no-cache Postman-Token: a8600dd1-28c1-4825-b6a2-68c42659942f Host: 172.16.106.149:8082 Accept-Encoding: gzip, deflate, BR Connection: keep-alive Content-Length: 0Copy the code

We can see that there are two main requests.

Run the following command to play back the traffic from the file to server two and double the size:

[root@vm-1 ~]./gor --input-file "requests.gor|200%" --output-http="http://172.16.106.237:8082"
2021/08/17 15:03:58 [PPID 12356 and PID 18187] Version:1.3.0
[DEBUG][elapsed 1.361742ms]: [INPUT-FILE] FileInput: end of file 'requests.gor'
Copy the code

Let’s take a look at the processing of service 2 logs:

Hibernate: select next_val as id_val from hibernate_sequence for update
Hibernate: update hibernate_sequence set next_val= ? where next_val=?
Hibernate: select next_val as id_val from hibernate_sequence for update
Hibernate: update hibernate_sequence set next_val= ? where next_val=?
Hibernate: insert into person (age, name, id) values (?, ?, ?)
Hibernate: insert into person (age, name, id) values (?, ?, ?)
Copy the code

We can see that the service processed four requests.

Note, of course, support request traffic 10%,20%, etc.

Five, the summary

Today is a brief introduction to the common gameplay of Gor. In fact, there is much more than the above mentioned gameplay.

File source:

Github.com/zuozewei/bl…