Gopher refers to north

In this article, we’ll briefly review the HTTPS handshake process and explain what JA3 fingerprints are and how to customize your own JA3 fingerprint with Go, based on reader questions.

The outline of this article is as follows, please follow Lao Xu’s ideas to gradually build their own exclusive JA3 fingerprint.

Review the HTTPS handshake process

Before we begin to understand what JA3 fingerprints are in earnest, let’s review the HTTPS handshake process to help with the rest of this article.

In the code of more than 2000 lines of code is to explain TLS handshake process this article mainly analyzes HTTPS one-way authentication and two-way authentication process (TLS1.3).

In one-way authentication, the client does not need a certificate, but only needs to verify that the server certificate is valid. The handshake process and exchanged MSGS are as follows.

In bidirectional authentication, both the server and client need to verify the validity of each other’s certificates. The handshake process and exchanged MSGS are as follows.

Comparison between one-way authentication and two-way authentication:

1. In one-way authentication and two-way authentication, the total data is sent and received only three times. The data sent at a time contains one or more messages

2. ClientHelloMsg and serverHelloMsg are not encrypted, and subsequent messages are encrypted

3. The Client and Server compute the key twice respectively, after reading the HelloMsg and finishedMsg of the other party

4. Compared to two-way authentication and one-way authentication, the service side more certificateRequestMsgTLS13 messages were sent

5. Compared with one-way authentication and two-way authentication, the client sends more certificateMsgTLS13 and certificateVerifyMsg messages

Regardless of one-way authentication or two-way authentication, the Server obtains basic information about the Client only when the Client actively informs the Server. The key information is TLS version supported by the client, cipherSuites supported by the client, signature algorithm supported by the client, key exchange protocol supported by the client, and its corresponding public key. This information is included in clientHelloMsg, which is key to generating JA3 fingerprints, and clientHelloMsg and serverHelloMsg are not encrypted. The lack of encryption meant it was easier to modify, which made it possible to customize our own JA3 fingerprint.

If you are interested in learning more about the HTTPS handshake process, read the following article:

More than 2000 lines of code were written to clarify the TLS handshake process

More than 2000 lines of code to explain the TLS handshake process (continued)

What are JA3 fingerprints

So what is JA3 fingerprint? JA3 is an online fingerprint identification method for TLS clients. JA3 is an online fingerprint identification method for TLS clients.

This method is used to collect the decimal byte values of the following fields in the clientHelloMsg packet: TLS Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats. It then concatenates the values, using “, “to separate the fields, and” – “to separate the values in the fields. Finally, the MD5 hash of these strings is computed, resulting in a 32-character fingerprint that is easy to use and share.

In order to further describe the source of these data, Lao Xu made field mapping by combining the packet capture diagram in John Althouse’s article with clientHelloMsg structure in Go source code.

Careful students may have noticed that JA3 fingerprint has a total of 5 data fields according to the previous description, while the figure above only maps 4. That is because TLS extension field is more, old xu not a collation. Although I did not list them all, Lao Xu prepared a unit test, through which students who are interested in in-depth research can conduct debugging analysis.

https://github.com/Isites/go-coder/blob/master/http2/tls/handsh/msg_test.go

Copy the code

JA3 Fingerprint usage

As described above, a JA3 fingerprint is an MD5 string. Think back to the use of MD5 in normal development.

• Determine if the content is consistent
• As a unique identifier

Md5 is not secure, but the main reason JA3 chose MD5 as a hash is for better backward compatibility

Apparently, JA3 fingerprints have similar uses. For a simple example, if an attacker builds an executable file, the JA3 fingerprint of that file is likely to be unique. As a result, we can identify some malware using JA3 fingerprints.

At the end of this section, Xu recommends a site that posts a list of malicious JA3 fingerprints.

https://sslbl.abuse.ch/ja3-fingerprints/
Copy the code

Build your own JA3 fingerprint

Unique fingerprint of HTTP1.1

As mentioned earlier, clientHelloMsg and serverHelloMsg are not encrypted, which makes it possible to customize your own JA3 fingerprint, and there is a library on Github (github.com/refraction-…). ClientHelloMsg can be modified to some extent. Let’s build our own JA3 fingerprint from this library.

/ / key to import
import (
    xtls "github.com/refraction-networking/utls"
    "crypto/tls"
)

// Clone a Transport
tr := http.DefaultTransport.(*http.Transport).Clone()
// Custom DialTLSContext function, which is used to create TCP connection and TLS handshake

tr.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
	dialer := net.Dialer{}
	// Create a TCP connection
	con, err := dialer.DialContext(ctx, network, addr)
	iferr ! =nil {
		return nil, err
	}
	// Get the host information based on the address
	host, _, err := net.SplitHostPort(addr)
	iferr ! =nil {
		return nil, err
	}
	/ / build tlsconf
	xtlsConf := &xtls.Config{
		ServerName:    host,
		Renegotiation: xtls.RenegotiateNever,
	}
	/ / build the TLS. UConn
	xtlsConn := xtls.UClient(con, xtlsConf, xtls.HelloCustom)
	clientHelloSpec := &xtls.ClientHelloSpec{
	    // The maximum and minimum TLS versions in hellomsg
		TLSVersMax: tls.VersionTLS12,
		TLSVersMin: tls.VersionTLS10,
		// CipherSuites for JA3 fingerprinting
		CipherSuites: []uint16{
			tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
			// tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
			tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
			tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
			tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,
			tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
			tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
			// tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
			tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
			tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
			tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
			tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,
		},
		CompressionMethods: []byte{
			0,},// The Extensions required for JA3 fingerprints
		Extensions: []xtls.TLSExtension{
			&xtls.RenegotiationInfoExtension{Renegotiation: xtls.RenegotiateOnceAsClient},
			&xtls.SNIExtension{ServerName: host},
			&xtls.UtlsExtendedMasterSecretExtension{},
			&xtls.SignatureAlgorithmsExtension{SupportedSignatureAlgorithms: []xtls.SignatureScheme{
				xtls.ECDSAWithP256AndSHA256,
				xtls.PSSWithSHA256,
				xtls.PKCS1WithSHA256,
				xtls.ECDSAWithP384AndSHA384,
				xtls.ECDSAWithSHA1,
				xtls.PSSWithSHA384,
				xtls.PSSWithSHA384,
				xtls.PKCS1WithSHA384,
				xtls.PSSWithSHA512,
				xtls.PKCS1WithSHA512,
				xtls.PKCS1WithSHA1}},
			&xtls.StatusRequestExtension{},
			&xtls.NPNExtension{},
			&xtls.SCTExtension{},
			&xtls.ALPNExtension{AlpnProtocols: []string{"h2"."HTTP / 1.1"}},
			// The Elliptic Curve Formats needed for JA3 fingerprints
			&xtls.SupportedPointsExtension{SupportedPoints: []byte{1}}, // uncompressed
			// Elliptic Curves required for JA3 fingerprints
			&xtls.SupportedCurvesExtension{
				Curves: []xtls.CurveID{
					xtls.X25519,
					xtls.CurveP256,
					xtls.CurveP384,
					xtls.CurveP521,
				},
			},
		},
	}
	// Define hellomsg encryption suite and other information
	err = xtlsConn.ApplyPreset(clientHelloSpec)
	iferr ! =nil {
		return nil, err
	}
	/ / TLS handshake
	err = xtlsConn.Handshake()
	iferr ! =nil {
		return nil, err
	}
	fmt.Println("Current request protocol:", xtlsConn.HandshakeState.ServerHello.AlpnProtocol)
	return xtlsConn, err
}

Copy the code

The above code can be summarized in three steps.

1. Creating a TCP Connection

2. Information needed to build clientHelloMsg

3. Complete TLS handshake

With the above code, we get our JA3 fingerprint by asking https://ja3er.com/json.

c := http.Client{
	Transport: tr,
}
resp, err := c.Get("https://ja3er.com/json")
iferr ! =nil {
	fmt.Println(err)
	return
}
bts, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
fmt.Println(string(bts), err)
Copy the code

The resulting JA3 fingerprint is shown below.

Now that we have our first JA3 fingerprint, we have tweaked the code to get our own. For example, if we add the number 2333 to the CipherSuites list, we get the following result.

Finally, the JA3 fingerprint has changed again and can be called its own fingerprint. I don’t need to tell you from the headlines that the problem is not over. HTTP: / / HTTP: / / HTTP: / / HTTP: / / HTTP: / / HTTP: / / HTTP: / / http2

For those of you who have seen the Go process for initiating an HTTP2.0 request, the http2 connection needs to send PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n string when it is set up. Obviously, the related flow is missing after the DialTLSContext function is customized. At this point, how do we build our own fingerprint for HTTP2?

Http2’s proprietary fingerprint

DialTLSContext does not support frames, multiplexing, and other features of HTTP2. Therefore, we need to build our own connection to support the various features of HTTP2.

Next, we go to golang.org/x/net/http2 to complete the HTTP2 request after the custom TLS handshake process.

// Manually dial to get a connection with TLS handshake completed
con, err := tr.DialTLSContext(context.Background(), "tcp"."dss0.bdstatic.com:443")
iferr ! =nil {
	fmt.Println("DialTLSContext", err)
	return
}

// Build an HTTP2 connection
tr2 := http2.Transport{}
// This step is very important
h2Con, err := tr2.NewClientConn(con)
iferr ! =nil {
	fmt.Println("NewClientConn", err)
	return
}

req, _ := http.NewRequest("GET"."https://dss0.bdstatic.com/5aV1bjqh_Q23odCf/static/superman/img/topnav/newzhidao-da1cf444b0.png".nil)
// Make a request to a link that supports HTTP2 and read the request status
resp2, err := h2Con.RoundTrip(req)
iferr ! =nil {
	fmt.Println("RoundTrip", err)
	return
}
io.CopyN(io.Discard, resp2.Body, 2<<10)
resp2.Body.Close()
fmt.Println("Response code:", resp2.StatusCode)

Copy the code

The results are as follows.

As you can see, after customizing the JA3 fingerprint, the http2 request can be read as well. At this point, you are done building your own JA3 fingerprint in http2-enabled requests. (The information to generate the JA3 fingerprint is in clientHelloMsg, and this section is done just to ensure that everything from the request to the response is working properly.)

As an additional note, there are significant limitations to manually completing http2 requests with NewClientConn. For example, you need to manage the life cycle of your connection, you can’t reconnect automatically, and so on. Of course, this is all for the future, and when it does come to this, developers may need to fork the NET package from the GO source to maintain it themselves.

Write in the last

Xu wrote this article not only to introduce you to JA3, but also to deepen your understanding of HTTP’s underbelly through your own practice.

Finally, I sincerely hope that this article can be of some help to all readers.

Note:

When writing this article, the author used the GO version as go1.17.7

Full example used in the article: github.com/Isites/go-c…