The background,
Http has become the most widely used application layer protocol on the Internet. With the continuous evolution of application forms, the security of traditional Http begins to face challenges. The main security problems of Http are as follows: 1. 2. The identity of the communicating party cannot be securely verified. 3. Communication information may be tampered with.
Therefore, Https combined with SSL/TLS protocol is widely used. Https is now the industry’s de facto standard for the secure Http protocol.
When terminals such as the Web front-end, Android, or iOS clients communicate with the server through Https, the communication process and principles are the same, and the problems to be solved are similar. This article mainly through combing Https communication process, and summarizes its practical application in Android, deepen the understanding of Https.
2. Https communication process
Https involves a lot of knowledge, and the process of migrating from Http to Https can be complicated if it is a complete old project. This article is more from the perspective of the client to sort out the knowledge system closely related to Https. Among them, encryption algorithm, hash algorithm, digital certificate, etc., are very important technical points.
2.1 Symmetric Encryption
Symmetric encryption, for encryption and decryption algorithm, encryption and decryption key is the same, just like the same lock, encryption key and decryption key are the same key. After encrypting with the key, the encrypted ciphertext can be obtained, which can be directly decrypted to obtain the original text.
For both sides of communication, the communication content can be easily encrypted through symmetric encryption, and then transmitted safely, which can effectively prevent the information from being obtained by the middleman in the middle. Even if the message can be intercepted, it is safe as long as the key is not leaked.
Symmetric encryption is widely used in information transmission due to its features of security, convenience and efficiency.
2.2 Asymmetric encryption
The biggest problem of symmetric encryption is how to safely inform the other party of the symmetric key. In a client and multiple server communication at the same time, we can a server and multiple clients at the same time communication of the realistic environment, the relationship is many-to-many, and symmetric key can only be by the current communication held by the client and server, therefore, in the actual process of communication, symmetric key negotiation become difficult in reality.
Unlike symmetric encryption, asymmetric encryption has both public and private keys. If the public key is used for encryption, only the corresponding private key can be used for decryption. If the private key is used for encryption, only the corresponding public key can be used for decryption. In asymmetric encryption system, the convention and the private key are a pair of treasures, without either party is not possible. The private key is usually reserved by the main object, while the public key can be directly exposed to the outside world.
The existence of public and private yue makes asymmetric encryption have obvious advantages over symmetric encryption. 1, public key directly open to the world and the outside any subject through the public key encryption, can be held directly to the main body of the private key is safe to send encrypted cryptograph, private key subject to get the ciphertext, held by their private key corresponding decryption, obtaining cleartext, to complete the security of the information transmission process, and does not need to face the security of the symmetric key negotiation problems of symmetric encryption.
2. The private key subject encrypts the information through the private key, and any external subject decrypts the information through the public key. If the decryption can be successful, the identity of the corresponding communication partner can be reversed, which is equivalent to completing the identity verification. Therefore, asymmetric encryption has good authentication capability.
Everything has advantages and disadvantages, asymmetric encryption compared to symmetric encryption, because its own function is more powerful, in the algorithm design of the corresponding is also more complex, in the encryption and decryption efficiency, far lower than symmetric encryption.
2.3 Hash algorithm
Hash algorithm, also known as digest algorithm or hash algorithm, can compress any data object into a data digest. For the same hash algorithm, the compressed data digest has a specific length and format, so as to form the “fingerprint” of data. Any small change to the original data object can make a big difference to the new “digital fingerprint”.
The hash algorithm is often used to determine whether two data objects are the same, because the digital fingerprint formed by the hash algorithm must be the same for the same data objects. However, the original data objects corresponding to the same “digital fingerprint” may not be the same, because there may be hash conflicts.
Hashing algorithm is widely used in real life. For example, MD5 is the most commonly used algorithm to determine whether the contents of two file objects are the same.
Strictly speaking, hash algorithm and encryption algorithm (symmetric encryption or asymmetric encryption) is fundamentally different, encryption algorithm, corresponding can be decrypted, the purpose is to carry out data encryption after the safe storage or transmission, can be through the key to get the original, is a reversible process. And hash algorithm, in essence, the category of “digital fingerprint”, through the hash algorithm to form the “digital signature”, directly in the algorithm level can not get the original text, is irreversible. Of course, so-called decryption in the form of rainbow tables or data dictionaries is essentially a brute force process.
2.4 Https Communication Process
Https = Http + SSL/TLS = Http + content encryption + identity authentication + data integrity protection. Currently, TLS is mainly used in version 1.2. According to the RFC official website, Https communication is as follows:
On the whole, Https communication process includes two stages: secret key negotiation and encryption communication.
2.4.1 Key Negotiation
Key negotiation is an important part of Https communication. The secret key here refers to the symmetric secret key negotiated by the client and the server for information encryption. The negotiation process is as follows: 1. The client sends a ClientHello message to the server, which contains a random number RNc generated by the client, and information about the client protocol version and encryption suite. 2. The server receives the message and sends ServerHello message to the client, including the random number RNs generated by the server, the server protocol version information, encryption suite and other information. 3. The server sends the certificate containing the server’s public key to the client. 4. The server requests the certificate containing the client’s public key from the client. 5. The client verifies the server certificate and sends the client certificate containing the client public key to the server. 6. The server verifies the client certificate. 7. The client generates a random number PMS, encrypts the PMS through the server public key, and sends the PMS to the server. 8, the client and the server through the same algorithm, as well as RNc, RNs, PMS, respectively generate symmetric encryption key, because the algorithm and parameters are the same, therefore, the symmetric key generated at both ends is also the same.
In this way, the negotiation process of symmetric secret keys is completed.
2.4.2 Encrypted Communication
After the key negotiation is complete, the SSL handshake is complete. The subsequent communication process is encrypted and transmitted based on the negotiated symmetric secret key. This process is no different from the general sense of encrypted communication process.
The ultimate goal of secret key negotiation is to generate symmetric secret keys for subsequent encrypted communication while securing the client and server. This is the essence of Https security.
2.5 CA and Digital Certificates
In the process of key negotiation, how to exchange asymmetrically encrypted public keys between client and server becomes the focus. Once the public key of the other party is securely known, it can be encrypted through the public key of the other party to further complete the subsequent negotiation of symmetric secret keys. When obtaining the public key of the other party, how to verify the identity of the other party to ensure that the public key is the real public key of the other party?
The public key cannot be directly transmitted over the network. Otherwise, information may be stolen or tampered with, which may lead to man-in-the-middle attacks. Therefore, it is a big difficulty to have a mechanism to verify the identity of the other party and the accuracy of the other party’s public key while obtaining the other party’s public key.
Directly through the Internet, it will always be a chicken-and-egg problem, security can never be guaranteed. Therefore, through the CA-based way, issuing digital certificates, to the identity of the communication party “guarantee”, to provide verification of data integrity, is designed.
By trusting the CA, you can trust the digital certificate issued by the CA, trust the communication party, and obtain the public key of the other party.Copy the codeCorrespondingly, there are two problems: 1. The identity of CA itself, how to trust? 2. After the CA is trusted, how to verify the digital certificate to confirm the identity of the communication partner and obtain the corresponding public key?
In practice, trust in CA identity provides two approaches. One is direct system-based integration, which integrates the digital certificates issued by the major CAS themselves into the system. Such as Mac computer systems or Android phones. The other is based on manual credit, such as download the CA digital certificate of the third party through the browser, and set the trust to it. Either way, the system trusts the integrated CA digital certificate and obtains the corresponding CA public key.
Next, the client verifies the server certificate as an example to explain the complete identity verification and public key acquisition process.
The administrator applies for a digital certificate from a CA, including the server public key, domain name, certificate validity period, organization, region, certificate serial number, and issuing authority. These information of text, through the hash algorithm H, the formation of information, the CA to generate information summary, through CA own asymmetric encryption private key is encrypted, get the ciphertext, this cipher is called digital signature (CA to the signed information), information of text, digital signatures, put together, form a digital certificate.
A digital certificate contains two contents: 1. Message body 2. Digital signature.Copy the codeAfter the server sends the certificate to the client, the client first obtains the CA information of the certificate and compares it with the CA that has been trusted in the system. If the same CA exists, the client decrypts it using the CA’s public key. Otherwise, the CA certificate on the server is authenticated using the certificate chain and decrypted using the CA public key. Otherwise, the server certificate verification fails.
After the CA public key decrypts the digital signature on the digital certificate, the CA public key can obtain the message digest. Then, the CA public key performs the same hashing algorithm on the message body to obtain the message digest. Then, the CA public key compares the two message digest to verify the data integrity. At the same time, the server identity and the server public key on the digital certificate are confirmed.
3. Https packet capture principle and Charles practice
3.1 Man-in-the-middle Attack
Https packet capture is essentially based on man-in-the-middle attack. Install the packet capture software, enable the packet capture service, and configure the IP address and port number of the proxy server on the client (mobile phone or browser). In this case, Http or Https requests from the client are forwarded through the packet capture service. For the Http or Https communication server, the packet capture service acts as the client, and for the source client, the packet capture service acts as the server. Therefore, the packet capture service forms a de facto proxy and functions as both a forward proxy and a reverse proxy.
The communication passes through the packet capture service. However, for Https requests, packets can be captured but the packet subject is garbled because the packet body is encrypted. In this case, the packet capture service cannot transparently decrypt the packet body or tamper with the packet. In this case, the main body of the packet plays the role of an intermediate proxy forwarding communication. At this point, the client still has a valid verification process for the identity of the communication server.
When using the packet capture software, the client is required to install a certificate and set the trust. This certificate is the corresponding certificate of the packet capture service. After the client trusts this certificate, it completely trusts the packet capture service. At this point, the packet capture service has the functions of communication transfer, distribution, information transparency and tampering in a complete sense, and achieves the effect of man-in-the-middle attack in a complete sense. The source client and the source server are no longer actually talking to each other, they just think they’re still talking to each other.
After trusted the certificate of the packet capture service, the source client actually communicates with the packet capture service and verifies the identity and information integrity of the server based on the public key of the packet capture service. Source client request, caught the service will be completely transparent, caught service can tamper with this request, and into the source client role, and the source server communication, because caught after consultation services with the source server’s public key and symmetric with secret key, therefore, the server returned as a result, the caught service is transparent, caught the service can be tampered with, And returns to the source client. This is also a lot of packet capture tools, not only has a transparent display of communication content, but also has additional debugging functions, such as through the request parameters, or return results, etc., modify, to achieve the desired debugging purpose.
The mainstream Http/Https packet grabber has Fiddler, Charles and so on. The most commonly used Mac is Charles. The following is a summary of Charles’s practice.
3.2 Charles crack
The current official version of Charles is a paid software with a 30-day free trial period. Even after the trial period, users who have not paid for Charles can continue to use the software, but the use time cannot exceed 30 minutes, and there is a 10-second delay when starting up. The actual use process is a little inconvenient. It’s probably working. You need to reboot it.
There are a lot of materials cracked by Charles on the Internet, and this is just a summary. Please support the legal version when you usually use it.
The latest Charles version is V4.2.8.
3.2.1 Charles Cracking Tool
Charles online cracking tool: www.zzzmode.com/mytools/cha… The corresponding version of the hack instructions and there is also a history making project address: blog.zzzmode.com/2017/05/16/… Github.com/8enet/Charl…
RegisterName Enter your favorite name, then select the version of Charles installed on your machine, and click “Generate” to download the generated charles.jar file.
Overwrite the corresponding Charles installation file on this machine. / Applications/Charles. App/Contents/Java/Charles. Jar restart, Charles, findings suggest “Registered to: how”, and has been officially activated.
3.2.2 Charles cracking principle
Charles’ judgment on cracking or not is written in the Charles. jar file. Find the confused corresponding class file, force the logic corresponding to cracking judgment to the registered logic by modifying the bytecode, and then regenerate the Charles. jar file.
3.2.3 Beyond Compare.classfile
Compare the two Charles.jar files using Beyond Compare to see exactly where the differences are.
Beyond Compare does not support file comparisons in. Class format by default. Therefore, by default, if you compare directly, the only difference can be found in the file: qhtb.class, but the contents of the file are unrecognizable and displayed as garbled characters.
The default file formats supported by Beyond Compare can be accessed via Beyond Compare >> file formats… Look at it. By default. Class files are not supported. We can add support for. Class files.
Website: www.scootersoftware.com/download.ph… Other file formats are provided, we can search and download the corresponding configuration through tools >> import… To support such file files. But the. Class file format is only supported on Windows, not Mac.
However, we can provide additional formatting support by converting configuration file formats to external programs. The support for.class files in the Windows platform is as follows:
Java class to source8-Feb-2019 *.class
- Compares Java class files decompiled to source.
- - Uses Java decompiler, Jad: http://varaneckas.com/jad
Copy the codeTherefore, we can also compare.class files directly by configuring jad directly.
A, http://varaneckas.com/jad corresponding jad version, decompression placed into the system after the suitable directory, such as/Users/corn/Research/jad158g /. B, Beyond Compare >> > > new format file, the General configuration *. In class, just check in on external programs, and the corresponding external configuration program command: / Users/corn/Research/jad158g/jad – > % t p % s.
qHTb.class
Format >> class file
We discovered that Charles. Jar simply changed the DdNM() method in qhtb.class bytecode and returned true. The other method is gbef(), which is the method to get the registration name.
Therefore, the corresponding logic is changed to registered by forcibly modifying the bytecode to complete the cracking.
3.3 Charles pay attention to packet capture
Charles packet capture, there are actually many articles on the Internet, all of which have special description. Here we mainly summarize the points that need to be paid attention to in the packet capture process: 1. The source client needs to install the corresponding certificate of Charels and trust it. For details, see Help >> SSL Proxying. 2. Proxy Setting and SSL Proxy Setting should also be configured, which is often forgotten. If you want all packages, you can configure * directly. Note that if the system Proxy mode is set after Proxy >> MacOS Proxy is selected, the use of other software in the system, such as Android Studio, may be affected. Because at this point, AS is also likely to go to the corresponding proxy (because there may be caching). External Proxy Settting can also be set to External Proxy Settting. The form goes external to other agents. 3. Start Recording enables packet capture. You can use various debugging techniques such as delay, breakpoint, rewrite, Map Remote/Local during packet capture. Breakpoints are not always valid because, in practice, discovery tends to be delayed. Rewrite and Map are useful in practice, such as interwriting with servers. 4. In some cases, the packet capture service can be disabled on the source client, for example, no proxy is configured. In this case, Charles and other packet capture services become invalid.
Iv. Android anti-packet capture principle and summary
In Android development, it is very common to analyze the communication information of competing apps by capturing packets. But it is often found that some apps can not catch the bag. To ensure the security of the App, anti-packet capture is necessary for the installation packages released to the public.
The essence of packet capture is man-in-the-middle attack. According to the principle of man-in-the-middle attack, in the key link, the condition that the packet capture tool becomes a complete man-in-the-middle can be broken, and the anti-packet capture protection of Android App can be realized.
1. The source client ignores proxy Settings. In Android projects, requests can be directly connected by ignoring proxy Settings by determining whether a proxy is currently in use. The general rule for deciding whether to use a proxy is as follows:
/** * check if it is in the proxy environment ** @return
*/
private boolean isUsingProxy() {// Whether the value is greater than or equal to 14 Final Boolean isIceOrUpper = build.version.sdk_int >= build.version_codes.ice_cream_sandwich; String proxyAddress; int proxyPort;if (isIceOrUpper) {
proxyAddress = System.getProperty("http.proxyHost");
String portStr = System.getProperty("http.proxyPort"); proxyPort = Integer.parseInt((portStr ! = null ? portStr :"1"));
} else {
proxyAddress = Proxy.getHost(BaseApplication.context);
proxyPort = Proxy.getPort(BaseApplication.context);
}
return(! TextUtils.isEmpty(proxyAddress)) && (proxyPort ! = 1); }Copy the codeOkHttp directly provides an interface to ignore proxy Settings. If you find that an agent is in use, you can set the agent to be ignored.
OkHttpClient.Builder builder = new OkHttpClient.Builder(); . builder.proxy(Proxy.NO_PROXY); .Copy the code2. The source client device improves the certificate trust level. Starting with Android 7.0, the certificates in network security configuration are changed from “system level certificate” and “user level certificate” to “system level certificate”. This means that user-level certificates will not be trusted by default, even if the user himself has been set to trust them. As a result, we’ll find that, by default, on Android 7.0 and above, you can’t catch bags. The packet capture agent displays the following information:
The source client displays the following information:
Caused by: java.security.cert.CertificateException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
at com.android.org.conscrypt.TrustManagerImpl.verifyChain(TrustManagerImpl.java:661)
at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:539)
at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:605)
at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:495)
at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:418)
at com.android.org.conscrypt.TrustManagerImpl.getTrustedChainForServer(TrustManagerImpl.java:339)
at android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted(NetworkSecurityTrustManager.java:94)
at android.security.net.config.RootTrustManager.checkServerTrusted(RootTrustManager.java:88)
at com.android.org.conscrypt.Platform.checkServerTrusted(Platform.java:197)
at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:399)
at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
at com.android.org.conscrypt.SslWrapper.doHandshake(SslWrapper.java:374)
at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:217)
....
Copy the codeAt the same time, Android provides network security configuration entry for developers. If you need to capture packets, you only need to add user-level certificate to the corresponding configuration item.
<? xml version="1.0" encoding="utf-8"? > <network-security-config> <base-config> <trust-anchors> <certificates src="user" />
<certificates src="system" />
</trust-anchors>
</base-config>
</network-security-config>
Copy the code
The official document also provides solutions to network security configuration requirements in different scenarios. For example, it provides special configuration when the Android :debuggable is set to true. However, in actual App development, this switch is always set to false. The solution is very simple. You can configure network-security-config separately for the buildTypes, productFlavors, and even variants of the App. If
3. The source client strengthens the trust verification on the server. The source client can perform trust verification on the server in multiple ways based on specific requirements and server certificate configuration modes. For example, the source client has a built-in server certificate, or performs security verification on the service domain name, or further verification on the certificate name. If a certificate information item that does not meet expectations is found, the request is terminated.
The following is a simple demonstration of the judgment of the server information. Once it is the packet capture service information, the request is terminated.
public class ServerCertificateSecurityVerifier implements HostnameVerifier {
....
private static final String[] DEFAULT_INSECURITY_CA_ISSUER_KEYWORDS = new String[]{
// fiddler
"fiddler.com",
// fiddler
"fiddler2.com",
// charles
"charlesproxy.com",
// whistle
"wproxy.org",
// Android Packet Capture
"Packet Capture CA Certificate",
// mitmproxy
"mitmproxy",
// debug proxy
"Debug Proxy"}; @override public Boolean verify(String hostname, SSLSession session) {verify(String hostname, SSLSession session) {verify(String hostname, SSLSession session) {verify(String hostname, SSLSession session) {verify(String hostname, SSLSession session) { Go to the default flow try {for (X509Certificate x509Certificate : session.getPeerCertificateChain()) {
if (x509Certificate.getIssuerDN() == null || StringUtil.isEmpty(x509Certificate.getIssuerDN().getName())) {
continue;
}
String caIssuerCompleteName = x509Certificate.getIssuerDN().getName();
for(String insecurityCAIssuerKey: DEFAULT_INSECURITY_CA_ISSUER_KEYWORDS) {// The certificate information contains the certificate information of the packet capture toolif (Pattern.compile(Pattern.quote(insecurityCAIssuerKey), Pattern.CASE_INSENSITIVE)
.matcher(caIssuerCompleteName).find()) {
Log.w(, TAG, "ServerCertificate error... -" + caIssuerCompleteName);
return false;
}
}
}
} catch (Exception e) {
Log.e(TAG, e);
}
return verifyByOkHttpDefault(hostname, session);
}
Copy the code
Five, the conclusion
The security of web requests is becoming more and more important, and Google and Apple are explicitly demanding that all apps on their marketplaces comply with Https security standards. The terms of the Google Play channel’s listing also clearly stipulate that further security treatment is required for possible man-in-the-middle attacks in logic, otherwise the listing cannot succeed. By default, the system only supports system-level certificates. Https security requirements and standards will also be improved. During the development of Android App, there are many practical scenarios related to security issues in the process of communication with the server. Based on Https, further encryption processing or security verification is needed to improve the security protection level of App as much as possible.
Reference site: tools.ietf.org/html/rfc524… Developer.android.com/training/ar… Codelabs.developers.google.com/codelabs/an… Mp.weixin.qq.com/s/3M0CqFQP2… zh.wikipedia.org/wiki/ Transport layer security… zh.wikipedia.org/wiki/ Hypertext transfer Ann…