How do I trigger and process HTTP requests using java.net.URLConnection? Recommended | Java Debug Notes

This article is participating in the Java Theme Month – Java Debug Notes Event, see the event link for details

Question: How do I use java.net.URLConnection to trigger and process HTTP requests?

Java.net.URLConnection often asks about usage here, and the Oracle tutorial is too concise.

This tutorial basically just shows how to trigger a GET request and read the response. It doesn’t explain anywhere how to use it to perform POST requests, set request headers, read response headers, process cookies, submit HTML forms, upload files, etc.

So how do you use it java.net.URLConnection to trigger and process “advanced” HTTP requests?

Answer:

$\color{purple} a lot of knowledge points, really need to write out just can master!! {~}$

We need to know at least the URL and character set first. Parameters are optional, depending on the functional requirements.

String url = "http://example.com";
String charset = "UTF-8";  // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
String param1 = "value1";
String param2 = "value2";
// ...

String query = String.format("param1=%s&param2=%s", 
     URLEncoder.encode(param1, charset), 
     URLEncoder.encode(param2, charset));
Copy the code

The query parameter name=value format must be, and is concatenated with &. In general, you can also urL-encode query parameters with the specified character set URLEncoder#encode().

The String#format() is just for convenience. I prefer it when I need to + the String concatenation operator more than twice. Trigger an HTTP GET request with (optional) query parameters

This is a trivial task. This is the default request method.

URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
// ...
Copy the code

Should any query string be concatenated to the URL? . The accept-Charset header may indicate what encoding parameters are in the server. If you don’t send any query strings, then you can leave the accept-charset header. If you don’t need to set any titles, you can even use the URL#openStream() shortcut.

InputStream response = new URL(url).openStream();
// ...
Copy the code

Either way, if an HttpServlet is on the other side, doGet() calls its methods, and the arguments pass through the available HttpServletRequest#getParameter().

For testing purposes, you can print the response body to STdout as follows:

try (Scanner scanner = new Scanner(response)) {
    String responseBody = scanner.useDelimiter("\\A").next();
    System.out.println(responseBody);
}
Copy the code

Trigger an HTTP POST request with query parameters

Setting urlConnect #setDoOutput() will true implicitly set the request method to POST. Like Web forms, the standard HTTP POST type is Application/X-www-form-urlencoded to write query strings to the request body.

URLConnection connection = new URL(url).openConnection(); connection.setDoOutput(true); // Triggers POST. connection.setRequestProperty("Accept-Charset", charset); connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=" + charset); try (OutputStream output = connection.getOutputStream()) { output.write(query.getBytes(charset)); } InputStream response = connection.getInputStream(); / /...Copy the code

Note: Whenever you submit an HTML form programmatically, don’t forget to put any pairs of name=value elements in the query string, and of course any pairs of elements that you want to programmatically “press” name=value (because this is usually used on the server side to distinguish whether or not the button was pressed, and if so, which button.

You can also cast URLConnection to HttpURLConnection and use it HttpURLConnection#setRequestMethod(). However, if you try to use the connection for output, you still need to set urlConnect #setDoOutput() to true.

HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("POST");
// ...
Copy the code

Either way, if an HttpServlet is on the other side, doPost() calls its methods and the parameters pass through the available HttpServletRequest#getParameter(). Actually triggers the HTTP request

You can explicitly trigger the HTTP request URlConnect #connect(), but when you want to get any information about the HTTP response (such as the response body used, etc.), the request will automatically trigger the URlConnect #getInputStream() on demand. The above example does just that, so the connect() call is actually redundant. Collect HTTP response information

HTTP response status:

You need HttpURLConnection here. If necessary, project first.

int status = httpConnection.getResponseCode(); HTTP response header: for (Entry<String, List<String>> header: connection.getHeaderFields().entrySet()) { System.out.println(header.getKey() + "=" + header.getValue()); }Copy the code

HTTP response encoding:

When the Content-Type contains the charset parameter, the response body may be text-based, so we want to use the character encoding specified on the server side to process the response body.

String contentType = connection.getHeaderField("Content-Type"); String charset = null; for (String param : contentType.replace(" ", "").split(";" )) { if (param.startsWith("charset=")) { charset = param.split("=", 2)[1]; break; } } if (charset ! = null) { try (BufferedReader reader = new BufferedReader(new InputStreamReader(response, charset))) { for (String line; (line = reader.readLine()) ! = null;) {/ /... System.out.println(line) ? } } } else { // It's likely binary content, use InputStream/OutputStream. }Copy the code

Maintain meeting cookies

Server-side sessions are typically supported by cookies. Some Web forms require you to be logged in and/or tracked by a session. You can use the CookieHandlerAPI to maintain cookies. You need to prepare a CookieManager with CookiePolicy ACCEPT_ALL before sending all HTTP requests.

// First set the default cookie manager.
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));

// All the following subsequent URLConnections will use the same cookie manager.
URLConnection connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...

connection = new URL(url).openConnection();
// ...
Copy the code

Note that this does not always work in all cases. If that fails for you, it is best to collect and set the Cookie header manually. Basically, you need set-cookies to GET all the headers from the response of the login or the first GET request and pass them on to subsequent requests.

// Gather all cookies on the first request. URLConnection connection = new URL(url).openConnection(); List<String> cookies = connection.getHeaderFields().get("Set-Cookie"); / /... // Then use the same cookies on all subsequent requests. connection = new URL(url).openConnection(); for (String cookie : cookies) { connection.addRequestProperty("Cookie", cookie.split(";" , 2) [0]); } / /...Copy the code

The split (“;” , 2)[0] there are server-side attributes that get rid of them that are irrelevant such as cookie Expires, PATH, etc. Alternatively, you can use cookie.substring(0, cookie.indexOf(‘; ‘) instead of split(). Streaming media model

HttpURLConnection By default, Will buffers the entire request body before actually sending the request, No matter whether you use to set the Length of the Content of the fixed connection. The setRequestProperty (” the Content – Length “, contentLength); . OutOfMemoryException This can cause whenever you simultaneously send a large POST request (for example, upload a file). To avoid this, you want to set

HttpURLConnection# setFixedLengthStreamingMode (). httpConnection.setFixedLengthStreamingMode(contentLength);Copy the code

However, if you really don’t know the content length, you can use the chunked streaming mode by setting httpurLConnect #setChunkedStreamingMode(). This sets the httpTransfer-Encoding header to chunked to force the request body to be sent as a block. The following example sends the body in 1KB blocks.

httpConnection.setChunkedStreamingMode(1024);
Copy the code

The user agent

The request may return an unexpected response that would work perfectly in a real Web browser. The server may be blocking requests based on the User-Agent request header. URLConnection by default, Will sets it to Java/1.6.0_19 as the last part obviously the location of the JRE version. You can override it as follows:

Connection. The setRequestProperty (" the user-agent ", "Mozilla / 5.0 (Windows NT 6.1) AppleWebKit / 537.36 (KHTML, Like Gecko) Chrome / 41.0.2228.0 Safari / 537.36 "); // Do as if you're using Chrome 41 on Windows 7.Copy the code

Use the user-Agent string from the most recent browser. Error handling

If the HTTP response code is 4nn (client error) or 5nn (server error), you might want to read, HttpURLConnection#getErrorStream() to see if the server sent any useful error information.

InputStream error = ((HttpURLConnection) connection).getErrorStream();
Copy the code

If the HTTP response code is -1, connection and response processing errors occur. This HttpURLConnection implementation is in older JRE and some bugs keep the connection alive. You may need to turn it off by setting the http.keepalivesystem property to false. You can do this programmatically at the beginning of an application by:

System.setProperty("http.keepAlive", "false");
Copy the code

Uploads and downloads

Typically, multipart/form-data uses encoding for mixed POST content (binary and character data). The encoding is described in more detail in RFC2388.

String param = "value"; File textFile = new File("/path/to/file.txt"); File binaryFile = new File("/path/to/file.bin"); String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value. String CRLF = "\r\n"; // Line separator required by multipart/form-data. URLConnection connection = new URL(url).openConnection(); connection.setDoOutput(true); connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary); try ( OutputStream output = connection.getOutputStream(); PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true); ) { // Send normal param. writer.append("--" + boundary).append(CRLF); writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF); writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); writer.append(CRLF).append(param).append(CRLF).flush(); // Send text file. writer.append("--" + boundary).append(CRLF); writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF); writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset! writer.append(CRLF).flush(); Files.copy(textFile.toPath(), output); output.flush(); // Important before continuing with writer! writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary. // Send binary file. writer.append("--" + boundary).append(CRLF); writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF); writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF); writer.append("Content-Transfer-Encoding: binary").append(CRLF); writer.append(CRLF).flush(); Files.copy(binaryFile.toPath(), output); output.flush(); // Important before continuing with writer! writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary. // End of multipart/form-data. writer.append("--" + boundary + "--").append(CRLF).flush();  }Copy the code

If the other side is HttpServlet, doPost() will call its method, and it will go through HttpServletRequest#getPart() (note, therefore not getParameter(), and so on!). To provide parts. GetPart () However, this method is relatively new, having been introduced in Servlet 3.0 (Glassfish 3, Tomcat 7, etc.). Prior to Servlet 3.0, your best bet was to use Apache Commons FileUpload to resolve multipart/form-data requests. See also this answer for examples of FileUpload and Servelt 3.0 methods. Handles untrusted or misconfigured HTTPS sites

If you are developing for Android rather than Java, beware: The following workaround may save you time if you do not deploy the correct certificates during development. But you should not use it for production. These days (April 2021), if Google detects unsafe hostname validation program, will not allow you to distribute your application in the Play store, please refer to the support.google.com/faqs/answer…

Sometimes you need to connect to an HTTPS URL, probably because you’re writing a Web crawler. In this case, you may encounter javax.net.ssl.SSLException: Not trusted server certificate in some HTTPS site did Not keep its SSL certificate on the latest status, Java security. Cert. CertificateException: No subject the alternative DNS name matching [hostname] found or javax.net.ssl.SSLProtocolException: handshake alert: Recognized_name faces or on some misconfigured HTTPS site.

The following one-time run initializers in the staticWeb crawler class should make HTTPS sites like HttpsURLConnection more lenient and therefore not throw these exceptions.

static { TrustManager[] trustAllCertificates = new TrustManager[] { new X509TrustManager() { @Override public X509Certificate[] getAcceptedIssuers() { return null; // Not relevant. } @Override public void checkClientTrusted(X509Certificate[] certs, String authType) { // Do nothing. Just allow them all. } @Override public void checkServerTrusted(X509Certificate[] certs, String authType) { // Do nothing. Just allow them all. } } }; HostnameVerifier trustAllHostnames = new HostnameVerifier() { @Override public boolean verify(String hostname, SSLSession session) { return true; // Just allow them all. } }; try { System.setProperty("jsse.enableSNIExtension", "false"); SSLContext sc = SSLContext.getInstance("SSL"); sc.init(null, trustAllCertificates, new SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames); } catch (GeneralSecurityException e) { throw new ExceptionInInitializerError(e); }}Copy the code

The last word

The HttpComponents in Apache’s HttpClient are much more convenient for all this 🙂

HttpClient tutorial HttpClient exampleCopy the code

The article translated from am2dgbqfb6mk75jcyanzabc67y ac4c6men2g7xr2a – stackoverflow – com. Translate. Goog/questions / 2…

Author’s advice: Good answer! If we are Java programmers, we can use Spring’s RestTemplate to implement JSON, FORM, file upload and download HTTP request, cookie, session, SSL

$Color {red} let’s talk about the principle:!! {~}$

• RestTemplate has built a factory ClientHttpRequestFactory request interface, the default implementation is java.net.HttpURLConnection, HttpURLConnection default need to set some parameters of the spring has helped us to set up in SimpleClientHttpRequestFactory such as overtime, the output stream and so on

  • The HttpURLConnection object cannot be constructed directly and needs to be obtained through the openConnection() method of the URL class
  • The Connect () function of HttpURLConnection actually establishes a TCP connection to the server without actually sending an HTTP request. The HTTP request is actually not officially sent until we get the server response data (such as calling getInputStream(), getResponseCode(), and so on)
  • HttpURLConnection is based on HTTP protocol, and its underlying implementation is through socket communication. If you do not set timeout, the program may freeze in case of a network exception. Be sure to set 100%
  • The HTTP body is written to the OutputStream stream. The data written to the stream is not immediately sent to the network, but stored in the memory buffer. When the stream is closed, the HTTP body is generated based on the written content
  • When the getInputStream() method is called, it returns an input stream from which to read the server’s return for the HTTP request.
  • HttpURLConnection. The connect () is not a must. When we need to return, such as we use HttpURLConnection. GetInputStream () method when it will automatically send the request, so no need to call the connect () method

• The OKHTTP implementation can also be implemented through the ClientHttpRequestFactory of RestTemplate, requiring an import package. Efficient, bandwidth saving, HTTP2, SPDY, connection pooling

• Of course you can also use Apache httpClient is also good.

The authors recommend RestTemplate+OKHttp

$\color{red} Welcome to my column StackOverFlow, I will filter the quality of the interview test!! {~}$

$\color{red} has the latest, elegant implementation, and I will also write my opinion on this question at the end of the article {~}$

Thank you for reading this, if this article is well written and if you feel there is something to it

Ask for a thumbs up 👍 ask for attention ❤️ ask for share 👥 for 8 abs I really very useful!!

If there are any mistakes in this blog, please comment, thank you very much! ❤ ️ ❤ ️ ❤ ️ ❤ ️