This is the 12th day of my participation in the August More Text Challenge. For details, see: August More Text Challenge

A, through theURLThe download file

To download the file through the URL, the main steps are as follows:

  1. Open theURLConnect, connect the input stream
  2. The download file

Methods are:

  1. Java IOway
  2. Java NIOway
  3. HttpClientway
  4. Apache Commons IOway
  5. Resumable breakpoint download

(1)Java IOway

The most basic API: Leverage Java IO.

The code is as follows:

@Test
public void test(a) {

    String FILE_NAME = "";
    String FILE_URL = "";

    try (BufferedInputStream in = new BufferedInputStream(new URL(FILE_URL).openStream());
         FileOutputStream fileOutputStream = new FileOutputStream(FILE_NAME)
        ) {
        byte[] dataBuffer = new byte[1024];
        int bytesRead;
        while ((bytesRead = in.read(dataBuffer, 0.1024)) != -1) {
            fileOutputStream.write(dataBuffer, 0, bytesRead); }}catch (IOException e) {
        // handle exception}}Copy the code

99.1 MB download, 90,564 ms, 90s.

Byte [] dataBuffer = new byte[1024]; Instead of BufferedInputStream.


(2)Java NIOway

Instead of caching data in application memory, Java NIO opens two channels and allows data to be transferred between them.

  • This is more efficient than reading streams from the buffer pool

  • Direct buffer from the file system to the specified file, do not have to cache a copy in the application memory

  • On Linux and UNIX systems, zero-copy is used to reduce switching from kernel mode to user mode

The code is as follows:

@Test
public void test(a) {
    String FILE_NAME = "";
    String FILE_URL = "";

    ReadableByteChannel readableByteChannel = 
        Channels.newChannel(new URL(FILE_URL).openStream());

    FileOutputStream fileOutputStream = new FileOutputStream(FILE_NAME);

    fileOutputStream.getChannel()
        .transferFrom(readableByteChannel, 0, Long.MAX_VALUE);
}
Copy the code

To download 99.1 MB, it took 84,359 ms, or 84s.


(3)HttpClientway

AsyncHttpClient is an asynchronous HTTP request that uses Netty at the bottom.

The code is as follows:

@Test
public void test(a) {        
    AsyncHttpClient client = Dsl.asyncHttpClient();

    FileOutputStream stream = new FileOutputStream(FILE_NAME);

    client.prepareGet(FILE_URL).execute(new AsyncCompletionHandler<FileOutputStream>() {

        @Override
        public State onBodyPartReceived(HttpResponseBodyPart bodyPart)
            throws Exception {
            // Write this data directly to the specified file using FileChannel
            stream.getChannel().write(bodyPart.getBodyByteBuffer());
            return State.CONTINUE;
        }

        @Override
        public FileOutputStream onCompleted(Response response) {
            System.out.println(System.currentTimeMillis() - now);
            returnstream; }}); }Copy the code


(4)Apache Commons IOway

This is the method in the Apache Commons package and is the most commonly used method in your work.

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>
Copy the code

Longest used:IOUtils.copy(input, output).

Something to keep in mind here: file stream closing

  • Can matchJava 7:Try-with-resourcesThe resource is automatically shut down
  • This method is the same as method 1, which also creates a buffer pool at the bottom, and the pool size is 4098

Fileutils.copyurltofile () : The underlying call also goes ioutils.copy (input, output).

The code is as follows:

@Test
public void test(a) {  

    FileUtils.copyURLToFile(
        new URL(FILE_URL),
        new File(FILE_NAME));
}
Copy the code

Added timeout mechanism:

URLConnection connection = source.openConnection();
connection.setConnectTimeout(connectionTimeout);
connection.setReadTimeout(readTimeout);
Copy the code


(5) resumable breakpoint download

The network connection may fail, so it is useful to recover breakpoint downloads.

@Test
public void test(a) {
    File outputFile = new File(FILE_NAME);

        URLConnection downloadFileConnection = 
            addFileResumeFunctionality(FILE_URL, outputFile);
        long size = transferDataAndGetBytesDownloaded(downloadFileConnection, outputFile);
        System.out.println("size : " + size);
}

private URLConnection addFileResumeFunctionality(String downloadUrl, File outputFile)
    throws IOException, URISyntaxException {
    long existingFileSize = 0L;
    URLConnection downloadFileConnection = new URI(downloadUrl).toURL()
        .openConnection();

    if (outputFile.exists() && downloadFileConnection instanceof HttpURLConnection) {
        HttpURLConnection httpFileConnection = (HttpURLConnection) downloadFileConnection;

        HttpURLConnection tmpFileConn = (HttpURLConnection) downloadFileConnection;
        tmpFileConn.setRequestMethod("HEAD");
        long fileLength = tmpFileConn.getContentLengthLong();
        existingFileSize = outputFile.length();

        if (existingFileSize < fileLength) {
            httpFileConnection.setRequestProperty("Range"."bytes=" + existingFileSize + "-" + fileLength);
        } else {
            throw new IOException("File Download already completed."); }}return downloadFileConnection;
}

private long transferDataAndGetBytesDownloaded(URLConnection downloadFileConnection, File outputFile)
    throws IOException {

    long bytesDownloaded = 0;
    try (InputStream is = downloadFileConnection.getInputStream();
         OutputStream os = new FileOutputStream(outputFile, true)) {byte[] buffer = new byte[1024];

        int bytesCount;
        while ((bytesCount = is.read(buffer)) > 0) {
            os.write(buffer, 0, bytesCount); bytesDownloaded += bytesCount; }}return bytesDownloaded;
}
Copy the code

Output:

Java.io.IOException: File Download already completed. at com.donaldy.file.DownloadFileTest.addFileResumeFunctionality(DownloadFileTest.java:116) at com.donaldy.file.DownloadFileTest.tesResumableDownload(DownloadFileTest.java:94)Copy the code