How do I use Java to read large text files line by line? Recommended | Java Debug Notes

This article is participating in the Java Theme Month – Java Debug Notes Event, see the event link for details

How do I use Java to read large text files line by line?

I need to read a large text file of about 5-6 GB line by line using Java.

How do I do this quickly?

Answer:

$\color{purple} a lot of knowledge points, really need to write out just can master!! {~}$

A common pattern is to use

try (BufferedReader br = new BufferedReader(new FileReader(file))) { String line; while ((line = br.readLine()) ! = null) { // process the line. } }Copy the code

If you assume no character encoding, you can read the data faster. Ascii-7, for example, is not much different. Your processing of the data will most likely take longer.

Edit: a less commonly used pattern to avoid line leaks in scope.

try(BufferedReader br = new BufferedReader(new FileReader(file))) { for(String line; (line = br.readLine()) ! = null; ) { // process the line. } // line is not visible here. }Copy the code

Update: In Java 8, you can do this

try (Stream<String> stream = Files.lines(Paths.get(fileName))) {
        stream.forEach(System.out::println);
}
Copy the code

Note: You must place the Stream in the try-with-resource block to ensure that the #close method is called on it; otherwise, the underlying file handle will never be closed until the GC executes at a later time.

Answer:

Check out this blog:

Java Reads files line by line -Java tutorial

The buffer size can be specified or the default size can be used. The default is large enough for most purposes.

// Open the file FileInputStream fstream = new FileInputStream("textfile.txt"); BufferedReader br = new BufferedReader(new InputStreamReader(fstream)); String strLine; //Read File Line By Line while ((strLine = br.readLine()) ! = null) { // Print the content on the console System.out.println (strLine); } //Close the input stream fstream.close();Copy the code

The article translated from am2dgbqfb6mk75jcyanzabc67y ac4c6men2g7xr2a – stackoverflow – com. Translate. Goog/questions / 5…

I use a FileChannel for the transfer, referring to my previous translation

Juejin. Cn/post / 696248…

I looked at the mmap + FileChannel test and found it to be even better.

As you can see, The Performance of FileChannel is relatively high.

Briefly, take rocketMQ, which is a popular file system for storing data, producing and consuming data as directly manipulated files, It involves page caching, FileChannel, and FileChannel reading a page of 4KB at a time, thanks to ByteBuffer buffers and MMAP memory mapping

RokcetMQ has also been tuned for better performance

Preallocate MappedFile mlock System call file preheat sequential read and sequential writeCopy the code

I’m just doing a piece here, everybody come on!

$\color{red} Welcome to my column StackOverFlow, I will filter the quality of the interview test!! {~}$

$\color{red} has the latest, elegant implementation, and I will also write my opinion on this question at the end of the article {~}$

Thank you for reading this, if this article is well written and if you feel there is something to it

Ask for a thumbs up 👍 ask for attention ❤️ ask for share 👥 for 8 abs I really very useful!!

If there are any mistakes in this blog, please comment, thank you very much! ❤ ️ ❤ ️ ❤ ️ ❤ ️