IO flow and system
IO technology is an extremely complex module in the JDK. One of the key reasons for its complexity is the correlation between IO operations and the system kernel. In addition, network programming and file management rely on IO technology, and they are difficult to program.
Linux space Isolation
Linux usage is user-specific. This is basic knowledge, and the underlying layer also distinguishes between user and kernel modules:
- User space: indicates the User space
- Kernel space: indicates the Kernel space
The permissions of common user space are much weaker than the permissions of kernel space operation, which involves the interaction between the user and the two modules of kernel. In this case, if the application deployed on the service needs to request system resources, the interaction will be more complicated:
The user space itself cannot directly issue scheduling instructions to the system, but must pass the kernel. Data operations in the kernel must be copied to the user space first. This isolation mechanism can effectively protect the security and stability of the system.
Parameter to see
You can run the Top command to dynamically view data analysis and resource usage of processes.
us: Percentage of CPU occupied by user space;sy: Percentage of CPU occupied by kernel space;id: Percentage of CPU occupied by idle processes;wa: CPU usage of I/O waiting.
Wa index is one of the core items of monitoring in large-scale document task flow.
IO Collaboration Process
Now look at the flow in Figure [1] above. When the application end initiates the IO operation request, the request flows along each node on the link, and there are two core concepts:
- Node interaction modes: synchronous and asynchronous;
- IO data manipulation: blocking and non-blocking;
Synchronous/asynchronous IO, blocking/non-blocking IO, see the details below.
Ii. IO model analysis
1. Synchronous blocking
The way the user thread interacts with the kernel, where the application request is processed by one thread and the Accept and read methods block until the entire action is complete:
In the common CS architecture, this is the basic process of an I/O operation. In the scenario of high concurrency, the client’s request response will have serious performance problems and occupy too many resources.
2. Synchronization is non-blocking
Optimized on the basis of synchronous blocking IO, the current thread does not wait for data to be ready until replication is complete:
The thread returns immediately after the request and polls until it gets the data before stopping the polling. The disadvantage of this pattern is also obvious. If the data is ready, the thread is told to complete the subsequent action, which saves a lot of intermediate interaction.
3. Asynchronous notification mode
In asynchronous mode, the blocking mechanism is completely abandoned and processes are interacted in segments, which is similar to the common third-party interconnection mode. When a local service requests a third-party service, if the request process takes a long time, it is executed asynchronously, and the third-party service calls back for the first time to confirm that the request can be executed. The second callback is to push the processing result. This idea can greatly improve performance and save resources when dealing with complex problems:
Asynchronous mode for performance improvement is huge, of course, its corresponding processing mechanism is also more complex, program iteration and optimization is endless, in NIO mode again IO flow mode optimization.
File class
1. Basic description
As an abstract representation of the path names of files and directories, the File class is used to obtain metadata information about disk files, such as File name, size, modification time, and permission judgment.
Note: File does not operate on the data content of the File. The content of the File is called data, and the information about the File itself is called metadata.
public class File01 {
public static void main(String[] args) throws Exception {
// 1. Read the specified file
File speFile = new File(IoParam.BASE_PATH+"fileio-03.text");if(! speFile.exists()){boolean creFlag = speFile.createNewFile() ;
System.out.println("Create:"+speFile.getName()+"; Result:"+creFlag);
}
// 2, read the specified position
File dirFile = new File(IoParam.BASE_PATH) ;
// Check whether it is a directory
boolean dirFlag = dirFile.isDirectory() ;
if (dirFlag){
File[] dirFiles = dirFile.listFiles() ;
printFileArr(dirFiles);
}
// delete the specified file
if (speFile.exists()){
boolean delFlag = speFile.delete() ;
System.out.println("Delete:"+speFile.getName()+"; Result:"+delFlag); }}private static void printFileArr (File[] fileArr){
if(fileArr ! =null && fileArr.length>0) {for(File file : fileArr) { printFileInfo(file) ; }}}private static void printFileInfo (File file) {
System.out.println("Name:"+file.getName());
System.out.println("Length:"+file.length());
System.out.println("Path:"+file.getPath());
System.out.println("Document judgment:"+file.isFile());
System.out.println("Directory judgment:"+file.isDirectory());
System.out.println("Last modified:"+newDate(file.lastModified())); System.out.println(); }}Copy the codeThe above case uses the File class in the basic structure and common methods (read, judge, create, delete), JDK source in the continuous update iteration, through the constructor of the class, method, annotation to judge the basic function of the class, is the necessary ability as a developer.
There are two key information descriptions missing in the File class: type and encoding. If you often develop the requirements of the File module, you know that these are two extremely complex points, which are prone to problems. Here is how to deal with them from the perspective of actual development.
2. File business scenario
As shown in the figure, in a normal file flow task, there are three basic forms of conversion: file, stream and data:
Basic process description:
- Source file generation, push file center;
- Notify the service to use the node to obtain the file;
- Business nodes for logical processing;
An obvious problem is that no node can be adapted to all file processing strategies, such as type and encoding. Faced with problems in complex scenarios, rule constraints are commonly used to solve problems, that is, to deal with things within the convention rules.
In the above flow, the data subject description when the source file node informs the business node:
public class BizFile {
/** * File task batch number */
private String taskId ;
/** * whether to compress */
private Boolean zipFlag ;
/** * file address */
private String fileUrl ;
/** * File type */
private String fileType ;
/** * File encoding */
private String fileCode ;
/** * Service association: database */
private String bizDataBase ;
/** * Business association: data table */
private String bizTableName ;
}
Copy the codeEncapsulate the whole process as a task, namely: task batches, file information, business library table routing, etc. Of course, these information can also be directly marked in the file naming policy, and the processing method is similar:
/** * Reads information based on the convention policy */
public class File02 {
public static void main(String[] args) {
BizFile bizFile = new BizFile("IN001",Boolean.FALSE, IoParam.BASE_PATH,
"csv"."utf8"."model"."score");
bizFileInfo(bizFile) ;
/* * Service check */
File file = new File(bizFile.getFileUrl());
if(! file.getName().endsWith(bizFile.getFileType())){ System.out.println(file.getName()+": wrong description..."); }}private static void bizFileInfo (BizFile bizFile){
logInfo("The task ID",bizFile.getTaskId());
logInfo("Decompress or not",bizFile.getZipFlag());
logInfo("File Address",bizFile.getFileUrl());
logInfo("File Type",bizFile.getFileType());
logInfo("File code",bizFile.getFileCode());
logInfo("Business base",bizFile.getBizDataBase());
logInfo("Business sheet",bizFile.getBizTableName()); }}Copy the codeThe information based on the body description can also be translated into the following naming rules: naming policy: number _ compression _Excel_ code _ library _ table. In this way, files that do not comply with the convention are directly excluded during service processing, reducing data problems caused by file exceptions.
4. Basic flow mode
1. Overall overview
IO flow
Basic encoding logic: Source file -> Input stream -> Logical processing -> Output stream -> target file;
Depending on the perspective, flows can be divided into many modes:
- Flow direction: input flow, output flow;
- Stream data types: byte stream, character stream;
IO flow mode has many kinds, the corresponding API design is also very complex, usually complex API to grasp the core interface and common implementation classes and principles.
Based on the API
-
Byte stream: InputStream input, OutputStream output; The basic unit of data transmission is the byte;
- Read () : the next byte in the input stream to read data;
- Read (byte b[]) : read data cached into a byte array;
- Write (int b) : specifies bytes to write to the output stream;
- Write (byte b[]) : array bytes write to the output stream;
-
Character streams: Reader reads, Writer writes. The basic unit of data transmission is the character;
- Read () : reads a single character;
- Read (char cbuf[]) : reads the character array;
- Write (int c) : write a specified character;
- Write (char cbuf[]) : write an array of characters;
Buffer mode
The normal read/write mode of the IO stream is that the data is read and written out, and the other buffer mode is that the data is loaded into the buffer array, and the buffer should be filled again when the data is read:
Buffering mode has obvious advantages. It ensures efficient reading and writing and performs in isolation from the data filling process. BufferedInputStream, BufferedReader class is the concrete implementation of buffering logic.
2. Byte stream
API diagram:
Byte stream base API:
public class IoByte01 {
public static void main(String[] args) throws Exception {
// Source file destination file
File source = new File(IoParam.BASE_PATH+"fileio-01.png"); File target =new File(IoParam.BASE_PATH+"copy-"+source.getName()) ;
// Input stream output stream
InputStream inStream = new FileInputStream(source) ;
OutputStream outStream = new FileOutputStream(target) ;
// read and write
byte[] byteArr = new byte[1024];
int readSign ;
while((readSign=inStream.read(byteArr)) ! = -1){
outStream.write(byteArr);
}
// Close the input and output streamsoutStream.close(); inStream.close(); }}Copy the codeByte stream buffer API:
public class IoByte02 {
public static void main(String[] args) throws Exception {
// Source file destination file
File source = new File(IoParam.BASE_PATH+"fileio-02.png"); File target =new File(IoParam.BASE_PATH+"backup-"+source.getName()) ;
// Buffering: input stream output stream
InputStream bufInStream = new BufferedInputStream(new FileInputStream(source));
OutputStream bufOutStream = new BufferedOutputStream(new FileOutputStream(target));
// read and write
int readSign ;
while((readSign=bufInStream.read()) ! = -1){
bufOutStream.write(readSign);
}
// Close the input and output streamsbufOutStream.close(); bufInStream.close(); }}Copy the codeByte stream application scenario: data is the file itself, such as pictures, video, audio, etc.
3. Character stream
API diagram:
Character stream basic API:
public class IoChar01 {
public static void main(String[] args) throws Exception {
// Read text write text
File readerFile = new File(IoParam.BASE_PATH+"io-text.txt"); File writerFile =new File(IoParam.BASE_PATH+"copy-"+readerFile.getName()) ;
// character input/output stream
Reader reader = new FileReader(readerFile) ;
Writer writer = new FileWriter(writerFile) ;
// characters read in and write out
int readSign ;
while((readSign = reader.read()) ! = -1){
writer.write(readSign);
}
writer.flush();
/ / close the flowwriter.close(); reader.close(); }}Copy the codeCharacter stream buffering API:
public class IoChar02 {
public static void main(String[] args) throws Exception {
// Read text write text
File readerFile = new File(IoParam.BASE_PATH+"io-text.txt"); File writerFile =new File(IoParam.BASE_PATH+"line-"+readerFile.getName()) ;
// Buffer character input and output streams
BufferedReader bufReader = new BufferedReader(new FileReader(readerFile)) ;
BufferedWriter bufWriter = new BufferedWriter(new FileWriter(writerFile)) ;
// characters read in and write out
String line;
while((line = bufReader.readLine()) ! =null){
bufWriter.write(line);
bufWriter.newLine();
}
bufWriter.flush();
/ / close the flowbufWriter.close(); bufReader.close(); }}Copy the codeCharacter stream application scenarios: Files are used as data carriers, such as Excel, CSV, and TXT files.
4. Coding and decoding
- Encoding: character to byte;
- Decoding: converting bytes into characters;
public class EnDeCode {
public static void main(String[] args) throws Exception {
String var = "IO stream" ;
/ / code
byte[] enVar = var.getBytes(StandardCharsets.UTF_8) ;
for (byte encode:enVar){
System.out.println(encode);
}
/ / decoding
String deVar = new String(enVar,StandardCharsets.UTF_8) ;
System.out.println(deVar);
/ / the code
String messyVar = newString(enVar,StandardCharsets.ISO_8859_1) ; System.out.println(messyVar); }}Copy the code
The root cause of the occurrence of garbled code is the different coding types used in the two stages of encoding and decoding.
5. Serialization
- Serialization: The process of converting an object into a stream;
- Deserialization: The process of converting a stream into an object;
public class SerEntity implements Serializable {
private Integer id ;
private String name ;
}
public class Seriali01 {
public static void main(String[] args) throws Exception {
// serialize objects
OutputStream outStream = new FileOutputStream("SerEntity.txt"); ObjectOutputStream objOutStream =new ObjectOutputStream(outStream);
objOutStream.writeObject(new SerEntity(1."Cicada"));
objOutStream.close();
// Deserialize the object
InputStream inStream = new FileInputStream("SerEntity.txt");
ObjectInputStream objInStream = newObjectInputStream(inStream) ; SerEntity serEntity = (SerEntity) objInStream.readObject(); System.out.println(serEntity); inStream.close(); }}Copy the codeNote: Member objects of reference types must also be serializable, otherwise NotSerializableException will be thrown.
NIO mode
1. Basic concepts
NIO (NonBlockingIO), a data block-oriented processing mechanism, synchronous non-blocking model, a single thread on the server can process multiple client requests, the processing speed of IO stream has a very high improvement, three core components:
- Buffer: The underlying maintenance array stores data;
- Channel: supports read and write bidirectional operations.
- Selector: provides Channel multiple registration and polling capabilities;
API Use Cases
public class IoNew01 {
public static void main(String[] args) throws Exception {
// Source file destination file
File source = new File(IoParam.BASE_PATH+"fileio-02.png"); File target =new File(IoParam.BASE_PATH+"channel-"+source.getName()) ;
// Enter the byte channel
FileInputStream inStream = new FileInputStream(source);
FileChannel inChannel = inStream.getChannel();
// Outputs the byte channel
FileOutputStream outStream = new FileOutputStream(target);
FileChannel outChannel = outStream.getChannel();
// Direct channel replication
// outChannel.transferFrom(inChannel, 0, inChannel.size());
// Buffer read/write mechanism
ByteBuffer buffer = ByteBuffer.allocateDirect(1024);
while (true) {
// Read the data in the channel to the buffer
int in = inChannel.read(buffer);
if (in == -1) {
break;
}
// Read/write switch
buffer.flip();
// Write out buffer data
outChannel.write(buffer);
// Clear the bufferbuffer.clear(); } outChannel.close(); inChannel.close(); }}Copy the codeThe above cases are only the most basic NIO file replication capability, and NIO mode has a very broad space to play in network communication.
2. Network communication
A single thread on the server can process multiple client requests and poll the multiplexer to see if there are ANY I/O requests. In this way, the concurrency capability of the server is greatly improved and the resource consumption is significantly reduced.
API example: server-side emulation
public class SecServer {
public static void main(String[] args) {
try {
// Start the service
ServerSocketChannel socketChannel = ServerSocketChannel.open();
socketChannel.socket().bind(new InetSocketAddress("127.0.0.1".8089));
// Set non-blocking to accept clients
socketChannel.configureBlocking(false);
// Turn on the multiplexer
Selector selector = Selector.open();
// The server Socket registers with the multiplexer and specifies the event of interest
socketChannel.register(selector, SelectionKey.OP_ACCEPT);
// Multiplexer polling
ByteBuffer buffer = ByteBuffer.allocateDirect(1024);
while (selector.select() > 0){
Set<SelectionKey> selectionKeys = selector.selectedKeys();
Iterator<SelectionKey> selectionKeyIter = selectionKeys.iterator();
while (selectionKeyIter.hasNext()){
SelectionKey selectionKey = selectionKeyIter.next() ;
selectionKeyIter.remove();
if(selectionKey.isAcceptable()) {
// Accept new connections
SocketChannel client = socketChannel.accept();
// Set read non-blocking
client.configureBlocking(false);
// Register with the multiplexer
client.register(selector, SelectionKey.OP_READ);
} else if (selectionKey.isReadable()) {
// The channel is readable
SocketChannel client = (SocketChannel) selectionKey.channel();
int len = client.read(buffer);
if (len > 0){
buffer.flip();
byte[] readArr = new byte[buffer.limit()];
buffer.get(readArr);
System.out.println(client.socket().getPort() + "Port data :" + new String(readArr));
buffer.clear();
}
}
}
}
} catch(Exception e) { e.printStackTrace(); }}}Copy the codeAPI example: Client emulation
public class SecClient {
public static void main(String[] args) {
try {
// Connect to the server
SocketChannel socketChannel = SocketChannel.open();
socketChannel.connect(new InetSocketAddress("127.0.0.1".8089));
ByteBuffer writeBuffer = ByteBuffer.allocate(1024);
String conVar = "[hello-8089]";
writeBuffer.put(conVar.getBytes());
writeBuffer.flip();
// Send data every 5S
while (true) {
Thread.sleep(5000); writeBuffer.rewind(); socketChannel.write(writeBuffer); writeBuffer.clear(); }}catch(Exception e) { e.printStackTrace(); }}}Copy the codeSelectionKey binds the association between Selector and Chanel, and fetches a collection of channels in the ready state.
Six, source code address
Making address GitEE, https://github.com/cicadasmile/java-base-parent, https://gitee.com/cicadasmile/java-base-parentCopy the code