At present, many high-performance Java RPC frameworks are implemented based on Netty, and the design principle of Netty is inseparable from Java NIO. This note is a summary of NIO’s core three pieces: Buffer, Selector, and Channel Buffer implementation principles.
1. Buffer inheritance system
As shown above, there is a specific Buffer type for all primitive types in Java, and ByteBuffer is the one we use most often.
2. Use case of Buffer operation API
Here’s an example of IntBuffer:
/ * * *@author csp
* @date 2021-11-26 3:51 下午
*/
public class IntBufferDemo {
public static void main(String[] args) {
// Allocate a new int buffer.
// The current position of the new buffer is 0, and its limit (limit position) is its capacity. It has an underlying implementation array with an array offset of 0.
IntBuffer buffer = IntBuffer.allocate(8);
for (int i = 0; i < buffer.capacity(); i++) {
int j = 2 * (i + 1);
// Writes the given integer to the current position of this buffer, which is incremented.
buffer.put(j);
}
// Reset this buffer, set the limit position to the current position, and then set the current position to 0.
buffer.flip();
// Check if there is an element between the current position and the restricted position:
while (buffer.hasRemaining()){
// Reads the integer of the current position of this buffer, then increments the current position.
int j = buffer.get();
System.out.print(j + ""); }}}Copy the codeRunning results:
2, 4, 6, 8, 10, 12, 14, 16Copy the codeAs you can see from this example, IntBuffer is essentially used as an array container, which can be read by get (and written by PUT).
3. Basic principles of Buffer
Buffer Buffer is essentially a special type of an array of objects, is different from common array, its built-in mechanisms, able to track and record the state of the Buffer, if we use the get () method to get data from the Buffer or using the put () method to write data Buffer, can cause the change of state of the Buffer.
The Buffer built-in array implements state changes and tracking, essentially through three field variables:
- position: specifies the index of the next element to be written or read
get()/put()Method is automatically updated, and position is initialized to 0 when a new Buffer object is created. - Limit: Specifies how much more data needs to be fetched (when writing to the channel from the buffer) or how much more space can be put in (when reading into the buffer from the channel).
- Capacity: Specifies the maximum amount of data that can be stored in the buffer. In effect, it specifies the size of the underlying array, or at least the capacity of the underlying array that we are allowed to use.
The source code is as follows:
public abstract class Buffer {
0 <= position <= limit <= capacity
private int position = 0;
private int limit;
private intcapacity; . }Copy the codeIf we create a new ByteBuffer with a capacity of 10, position is set to 0 and limit and Capacity are set to 10, capacity will not change during future use of the object. The other two will change with use.
Let’s take a look at the Netty core principles and handwriting PRC framework combat this book introduced an example:
Prepare a TXT file and store it in the project directory. Enter the following contents in the file:
Java
Copy the codeTo verify the changing process of position, limit, and capacity, we use a code as follows:
/ * * *@author csp
* @date 2021-11-26 4:09 下午
*/
public class BufferDemo {
public static void main(String[] args) throws IOException {
FileInputStream fileInputStream = new FileInputStream("/Users/csp/IdeaProjects/netty-study/test.txt");
// Create an operation pipe for the file
FileChannel channel = fileInputStream.getChannel();
// Allocate a buffer of size 10 (essentially a byte array of size 10)
ByteBuffer buffer = ByteBuffer.allocate(10);
output("Initialize", buffer);
channel.read(buffer);// Read data from the pipe into the buffer container
output("Calls read ()", buffer);
// To prepare the operation, lock the operation scope:
buffer.flip();
output("Call the flip ()", buffer);
// Determine whether there is readable data
while (buffer.remaining() > 0) {byte b = buffer.get();
}
output("Call the get ()", buffer);
// It can be interpreted as unlock
buffer.clear();
output("Call the clear ()", buffer);
// Close the pipe
fileInputStream.close();
}
/** * Prints the real-time status in the buffer **@param step
* @param buffer
*/
public static void output(String step, Buffer buffer) {
System.out.println(step + ":");
// Capacity (array size) :
System.out.print("capacity" + buffer.capacity() + ",");
// The location of the current operation data, also known as the cursor:
System.out.print("position" + buffer.position() + ",");
// Lock value, flip, data operation range index can only be between position-limit:
System.out.println("limit"+ buffer.limit()); System.out.println(); }}Copy the codeThe following output is displayed:
Init: Capacity10, position0, limit10 call flip() Capacity10, POSItion0, limit4 Get () : Capacity10, POSItion4, limit4 Clear () : Capacity10, POSItion0, limit10Copy the codeHere is a graphical analysis of the results of the above code (around the values of position, limit, and capacity) :
// Allocate a buffer of size 10 (essentially a byte array of size 10)
ByteBuffer buffer = ByteBuffer.allocate(10);
// Read data from the pipe into the buffer container
channel.read(buffer);
output("Calls read ()", buffer);
Copy the codeFirst read some data from the channel into the buffer (note that reading data from the channel is equivalent to writing data to the buffer). If four bytes of data are read, then the value of position is 4, that is, the next byte index to be written is 4, and limit is still 10, as shown below.
// To prepare the operation, lock the operation scope:
buffer.flip();
output("Call the flip ()", buffer);
Copy the codeThe flip() method must be called before the next step is to write the read data to the output channel, equivalent to reading from the buffer. This method will do two things:
- First, set limit to the current position value.
- Second, set position to 0.
Since position is set to 0, it is guaranteed that the first byte of the buffer will be read in the next output, and limit being set to the current position ensures that the data read is exactly the data previously written to the buffer, as shown below.
// Determine whether there is readable data
while (buffer.remaining() > 0) {byte b = buffer.get();
}
output("Call the get ()", buffer);
Copy the codeCalling the get() method to read data from the buffer and write it to the output channel causes position to increase while limit remains the same, but position does not exceed the value of limit, so after the 4 bytes written to the buffer before reading, both position and limit are 4, as shown below.
// It can be interpreted as unlock
buffer.clear();
output("Call the clear ()", buffer);
// Close the pipe
fileInputStream.close();
Copy the codeAfter reading from the buffer, the limit value remains the same as when the flip() method was called, and the clear() method sets all state changes to the initialization value, closing the stream, as shown in the figure below.
Buffer is a special array container that contains three pointer variables: position, limit, and Capacity. These variables are used to track and record the state of the Buffer.
4. The allocate method initializes a buffer of a specified size
When we create a buffer object, we call static allocate() to specify the size of the buffer. When we call allocate(), we create an array of a specified size and wrap it as a buffer object.
Allocate () = allocate()
// Under ByteBuffer
public static ByteBuffer allocate(int capacity) {
if (capacity < 0)
throw new IllegalArgumentException();
// Create a ByteBuffer array object with capacity and limit values of capacity
return new HeapByteBuffer(capacity, capacity);
}
// It is located under HeapByteBuffer and its parent is: ByteBuffer
HeapByteBuffer(int cap, int lim) {
super(-1.0, lim, cap, new byte[cap], 0);// Call the parameter constructor of ByteBuffer
}
// It is under ByteBuffer and its parent is: Buffer
ByteBuffer(int mark, int pos, int lim, int cap,
byte[] hb, int offset){
super(mark, pos, lim, cap);// Call the Buffer constructor
this.hb = hb;// final byte[] hb; An immutable byte array
this.offset = offset;/ / the offset
}
// The Buffer constructor
Buffer(int mark, int pos, int lim, int cap) {
if (cap < 0)
throw new IllegalArgumentException("Negative capacity: " + cap);
this.capacity = cap;// Array capacity
limit(lim);// Array limit
position(pos);// Array positio
if (mark >= 0) {
if (mark > pos)
throw new IllegalArgumentException("mark > position: ("
+ mark + ">" + pos + ")");
this.mark = mark; }}Copy the codeEssentially equivalent to the following code:
// Initialize a byte array
byte[] bytes = new byte[10];
// Wrap the array around ByteBuffer
ByteBuffer buffer = ByteBuffer.wrap(bytes);
Copy the code5, slice method buffer fragmentation
In Java NIO, you can create a child Buffer from the Buffer object that was used first. That is, a slice is cut out of the existing buffer as a new buffer, but the existing buffer shares data with the created child buffer at the underlying array level.
The sample code looks like this:
/ * * *@author csp
* @date 2021-11-28 6:20 下午
*/
public class BufferSlice {
public static void main(String[] args) {
ByteBuffer buffer = ByteBuffer.allocate(10);
// Put data to the buffer: 0~9
for (int i = 0; i < buffer.capacity(); i++) {
buffer.put((byte) i);
}
// Create a subbuffer from subscript 3 to subscript 7
buffer.position(3);
buffer.limit(7);
ByteBuffer slice = buffer.slice();
// Change the contents of the child buffer
for (int i = 0; i < slice.capacity(); i++) {
byte b = slice.get(i);
b *= 10;
slice.put(i, b);
}
// Position and limit are restored to their original positions:
buffer.position(0);
buffer.limit(buffer.capacity());
// Print the contents of the buffer container:
while(buffer.hasRemaining()) { System.out.println(buffer.get()); }}}Copy the codeIn this example, a buffer of size 10 is allocated, and data 0 to 9 are placed in it. A child buffer is created based on this buffer, and the contents of the child buffer are changed. The output shows that only the “visible” part of the child buffer is changed. The subbuffer is shared with the original buffer, and the output is as follows:
0
1
2
30
40
50
60
7
8
9
Copy the code6. Read-only buffers
A read-only buffer, as its name implies, only reads data from the buffer, not writes to it.
Convert an existing buffer to a read-only buffer by calling asReadOnlyBuffer(). This method returns a buffer exactly the same as the original buffer and shares data with the original buffer, except that it is read-only. If the contents of the original buffer change, the contents of the read-only buffer also change.
The sample code looks like this:
/ * * *@author csp
* @date 2021-11-28 6:33 下午
*/
public class ReadOnlyBuffer {
public static void main(String[] args) {
// Initialize a buffer of size 10
ByteBuffer buffer = ByteBuffer.allocate(10);
// Put data to the buffer: 0~9
for (int i = 0; i < buffer.capacity(); i++) {
buffer.put((byte) i);
}
// Make the buffer read-only
ByteBuffer readOnlyBuffer = buffer.asReadOnlyBuffer();
// Since buffer and readOnlyBuffer essentially share a byte[] array object,
// Therefore, changing the contents of the buffer will cause the contents of readOnlyBuffer to change.
for (int i = 0; i < buffer.capacity(); i++) {
byte b = buffer.get(i);
b *= 10;
buffer.put(i, b);
}
// Position and limit are restored to their original positions:
readOnlyBuffer.position(0);
readOnlyBuffer.limit(buffer.capacity());
// Print the contents of readOnlyBuffer:
while(readOnlyBuffer.hasRemaining()) { System.out.println(readOnlyBuffer.get()); }}}Copy the codeThe following output is displayed:
0
10
20
30
40
50
60
70
80
90
Copy the codeIf you attempt to modify the contents of a read-only buffer, the ReadOnlyBufferException exception is reported. Only regular buffers can be converted to read-only buffers, not read-only buffers to writable buffers.
7. Direct buffer
Reference article: Direct and indirect buffers in the Java NIO learning article
For the definition of a direct buffer, the book Understanding the Java Virtual Machine goes like this:
- Java NIO byte buffers are either direct or indirect. If for direct byte buffer, the Java virtual opportunity to try their best to direct buffer on the implementation of the native IO operations, that is, based on each call to an operating system before and after the machine IO operations, virtual machine can avoid the kernel buffer content is copied to the user process buffer, or, in turn, Try to avoid copying from user process buffers into kernel buffers.
- Direct buffers can be made by calling the buffer class
allocateDirect(int capacity)Method creation,The cost of allocating and unallocating the buffers returned by this method is higher than that of indirect buffers. The contents of the direct buffers reside outside the garbage collection heap, so they require little application memory (JVM memory). Therefore, it is recommended that direct buffers be allocated to buffers that are large and persistent (that is, buffers whose data will be reused). In general, it is best to allocate direct buffers only when they provide a significant benefit in program performance. - Direct buffers can also be accessed via FileCHannel
map()The MappedByteBuffer method is created by mapping the file region into memory. The implementation of the Java platform helps create directly through JNI native code byte buffer, if these points to a buffer in the buffer instance is an access to the area of memory, tries to approach the area will not change the contents of the buffer, and during the visit, or some time later lead to abnormal reported uncertainty. - A byte buffer is either a direct buffer or an indirect buffer by calling it
isDIrect()Method to judge.
Case code:
/ * * *@author csp
* @date 2021-11-28 7:07 下午
*/
public class DirectBuffer {
public static void main(String[] args) throws IOException {
// Read the contents of the test. TXT file from disk
FileInputStream fileInputStream = new FileInputStream("/Users/csp/IdeaProjects/netty-study/test.txt");
// Create an operation pipe for the file
FileChannel inputStreamChannel = fileInputStream.getChannel();
// Write the read to a new file
FileOutputStream fileOutputStream = new FileOutputStream("/Users/csp/IdeaProjects/netty-study/test2.txt");
FileChannel outputStreamChannel = fileOutputStream.getChannel();
// Create a direct buffer
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(1024);
while (true){
byteBuffer.clear();
int read = inputStreamChannel.read(byteBuffer);
if (read == -1) {break; } byteBuffer.flip(); outputStreamChannel.write(byteBuffer); }}}Copy the codeTo allocate a direct buffer, the allocateDirect() method is called instead of the allocate() method, and is used in the same way as normal buffers.
8. Memory mapping
Memory mapping is a method of reading and writing file data that can be much faster than regular stream-based or channel-based I/O. Memory-mapped file I/O is done by having the data in the file represented as the contents of an in-memory array, which at first sounds like nothing more than reading the entire file into memory, but it’s not. In general, only the parts of the file that are actually read or written are mapped to memory. Take a look at the following example code:
/ * * *@author csp
* @date 2021-11-28 7:16 下午
*/
public class MapperBuffer {
static private final int start = 0;
static private final int size = 10;
public static void main(String[] args) throws IOException {
RandomAccessFile randomAccessFile = new RandomAccessFile("/Users/csp/IdeaProjects/netty-study/test.txt"."rw");
FileChannel channel = randomAccessFile.getChannel();
// Create a mapping between the buffer and the file system. If you manipulate the contents of the buffer, the contents of the file will change accordingly
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_WRITE, start, size);
mappedByteBuffer.put(4, (byte) 97);// a
mappedByteBuffer.put(5, (byte) 122);// zrandomAccessFile.close(); }}Copy the codeThe original test. TXT file contains the following contents:
Java
Copy the codeAfter executing the above code, the contents of the test.txt file are updated to:
Javaaz
Copy the codeReference books: “Netty core principles and handwritten PRC framework combat”, the book PDF can be obtained from my public number [interest made of straw hat Lu Fei] reply 001!
Recommended information: Java NIO buffers