preface
Recently, I worked on a refrigerator project with a screen, which has a file upload function. The basic idea is to start a service on a LAN device, so that other devices in the LAN can access the file upload function through Http. Start a service on the device, using an open source microservice project called NanoHTTPD. There is only one Java file, but it contains a lot of details about network processing. Small as the sparrow is, it has all the organs. This article introduces the source code for NanoHTTPD, but not only that. I hope this article will cover some of the basic concepts of socket and HTTP, as I found when reading the source code of NanoHTTPD that these concepts are important to understanding. In addition, NanoHTTPD contains some of the details of network processing, and if you haven’t studied these details in depth before, it’s difficult to get a clear and systematic understanding of network transmission.
demo
The following code is the Sample provided by the official website, actually quite simple, specify a port, call start method can be. The serve method is used to process the request and respond, and the SAMPLE returns an HTML page.
public class App extends NanoHTTPD {
public App() throws IOException {
super(8080);
start(NanoHTTPD.SOCKET_READ_TIMEOUT, false);
System.out.println("\nRunning! Point your browsers to http://localhost:8080/ \n");
}
public static void main(String[] args) {
try {
new App();
} catch (IOException ioe) {
System.err.println("Couldn't start server:\n" + ioe);
}
}
@Override
public Response serve(IHTTPSession session) {
String msg = "<html><body><h1>Hello server</h1>\n";
Map<String, String> parms = session.getParms();
if (parms.get("username") == null) {
msg += "
+ "</form>\n";
} else {
msg += "<p>Hello, " + parms.get("username") + ! "" ";
}
return newFixedLengthResponse(msg + "</body></html>\n"); // Return the HTML page}}Copy the codeWhen the start method is called, the service is up. So let’s look inside and see what this start does.
/**
* Start the server.
*
* @param timeout timeout to use for socket connections.
* @param daemon start the thread daemon or not.
* @throws IOException if the socket is in use.
*/
public void start(final int timeout, boolean daemon) throws IOException {
this.myServerSocket = this.getServerSocketFactory().create();
this.myServerSocket.setReuseAddress(true); ServerRunnable serverRunnable = createServerRunnable(timeout); this.myThread = new Thread(serverRunnable); this.myThread.setDaemon(daemon); When the User thread terminates, the JVM also exits, and the Daemon thread terminates, but as long as the User thread is present, the JVM does not exit. this.myThread.setName("NanoHttpd Main Listener");
this.myThread.start();
while(! serverRunnable.hasBinded && serverRunnable.bindException == null) { try { Thread.sleep(10L); } catch (Throwable e) { // on android this may not be allowed, that's why we // catch throwable the wait should be very short because we are // just waiting for the bind of the socket } } if (serverRunnable.bindException ! = null) { throw serverRunnable.bindException; }}Copy the codeA ServerSocket instance is created, and then a thread is created, and operations are performed in this thread. The logic for what operations are performed is in the ServerRunnable.
/**
* The runnable that will be used for the main listening thread.
*/
public class ServerRunnable implements Runnable {
private final int timeout;
private IOException bindException;
private boolean hasBinded = false;
private ServerRunnable(int timeout) {
this.timeout = timeout;
}
@Override
public void run() { try { myServerSocket.bind(hostname ! = null ? new InetSocketAddress(hostname, myPort) : new InetSocketAddress(myPort)); hasBinded =true;
} catch (IOException e) {
this.bindException = e;
return;
}
do {
try {
final Socket finalAccept = NanoHTTPD.this.myServerSocket.accept();
if (this.timeout > 0) {
finalAccept.setSoTimeout(this.timeout);
}
final InputStream inputStream = finalAccept.getInputStream();
NanoHTTPD.this.asyncRunner.exec(createClientHandler(finalAccept, inputStream));
} catch (IOException e) {
NanoHTTPD.LOG.log(Level.FINE, "Communication with the client broken", e); }}while (!NanoHTTPD.this.myServerSocket.isClosed());
}
}
Copy the codeLet’s look at the run method of ServerRunnable. We created an instance of ServerSocket, and now we call its bind method. This only exposes me to a bind method on the Java level, but we need to know that Java also calls the interface provided by the system. In Java, the bind method corresponds to the bind and LISTEN methods of the system call. Let’s see what the bind and LISTEN system calls do. You can view the documentation of these system call apis by using the following command:
man 2 bind
Copy the codeThe bind () function
The bind() function assigns a specific address from an address family to a socket. For example, for AF_INET and AF_INET6, a combination of an ipv4 or ipv6 address and a port number is assigned to the socket. The IP address and port are connected to the socket.
listen
There is also a LISTEN method in real system-level Socket interfaces. It allows sockets of other processes to access the current socket.
Accept () function
ServerSocket receives a request from a client when it calls the Accept method. This method is a blocking method. Cannot proceed with the execution. This method is called when there is a client connect. It returns a Socket that contains both the server’s Socket descriptor and the address information returned by the client. There are two methods from this Socket: getInputStream and getOutputStream, which represent the data stream we send from the client and the data stream we return to the client.
What is a Socket
In the previous section we have created a Socket on the server and called its bind, LISTEN and other methods. But what exactly is Socket, let’s discuss now, before discussing what is Socket, let’s first understand what is network protocol, when we talk to each other, speak the language to comply with the grammar and language specification. Calls between machines must also conform to certain protocols. Otherwise, a chicken and a duck cannot understand each other. The network we use is made up of:
- Application layer protocols (HTTP, FTP, etc.)
- Transport layer protocols (TCP, UDP, etc.)
- Network layer protocol (IP protocol, etc.)
- Connection Layer Protocols (Ethernet and Wifi protocols)
These four layers of protocols are composed. They are ordered from top to bottom, meaning that upper-layer protocols depend on lower-layer protocols. For example, HTTP relies on TCP. After understanding these content we talk about what is Socket, it is the system of TCP/IP protocol encapsulation interface, that is to say Socket is not a protocol, it is just the TCP/IP protocol implementation, convenient for developers to develop the network. In Unix, everything is a file, and a Socket is a file. In other words, a network connection is a file. Because it has read and write and close functions to the data stream. When we set up a network connection, we create a socket file so that we can read data from other computers and write data to other computers.
Concurrent processing
As a server, one of the situations you face is concurrent access. NanoHTTPD actually does the processing in AsyncRunner’s exec method, and in DefaultAsyncRunner’s exec method, it starts a thread to process each access connection. The default upper limit for connections is 50. The real way to process requests is in ClientHandler.
ClientHandler’s run method
Client request processing is done in this method.
/**
* The runnable that will be used for every new client connection.
*/
public class ClientHandler implements Runnable {
...
@Override
public void run() {
OutputStream outputStream = null;
try {
outputStream = this.acceptSocket.getOutputStream();
TempFileManager tempFileManager = NanoHTTPD.this.tempFileManagerFactory.create();
HTTPSession session = new HTTPSession(tempFileManager, this.inputStream, outputStream, this.acceptSocket.getInetAddress());
while(! this.acceptSocket.isClosed()) { session.execute(); } } catch (Exception e) { // When the socket is closed by the client, // we throw our own SocketException // tobreak the "keep alive" loop above. If
// the exception was anything other
// than the expected SocketException OR a
// SocketTimeoutException, print the
// stacktrace
if(! (e instanceof SocketException &&"NanoHttpd Shutdown".equals(e.getMessage())) && ! (e instanceof SocketTimeoutException)) { NanoHTTPD.LOG.log(Level.FINE,"Communication with the client broken", e); } } finally { safeClose(outputStream); safeClose(this.inputStream); safeClose(this.acceptSocket); NanoHTTPD.this.asyncRunner.closed(this); }}}Copy the codeFrom the above logic, we can see that there is an HTTPSession created to handle each HTTP request, so let’s look at the httpsession.execute method.
HTTPSession.execute
In this method we get the InputStream, which is the client request data stream, and we get the request data from the client. The HTTP protocol we often hear about is useful here. Sockets encapsulate TCP/IP, but do not encapsulate application layer protocols. This level of protocol needs to be handled by ourselves. The logic in this execute method is our implementation of the Http protocol. While our service is running on one machine, we enter the Ip and port of the previous machine in the browser of the other machine. I know this is an Http request, so the request data goes to the inputStream. The next step is to parse the data according to the HTTP protocol.
@Override public void execute() throws IOException { Response r = null; Byte [] buf = new byte[httpsession.bufsize]; // Httpsession.bufsize; // Httpsession.bufsize; // Httpsession.bufsize; this.splitbyte = 0; this.rlen = 0; intread= 1; this.inputStream.mark(HTTPSession.BUFSIZE); try {read = this.inputStream.read(buf, 0, HTTPSession.BUFSIZE);
} catch (Exception e) {
safeClose(this.inputStream);
safeClose(this.outputStream);
throw new SocketException("NanoHttpd Shutdown");
}
if (read == -1) {
// socket was been closed
safeClose(this.inputStream);
safeClose(this.outputStream);
throw new SocketException("NanoHttpd Shutdown");
}
while (read > 0) {
this.rlen += read;
this.splitbyte = findHeaderEnd(buf, this.rlen);
if (this.splitbyte > 0) {
break;
}
read = this.inputStream.read(buf, this.rlen, HTTPSession.BUFSIZE - this.rlen);
}
if (this.splitbyte < this.rlen) {
this.inputStream.reset();
this.inputStream.skip(this.splitbyte);
}
this.parms = new HashMap<String, String>();
if(null == this.headers) { this.headers = new HashMap<String, String>(); // Create a header to store the parsed request header}else{ this.headers.clear(); } // Create a BufferedReader to parse the header BufferedReader hin = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(buf, 0, this.rlen))); // Decode the header into parms and header java properties Map<String, String> pre = new HashMap<String, String>(); decodeHeader(hin, pre, this.parms, this.headers); // We will save the parsed header in the map // we will print the parsed header data, and the author added the logfor (Map.Entry<String, String> entry : headers.entrySet()) {
Log.d(TAG, "header key = " + entry.getKey() + " value = " + entry.getValue());
}
if(null ! = this.remoteIp) { this.headers.put("remote-addr", this.remoteIp);
this.headers.put("http-client-ip", this.remoteIp);
}
this.method = Method.lookup(pre.get("method")); // Get the request methods get, POST, and PUTif (this.method == null) {
throw new ResponseException(Response.Status.BAD_REQUEST, "BAD REQUEST: Syntax error.");
}
this.uri = pre.get("uri"); This.cookies = new CookieHandler(this.headers); String connection = this.headers. Get ()"connection");
boolean keepAlive = protocolVersion.equals("HTTP / 1.1") && (connection == null || ! connection.matches("(? i).*close.*")); // Determine whether keepAlive is supported // Ok, nowdo the serve()
// TODO: long body_size = getBodySize();
// TODO: long pos_before_serve = this.inputStream.totalRead()
// (requires implementaion fortotalRead()) r = serve(this); // The serve is the method we implement in the opening instance code, where we create the response data to return to the client. // TODO: this.inputStream.skip(body_size - // (this.inputStream.totalRead() - pos_before_serve))if (r == null) {
throw new ResponseException(Response.Status.INTERNAL_ERROR, "SERVER INTERNAL ERROR: Serve() returned a null response.");
} elseString acceptEncoding = this.headers. Get (String acceptEncoding = this.headers."accept-encoding"); this.cookies.unloadQueue(r); r.setRequestMethod(this.method); r.setGzipEncoding(useGzipWhenAccepted(r) && acceptEncoding ! = null && acceptEncoding.contains("gzip")); r.setKeepAlive(keepAlive); r.send(this.outputStream); // Return the data to the client}if(! keepAlive ||"close".equalsIgnoreCase(r.getHeader("connection"))) {
throw new SocketException("NanoHttpd Shutdown");
}
} catch (SocketException e) {
// throw it out to close socket object (finalAccept)
throw e;
} catch (SocketTimeoutException ste) {
// treat socket timeouts the same way we treat socket exceptions
// i.e. close the stream & finalAccept object by throwing the
// exception up the call stack.
throw ste;
} catch (IOException ioe) {
Response resp = newFixedLengthResponse(Response.Status.INTERNAL_ERROR, NanoHTTPD.MIME_PLAINTEXT, "SERVER INTERNAL ERROR: IOException: "+ ioe.getMessage()); resp.send(this.outputStream); safeClose(this.outputStream); } catch (ResponseException re) { Response resp = newFixedLengthResponse(re.getStatus(), NanoHTTPD.MIME_PLAINTEXT, re.getMessage()); resp.send(this.outputStream); safeClose(this.outputStream); } finally { safeClose(r); this.tempFileManager.clear(); }}Copy the codeHere’s what this code does:
- Read the Header
- GET the request method, is it GET, PUT, POST?
- Get the URI, the request address
- Whether to support persistent connection
- Call the serve method, leave it to us to handle the implementation, and return the Response object. Response contains the data returned by the server to the client, including the corresponding header data, etc. It then writes to the socket via the outputStream, so that the server sends the data back to the client.
- So when we really understand the details of these network protocols, it’s really important. We can’t really enjoy the beauty of the mountain until we climb to the top, can’t we?
serve
Let’s take a look at the serve method, which we need to implement, but has an internal default implementation that handles file uploads
public Response serve(IHTTPSession session) {
Map<String, String> files = new HashMap<String, String>();
Method method = session.getMethod();
if(Method.PUT.equals(method) || Method.POST.equals(method)) { try { session.parseBody(files); // Important logic is in parseBody} catch (IOException ioe) {return newFixedLengthResponse(Response.Status.INTERNAL_ERROR, NanoHTTPD.MIME_PLAINTEXT, "SERVER INTERNAL ERROR: IOException: " + ioe.getMessage());
} catch (ResponseException re) {
return newFixedLengthResponse(re.getStatus(), NanoHTTPD.MIME_PLAINTEXT, re.getMessage());
}
}
Map<String, String> parms = session.getParms();
parms.put(NanoHTTPD.QUERY_STRING_PARAMETER, session.getQueryParameterString());
return serve(session.getUri(), method, session.getHeaders(), parms, files);
}
Copy the codeYes, that’s the session.parseBody() method.
ParseBody method
@Override
public void parseBody(Map<String, String> files) throws IOException, ResponseException {
RandomAccessFile randomAccessFile = null;
try {
long size = getBodySize();
ByteArrayOutputStream baos = null;
DataOutput request_data_output = null;
// Store the request in memory or a file, depending on size
if (size < MEMORY_STORE_LIMIT) {
baos = new ByteArrayOutputStream();
request_data_output = new DataOutputStream(baos);
} else {
randomAccessFile = getTmpBucket();
request_data_output = randomAccessFile;
}
// Read all the body and write it to request_data_output
byte[] buf = new byte[REQUEST_BUFFER_LEN];
while (this.rlen >= 0 && size > 0) {
this.rlen = this.inputStream.read(buf, 0, (int) Math.min(size, REQUEST_BUFFER_LEN));
size -= this.rlen;
if (this.rlen > 0) {
request_data_output.write(buf, 0, this.rlen);
}
}
ByteBuffer fbuf = null;
if(baos ! = null) { fbuf = ByteBuffer.wrap(baos.toByteArray(), 0, baos.size()); }else {
fbuf = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length());
randomAccessFile.seek(0);
}
// If the method is POST, there may be parameters
// in data section, too, read it:
if (Method.POST.equals(this.method)) {
String contentType = "";
String contentTypeHeader = this.headers.get("content-type");
Log.d(TAG, "contentTypeHeader = " + contentTypeHeader);
StringTokenizer st = null;
if(contentTypeHeader ! = null) { st = new StringTokenizer(contentTypeHeader,",; ");
if(st.hasMoreTokens()) { contentType = st.nextToken(); }}if ("multipart/form-data".equalsignoRecase (contentType)) {// File upload // Handle multipart/form-dataif(! st.hasMoreTokens()) { throw new ResponseException(Response.Status.BAD_REQUEST,"BAD REQUEST: Content type is multipart/form-data but boundary missing. Usage: GET /example/file.html"); } / / processing method of file upload decodeMultipartFormData (getAttributeFromContentHeader (contentTypeHeader BOUNDARY_PATTERN, null). // getAttributeFromContentHeader(contentTypeHeader, CHARSET_PATTERN,"US-ASCII"), fbuf, this.parms, files);
} else {
byte[] postBytes = new byte[fbuf.remaining()];
fbuf.get(postBytes);
String postLine = new String(postBytes).trim();
// Handle application/x-www-form-urlencoded
if ("application/x-www-form-urlencoded".equalsIgnoreCase(contentType)) {
decodeParms(postLine, this.parms);
} else if(postLine.length() ! = 0) { // Specialcase for raw POST data => create a
// special files entry "postData" with raw content
// data
files.put("postData", postLine); }}}else if (Method.PUT.equals(this.method)) {
files.put("content", saveTmpFile(fbuf, 0, fbuf.limit(), null)); } } finally { safeClose(randomAccessFile); }}Copy the codeWhat this code does:
- Gets the size of the body and, if less than 1k, stores it in the cache. If it is greater than 1K, the body data will be stored as a temporary file.
- This method mainly handles the body data of the POST request.
- For file upload, contentType is multipart/form-data. Read the uploaded file information, and save them in a specific location, in fact, through the specific protocol format parsing data, the uploaded file parsing out. This place is a little attention to the file name resolution, we receive the client upload, in fact, is in the upload information can get the file name. Give the style, through the regular way to resolve out.
decodeMultipartFormData
This method handles the file upload logic
/**
* Decodes the Multipart Body data and put it into Key/Value pairs.
*/
private void decodeMultipartFormData(String boundary, String encoding, ByteBuffer fbuf, Map<String, String> parms, Map<String, String> files) throws ResponseException {
try {
int[] boundary_idxs = getBoundaryPositions(fbuf, boundary.getBytes());
if (boundary_idxs.length < 2) {
throw new ResponseException(Response.Status.BAD_REQUEST, "BAD REQUEST: Content type is multipart/form-data but contains less than two boundary strings.");
}
byte[] part_header_buff = new byte[MAX_HEADER_SIZE];
for (int bi = 0; bi < boundary_idxs.length - 1; bi++) {
fbuf.position(boundary_idxs[bi]);
int len = (fbuf.remaining() < MAX_HEADER_SIZE) ? fbuf.remaining() : MAX_HEADER_SIZE;
fbuf.get(part_header_buff, 0, len);
BufferedReader in = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(part_header_buff, 0, len), Charset.forName(encoding)), len);
int headerLines = 0;
// First line is boundary string
String mpline = in.readLine();
headerLines++;
if(! mpline.contains(boundary)) { throw new ResponseException(Response.Status.BAD_REQUEST,"BAD REQUEST: Content type is multipart/form-data but chunk does not start with boundary.");
}
String part_name = null, file_name = null, content_type = null;
// Parse the reset of the header lines
mpline = in.readLine();
headerLines++;
while(mpline ! = null && mpline.trim().length() > 0) {Matcher Matcher = CONTENT_DISPOSITION_PATTERN.matcher(mpline);if (matcher.matches()) {
String attributeString = matcher.group(2);
matcher = CONTENT_DISPOSITION_ATTRIBUTE_PATTERN.matcher(attributeString);
while (matcher.find()) {
String key = matcher.group(1);
if (key.equalsIgnoreCase("name")) {
part_name = matcher.group(2);
} else if (key.equalsIgnoreCase("filename")) {
file_name = matcher.group(2);
}
}
}
matcher = CONTENT_TYPE_PATTERN.matcher(mpline);
if (matcher.matches()) {
content_type = matcher.group(2).trim();
}
mpline = in.readLine();
headerLines++;
}
int part_header_len = 0;
while (headerLines-- > 0) {
part_header_len = scipOverNewLine(part_header_buff, part_header_len);
}
// Read the part data
if (part_header_len >= len - 4) {
throw new ResponseException(Response.Status.INTERNAL_ERROR, "Multipart header size exceeds MAX_HEADER_SIZE.");
}
int part_data_start = boundary_idxs[bi] + part_header_len;
int part_data_end = boundary_idxs[bi + 1] - 4;
fbuf.position(part_data_start);
if (content_type == null) {
// Read the part into a string
byte[] data_bytes = new byte[part_data_end - part_data_start];
fbuf.get(data_bytes);
parms.put(part_name, new String(data_bytes, encoding));
} else{ // Read it into a file String path = saveTmpFile(fbuf, part_data_start, part_data_end - part_data_start, file_name); // Store files in a temporary directoryif(! files.containsKey(part_name)) { files.put(part_name, path); }else {
int count = 2;
while (files.containsKey(part_name + count)) {
count++;
}
files.put(part_name + count, path);
}
if(! parms.containsKey(part_name)) { parms.put(part_name, file_name); }else {
int count = 2;
while(parms.containsKey(part_name + count)) { count++; } parms.put(part_name + count, file_name); } } } } catch (ResponseException re) { throw re; } catch (Exception e) { throw new ResponseException(Response.Status.INTERNAL_ERROR, e.toString()); }}Copy the codeThe above code parses the uploaded file from the client and stores it on the server. There is a lot of regex matching in parsing the data, which is also a side effect of the Http protocol. Without THE Http protocol, it is impossible to parse the data using regex.
conclusion
We’ve covered the main logic in NanoHTTPD, which implements a server via Socket, while our client is implemented by a browser. Here is a complete flowchart of Socket client and server implementation to help you understand.
Well, NanoHTTPD source code is finished parsing, through the above introduction, we have a more profound understanding of Socket, HTTP protocol, TCP/IP protocol and so on. Not so confused. Of course, there are some system socket apis involved, and what I want to say is that the Java API and the system API are the same, but we should be more concerned with the underlying implementation and nature.
supplement
Format of an IPV6 address
When ServerSocket was created earlier, it was designed with IPV6 in mind. Here are some IPV6 formats.
IPv4 addresses are 32-bit in size, such as 192.168.0.1, with 8 bits between each dot. The IPv6 address is 128 bits. Use: segmentation, divided into eight parts, each part is 16, so most of the IPv6 address using hexadecimal representation, such as FFFF: FFFF: FFFF: FFFF: FFFF: FFFF: FFF.
The size of an IPv6 address is 128 bits. The preferred IPv6 address representation is X: X: X: X: X: X: X: X: X, where each X is a hexadecimal value of eight 16-bit parts of the address. IPv6 address range from 0000:0000-0000:0000-0000:0000-0000:0000 to FFFF: FFFF: FFFF: FFFF: FFFF: FFFF: FFFF: FFFF. This is the full representation format, but there are actually two common shorthand formats:
- Omit leading zeros Specify an IPv6 address by omitting leading zeros. IPv6 address, for example, 1050:0000-0000, 0000:0005-0600:300 c: 326 b can write 1050:0:0:0: hew 00:30 0 c: 326 b.
- A double colon specifies an IPv6 address by replacing a series of zeros with a double colon (::). For example, the IPv6 address FF06:0:0:0:0:0:0:c3 can be written as FF06 :: C3. An IP address can use the double colon only once. The alternative format for IPv6 addresses combines the colon and dot notation, so an IPv4 address can be embedded into an IPv6 address. Specify hexadecimal values for the leftmost 96 bits and decimal values for the rightmost 32 bits to indicate the embedded IPv4 address. This format ensures compatibility between IPv6 and IPv4 nodes when working in a mixed network environment.
reference
- The difference between socket and HTTP
- Unix Network Programming Volume one
- IPv6 Address Format