One day a lady was walking by a construction site when she saw three men working. She asks the first man, “What are you doing?” The first man gets a little annoyed and shouts, “Can’t you see I’m laying bricks?” Not satisfied with this answer, the woman then asks the second man what he is doing. The second man replies, “I’m building a brick wall.” Then he turned to the first man and said, “Hey, the bricks you built are already higher than the wall. You must take out the last brick.” Not satisfied with this answer, the woman then asked the third man what he was doing. The third man looked up at her and said, “I’m building the biggest church the world has ever seen.” While he was staring up at the sky, the other two were already arguing about the extra brick. He turned slowly to the first two and said, “Never mind the brick, brothers. It’s an interior wall, and it’s going to be plastered, and no one’s going to notice it. Then build the lower layer.”
The moral of the story is that when you understand the design of the whole system and understand how the different components fit together (bricks, walls, churches), you will be able to find and solve problems (extra bricks) much faster.
But what does this story have to do with developing a web server from scratch?
In my opinion, to become a better programmer, you must have a better understanding of the software systems you use on a daily basis, and this includes programming languages, compilers, interpreters, databases and operating systems, web servers, and web development frameworks. To understand these systems better and more deeply, you have to reinvent them from scratch, one step at a time.
Confucius said: not to smell is not to smell, smell is not to see, see is not to know, know is not to do.
Don’t smell it, don’t smell it
Review images
What you hear is not what you see
Review images
See not if you know, know not if you do.
Review images
I hear and I forget, I see and I remember, I do and I understand. Foreigners generally believe that it comes from Confucius, but when looking for the English origin of this sentence, I found a blog post claiming that the Chinese version of this sentence actually comes from Xunzi’s “Confucianism Xiaoxuan”, which is indeed true.
I hope that by the time you read this, you will have endorsed the idea of learning how it works by redeveloping different software systems.
Do-it-yourself Web Server is a three-part guide to developing a simple web server from scratch. Let’s get started.
First, what exactly is a web server?
Review images
Simply put, it is a networking server set up on a physical server that waits permanently for a client to send a request. When the server receives the request, it generates a response and returns it to the client. The communication between client and server is carried out by HTTP protocol. The client can be a browser or any software that supports the HTTP protocol.
So what would a simple implementation of a web server look like? Here’s how I understand it. The sample code is implemented in Python, but even if you don’t know Python, you should be able to understand the concepts from the code and the following explanations:
import socket HOST, PORT = '', 8888 listen_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) listen_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) listen_socket.bind((HOST, PORT)) listen_socket.listen(1) print 'Serving HTTP on port %s ... ' % PORT while True: client_connection, client_address = listen_socket.accept() request = client_connection.recv(1024) print request http_response = """\ HTTP/1.1 200 OK Hello, World! """ client_connection.sendall(http_response) client_connection.close()
Save the above code as webServer1.py, or download it directly from my Github repository and run the file from the command line:
$python webServer1.py Serving HTTP on port 8888...
Next, in the browser’s address bar enter this link: http://localhost:8888/hello, and then press the enter key, you will see the magic scene. In your browser, you should see “Hello, World!” This sentence:
Review images
Isn’t that amazing? Next, let’s look at the implementation principle behind it.
First, let’s look at the network address you entered. A Uniform Resource Locator (URL) has the following basic structure:
Review images
With the URL, you tell the browser the address of the web server it needs to discover and connect to, and the path to the page on that server. But before the browser can send an HTTP request, it first establishes a TCP connection with the target web server. The browser then sends the HTTP request to the server over the TCP connection and waits for the server to return the HTTP response. When the browser receives a response, it displays the response on the page. In the example above, the browser displays “Hello, World!” This sentence.
So how do you establish a TCP connection before the client sends a request and the server returns a response? To establish a TCP connection, both the server and the client use what are called sockets. Next, instead of using the browser directly, we simulate the browser manually using Telnet at the command line.
On the same computer running the network server, open a Telnet session from the command line, set the host to connect to localhost, set the connection port to 8888, and press Enter:
$ telnet localhost 8888 Trying 127.0.0.1 … Connected to localhost.
Once you’ve done this, you’ve actually established a TCP connection with a local running web server, ready to send and receive HTTP messages. The following image shows the standard process that the server must complete to accept a new TCP connection.
Review images
In the Telnet session above, we type GET /hello HTTP/1.1 and press Enter:
$ telnet localhost 8888 Trying 127.0.0.1 … Connected to localhost. GET /hello HTTP/1.1 HTTP/1.1 200 OK Hello, World!
You successfully simulated the browser manually! You manually send an HTTP request and receive an HTTP response. The following diagram shows the basic structure of an HTTP request:
Review images
The HTTP request line contains the HTTP method (GET method is used here because we want to GET content from the server), the server page path (/hello), and the version of the HTTP protocol.
To keep things simple, the web server we’re currently implementing doesn’t parse the request, you can just type in code that doesn’t make any sense and still get “Hello, World!” The response.
After you enter the request code and press Enter, the client sends the request to the server, which parses the request and returns an HTTP response.
The following diagram shows the details of the HTTP response returned by the server to the client:
Review images
So let’s analyze it. The response contains the status line HTTP/1.1 200 OK, followed by the required blank line, followed by the body of the HTTP response.
The status line HTTP/1.1 200 OK contains the HTTP version, HTTP status code, and Reason Phrase corresponding to the status code. When the browser receives the response, it displays the body of the response, which is why you’ll see “Hello, World! This sentence.
This is how web servers work. To recap: The network server first creates a listening socket and starts a persistent loop to receive new connections. The client initiates a TCP connection with the server and sends an HTTP request to the server after a successful connection is established. The server then returns an HTTP response. To establish a TCP connection, both the client and the server use sockets.
Now that you have a basic working simple Web server, you can test it using a browser or other HTTP client. As shown above, you can become an HTTP client yourself by using the Telnet command and manually entering HTTP requests.
Here’s a puzzle: How do you run Djando, Flask, and Pyramid applications on this server without making any changes to the server code, while meeting the requirements of these different network frameworks?
The answer will be found in the second part of this do-It-Yourself Web server series.