I always think that HTTP protocol and other common network knowledge is a must for students engaged in Web front-end development. On the one hand, a lot of front-end work such as Web performance optimization, most of the rules are directly related to the characteristics of HTTP, HTTPS, SPDY, TCP and other protocols, if you do not start from the protocol itself but blindly follow the doctrine, it is likely to backfire. On the other hand, with the development and expansion of Node.js, more and more front-end students start to write server-side programs, or even server-side framework (ThinkJS framework is developed by front-end engineers and has a large number of front-end engineer users), and master the necessary network knowledge. It is very important for server program security, deployment, operation and maintenance.

My blog has a “HTTP related” topic, and will be updated with more content in the future, welcome to follow. Today’s topic is x-Forwarded-For (XFF).

background

As the name suggests, X-Forwarded-For is an HTTP extension header. The HTTP/1.1 (RFC 2616) protocol does not define it. Squid was originally introduced by the cache proxy software to represent the real IP address of the HTTP requestor. Today, it has become a de facto standard. It is widely used by HTTP proxies, load balancers and other forwarding services. It is written into THE STANDARD OF RFC 7239 (ForwardedHTTP Extension).

The format of the X-Forwarded-For header is very simple. It goes like this:

X-Forwarded-For: client, proxy1, proxy2
Copy the code

As you can see, the XFF content consists of multiple sections separated by commas and Spaces, starting with the IP address of the device farthest from the server and then the IP address of each level of the proxy device.

If an HTTP request passes through three proxies Proxy1, Proxy2 and Proxy3 with IP addresses of IP1, IP2 and IP3 respectively, and the user’s real IP address is IP0, then according to XFF standard, the server will receive the following information:

X-Forwarded-For: IP0, IP1, IP2
Copy the code

Proxy3 directly connects to the server, and it appends IP2 to XFF to indicate that it is forwarding requests for Proxy2. There is no IP3 in the list, which is available on the server via the Remote Address field. We know that HTTP connection is based on TCP connection, and there is no concept of IP in HTTP protocol. Remote Address comes from TCP connection and represents the IP Address of the device that establishes TCP connection with the server. In this case, it is IP3.

Remote Address cannot be forged, because establishing a TCP connection requires a three-way handshake. If the source IP Address is forged, the TCP connection cannot be established, let alone subsequent HTTP requests. Different languages for Remote Address the way different, such as PHP is $_SERVER [” REMOTE_ADDR “], Node. Js is the req. Connection. RemoteAddress, but the principles are the same.

The problem

With that background, ask questions. I wrote the simplest Web Server for testing in Node.js. The HTTP protocol is language agnostic, and node.js is used here just for demonstration purposes. You can get the same results in any other language. The same is true for Nginx in this article, or Apache or another Web Server if you’re interested.

This code listens on port 9009 and prints some information when an HTTP request is received:

var http = require('http');

http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.write('remoteAddress: ' + req.connection.remoteAddress + '\n');
    res.write('x-forwarded-for: ' + req.headers['x-forwarded-for'] + '\n');
    res.write('x-real-ip: ' + req.headers['x-real-ip'] + '\n');
    res.end();
}).listen(9009.'0.0.0.0');
Copy the code

This code, in addition to the Remote Address and X-Forwarded-For mentioned earlier, also has an X-real-IP, which is another custom header field. X-real-ip is typically used by AN HTTP proxy to represent the IP address of the device with which it makes a TCP connection, which may be another proxy or the actual requester. It is important to note that X-real-IP does not currently belong to any standard, and any custom header can be agreed between the proxy and the Web application to pass this information.

The node.js service can now be accessed directly using the domain name + port number, with a Nginx reverse proxy:

location / {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_set_header X-NginX-Proxy true;

    proxy_pass http://127.0.0.1:9009/;
    proxy_redirect off;
}
Copy the code

My Nginx listens on port 80, so I can access Nginx forwarded services without a port.

Test direct access to Node service:

Curl http://t1.imququ.com:9009/ remoteAddress: 114.248.238.236 X-ray forwarded - for: undefined x - real - IP: undefinedCopy the code

Since my computer is directly connected to the Node.js service, Remote Address is my IP. I also didn’t specify additional custom headers, so the last two fields are undefined.

To access the service that Nginx forwarded:

Curl http://t1.imququ.com/ remoteAddress: 127.0.1x-Forwarded-For: 114.248.238.236 x-real-IP: 114.248.238.236Copy the code

This time, my computer accesses the Node.js service through Nginx and the Remote Address I get is actually the local IP of Nginx. These two lines from the previous Nginx configuration do work, adding two additional custom headers to the request:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
Copy the code

In fact, deploying A Web application in a production environment, using the second approach above, has many benefits. But this introduces a catch: Many Web applications get the IP from the HTTP request header in order to get the user’s real IP.

Http_forwarded_for () {http_forwarded_for () {http_forwarded_for () {http_forwarded_for () {http_forwarded_for () {http_forwarded_for () {http_forwarded_for ();

Access the Node.js service directly:

BASHcurl http://t1.imququ.com:9009/ - H 'X-ray Forwarded - For: 1.1.1.1 -h' X - Real - IP: 2.2.2.2 remoteAddress: 114.248.238.236 X-Forwarded-For: 1.1.1.1 x-real-IP: 2.2.2.2Copy the code

For Web applications, X-Forwarded-For and X-real-IP are just two common request headers. They are automatically Forwarded without any processing. This indicates that for direct deployment, except for the Remote Address obtained from the TCP connection, the IP information carried in the request header cannot be trusted.

To access Nginx forwarded services:

curl http://t1.imququ.com/ -H 'X - Forwarded - For: 1.1.1.1' -H 'X - Real - IP: 2.2.2.2'RemoteAddress: 127.0.1x-Forwarded-For: 1.1.1.1, 114.248.238.236 x-real-IP: 114.248.238.236Copy the code

This time, Nginx will append my IP to X-Forwarded-For. Overwrite the X-real-IP request header with my IP. This means that with Nginx processing, the last section of X-Forwarded-For and the entire x-real-IP content cannot be constructed and can be used to obtain user IP addresses.

User IP is often used in Web security-related scenarios, such as checking where users log in, ip-based access frequency control, and so on. In this scenario, it is more important to ensure that the IP cannot be constructed. After the previous tests and analysis, for Web applications deployed directly to users, the Remote Address obtained from TCP connections must be used. For Web applications with reverse proxies like Nginx, if the Set Header behavior is correctly configured, you can use either the x-real-IP or X-Forwarded-For last section (which must be equivalent) sent from Nginx.

So how does the Web application itself determine whether a request is coming directly or forwarded by a controlled proxy? Adding extra headers to proxy forwarding is one option, but not a safe one because headers are too easy to construct. If you must use it this way, the custom head must be long and rare, and must be kept under lock and key.

There is also a way to determine whether Remote Address is a local IP, but it is not perfect because Remote Address is 127.0.0.1 when accessed from the Nginx server, whether directly connected or through the Nginx proxy. This problem is usually ignored, but to make matters worse, the reverse proxy server and the actual Web application are not necessarily deployed on the same server. Therefore, it is more reasonable to collect the list of all proxy server IP addresses and compare them one by one with the Remote Address of the Web application to determine which way to access.

In general, to simplify logic, production environments block direct access to Web applications through ports, allowing only access through Nginx. Does that make it okay? Not really.

First, if the user actually accesses Nginx through a proxy, x-Forwarded-For, the last section, and X-real-IP get the proxy’S IP address. This is only used For security-related scenarios. This is when the first IP of x-Forwarded-For comes into play. Here’s a question to note, again using the previous example to test:

http://t1.imququ.com/ -h 'X-Forwarded-For: unknown, <>"1.1.1.1' remoteAddress: 127.0.1x-Forwarded-For: Unknown, <>"1.1.1.1, 114.248.238.236 x-real-IP: 114.248.238.236Copy the code

X-forwarded-for The last section is appended to by Nginx, but before that, it comes from the request header that Nginx receives. This portion of user input is completely untrusted. Use it in IP format. Otherwise, security vulnerabilities such as SQL injection and XSS may occur.

conclusion

  1. For Web applications that directly provide services externally, when performing operations related to security, they can only obtain IP through Remote Address and cannot trust any request headers.
  2. If the configuration is correct, this parameter is required for Web applications that use a Web Server, such as Nginx, to implement reverse proxyX-Forwarded-ForThe last section orX-Real-IPRemote Address is the internal IP Address of the server where Nginx resides. It should also prohibit Web applications from providing services directly to the outside world;
  3. In non-security-related scenarios, such as displaying local weather by IP, you can use theX-Forwarded-ForThe first position obtains the IP address, but needs to verify the VALIDITY of the IP format.

PS: Some articles on the Internet suggest that Nginx should be configured in this way, which is not reasonable:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
Copy the code

This configuration does improve security, but it also results in all proxy information being erased before the request reaches Nginx, rendering the actual proxy user no better served. Still should understand this middle principle, case by case analysis.

Originally published as 2015-05-02 02:40:45.

I am the blogger of “JerryQu’s Little Station (imququ.com)”. My independent blog has been suspended for a few years now, and I will bring some useful content to the nuggets community in the near future, as well as write some new articles in nuggets, please follow me.