The problem background

A previous online vulnerability scan encountered a strange problem :requests. Get stuck even with timeout set.

See lijiejie big guy requests. Get exception hang live also encountered this problem.

So, I wanted to explore the following questions:

• What does the timeout parameter in the Requests library mean?
• Why does the timeout parameter “expire” in requests. Get?

The analysis process

What does the timeout parameter in the Requests library mean?

• What is the timeout parameter in the Requests library?

  Timeout: connect timeout: read timeout

• What is a Connect timeout?

  The connect system call is required by the client to do the TCP three-way handshake with the server. When the service address does not exist on the Internet, the CONNECT system call takes a long time.

  For example, a request for 1.1.2.3 will return a connect tiemout after some time:

  python -c 'import requests; Requests. The get (" http://1.1.2.3 ") 'Copy the code

  If the client does not respond to a request for IP address 1.1.2.3, the client retransmits the SYN packet

  • The retransmission times can be configured in /proc/sys/net/ipv4/tcp_syn_retries
  • The retransmission interval is not fixed, and on Linux the test results are [1,3,7,15,31]s, which seems to be 2^(n+1)-1

  If no ACK packet is received after the retry, a Connect timeout occurs

  The timeout parameter of the request reduces the wait for retransmission.

  python -c 'import requests; Get ("http://1.1.2.3", timeout=(1, 100))' # 1s connect timeoutCopy the code

• How to implement the connect timeout control?

  System calls such as connect and read have no parameters to control the timeout.

  An implementation of the connect function can be found in Modules/ socketModule. c

  1. Set the socket to non-blocking mode... sock_call_ex(... ,_PyTime_t timeout) { ... interval = timeout; . res = internal_select(s, writing, interval, connect); . } static int internal_select(PySocketSockObject *s, int writing, _PyTime_t interval, int connect){ ... ms = _PyTime_AsMilliseconds(interval, _PyTime_ROUND_CEILING); . n = poll(&pollfd, 1, (int)ms); 2. The poll system call, which returns the poll system call if it times outCopy the code

  The process is as follows:

  After setting the socket to non-blocking mode, the connect system call is called. The poll system call is used to determine whether a timeout occursCopy the code

  In fact, this is a very common way to timeout connect, and it can be done in other TCP clients.

• What is read timeout?

  The client needs to invoke the read system call to read the data sent by the server, and if the server keeps not sending the data, it will get stuck reading the data.

  For example, we use nc command to open a server that only listens to establish links, but does not send data.

  nc -l 8081
  Copy the code

  The client requests the service enabled by NC. The code is as follows. The read timeout occurs after 3s

  Get ("http://127.0.0.1:8081", timeout=(1,3)) # read set timeout to 3sCopy the code

Why does the timeout parameter “expire” in requests. Get?

Get can take a long time in the DNS resolution, connect, and read phases. Here is whether timeout takes effect in the three phases.

The document only says that timeout controls connect and read phases, indicating that timeout cannot control DNS resolution when it takes a long time. In my own experiments, I came to the same conclusion: EVEN if the DNS resolution time exceeds timeout, it will not throw an exception.

The Connect phase was analyzed above, and a timeout can control how long the phase takes.

The Read timeout in Python is not a global time, it simply cannot be exceeded every time a socket is read. A read of a response may have multiple reads. This may not have the same meaning as other HTTP client timeouts such as curl.

If the server can make the client read too many times, each time within the read timeout value, the client will freeze.

Therefore, the read timeout argument can be “invalidated” in two cases:

• The content-Length of the response is an unusually large number, with the server slowly responding 1 byte at a time
• The response code returned by the server is 100, and the continuous return of the response header from the server also results in continuous reads from the client

For example, the following server keeps returning response headers, causing the client to get stuck.

# coding:utf-8 from socket import * from multiprocessing import * from time import sleep def dealWithClient(newSocket,destAddr): RecvData = newsocket.recv (1024) newsocket.send (b""" "HTTP/1.1 100 OK\n""") while True: # recvData = newSocket.recv(1024) newSocket.send(b"""x:a\n""") if len(recvData)>0: # print('recv[%s]:%s'%(str(destAddr), recvData)) pass else: print('[%s]close'%str(destAddr)) sleep(10) print('over') break # newSocket.close() def main(): serSocket = socket(AF_INET, SOCK_STREAM) serSocket.setsockopt(SOL_SOCKET, SO_REUSEADDR , 1) localAddr = ('', 8085) serSocket.bind(localAddr) serSocket.listen(5) try: while True: newSocket,destAddr = serSocket.accept() client = Process(target=dealWithClient, args=(newSocket,destAddr)) client.start() newSocket.close() finally: serSocket.close() if __name__ == '__main__': main()Copy the code

More discussion can be found in the submitted bug URllib HTTP Client Possible Infinite Loop on a 100 Continue Response

conclusion

Requests can take a long time in the DNS resolution, Connect, and read phases, where:

• The DNS resolution phase is not controlled by timeout
• The connect phase is controlled by the timeout parameter
• The timeout read phase is not global, and if the server lets the client read too many times, it can cause the client to stall

Connect system calls that block have a default maximum time limit, depending on system configuration; Timeout control of connect can be implemented with “non-blocking CONNECT + SELECT /poll”.

While investigating the cause of this case, a potential DOS attack was found, which was reported to Python authorities and quickly fixed.

If you find it helpful, search leveryd.