Of Python’s many HTTP clients, the best known are Requests, AIoHTTP, and HTTPX. Requests can only send synchronous requests without the help of other third-party libraries; Aiohttp can only send asynchronous requests; HTTPX can send both synchronous and asynchronous requests.
Synchronous request means that in a single-process, single-threaded code, after a request is made, the next request cannot be made until the result is returned. The so-called asynchronous request refers to the code in a single process, single thread, after a request, waiting for the site to return the result of the time, can continue to send more requests.
Today we’ll take a shallow look at the performance of the three libraries in terms of sending multiple POST requests.
Test using the HTTP service address for http://122.51.39.219:8000/query, send it to the format of the POST request as shown in the figure below:
Return {“success”: false} or {“success”: true} if the request is sent in a TS field whose date is more than 10 days from today.
First let’s see what happens when we send the request once from each client using the same parameters.
Send a request
requests
import requests
resp = requests.post('http://122.51.39.219:8000/query',
json={'ts': 'the 2020-01-20 13:14:15'}).json()
print(resp)
Copy the codeThe operating effect is shown in the figure below:
httpx
Sending a synchronization request using HTTPX:
import httpx
resp = httpx.post('http://122.51.39.219:8000/query',
json={'ts': 'the 2020-01-20 13:14:15'}).json()
print(resp)
Copy the codeHTTPX synchronizes requests code 99% of the time, just change Requests to HTTPX to work. As shown below:
To send an asynchronous request using HTTPX:
import httpx
import asyncio
async def main(a):
async with httpx.AsyncClient() as client:
resp = await client.post('http://122.51.39.219:8000/query',
json={'ts': 'the 2020-01-20 13:14:15'})
result = resp.json()
print(result)
asyncio.run(main())
Copy the codeThe operating effect is shown in the figure below:
aiohttp
import aiohttp
import asyncio
async def main(a):
async with aiohttp.ClientSession() as client:
resp = await client.post('http://122.51.39.219:8000/query',
json={'ts': 'the 2020-01-20 13:14:15'})
result = await resp.json()
print(result)
asyncio.run(main())
Copy the codeThe operating effect is shown in the figure below:
Aiohttp code is 90% identical to HTTPX asynchronous code, except AsyncClient is replaced with ClientSession. Additionally, when using HTTPX, you are sending the request while you await client.post. But when using aioHTTP, the request is only actually sent when aWIat resp.json() is used.
Send 100 requests
We now randomly generate a date 5-15 days from today and send it to the HTTP interface. False if the date is more than 10 days from today, True if it is less than or equal to 10 days from today.
We send 100 requests and calculate the total time.
requests
In an article the other day, we mentioned that using requests. Post creates a new connection each time, which is slow. If a Session is initiated first, requests will remain connected, greatly speeding up requests. So in this assessment, we tested both scenarios.
Disconnecting
import random
import time
import datetime
import requests
def make_request(body):
resp = requests.post('http://122.51.39.219:8000/query', json=body)
result = resp.json()
print(result)
def main(a):
start = time.time()
for _ in range(100):
now = datetime.datetime.now()
delta = random.randint(5.15)
ts = (now - datetime.timedelta(days=delta)).strftime('%Y-%m-%d %H:%M:%S')
make_request({'ts': ts})
end = time.time()
print(F 'Send 100 requests, time:{end - start}')
if __name__ == '__main__':
main()
Copy the codeThe operating effect is shown in the figure below:
It took 2.7 seconds to send 100 requests without maintaining a connection
Stay connected
Modify the code slightly to send requests using the same Session:
import random
import time
import datetime
import requests
def make_request(session, body):
resp = session.post('http://122.51.39.219:8000/query', json=body)
result = resp.json()
print(result)
def main(a):
session = requests.Session()
start = time.time()
for _ in range(100):
now = datetime.datetime.now()
delta = random.randint(5.15)
ts = (now - datetime.timedelta(days=delta)).strftime('%Y-%m-%d %H:%M:%S')
make_request(session, {'ts': ts})
end = time.time()
print(F 'Send 100 requests, time:{end - start}')
if __name__ == '__main__':
main()
Copy the codeThe operating effect is shown in the figure below:
100 requests were sent and requests remained connected in 1.4 seconds
httpx
Synchronous mode
The code is as follows:
import random
import time
import datetime
import httpx
def make_request(client, body):
resp = client.post('http://122.51.39.219:8000/query', json=body)
result = resp.json()
print(result)
def main(a):
session = httpx.Client()
start = time.time()
for _ in range(100):
now = datetime.datetime.now()
delta = random.randint(5.15)
ts = (now - datetime.timedelta(days=delta)).strftime('%Y-%m-%d %H:%M:%S')
make_request(session, {'ts': ts})
end = time.time()
print(F 'Send 100 requests, time:{end - start}')
if __name__ == '__main__':
main()
Copy the codeThe operating effect is shown in the figure below:
Sending 100 requests takes about 1.5 seconds in HTTPX synchronous mode.
Asynchronous mode
The code is as follows:
import httpx
import random
import datetime
import asyncio
import time
async def request(client, body):
resp = await client.post('http://122.51.39.219:8000/query', json=body)
result = resp.json()
print(result)
async def main(a):
async with httpx.AsyncClient() as client:
start = time.time()
task_list = []
for _ in range(100):
now = datetime.datetime.now()
delta = random.randint(5.15)
ts = (now - datetime.timedelta(days=delta)).strftime('%Y-%m-%d %H:%M:%S')
req = request(client, {'ts': ts})
task = asyncio.create_task(req)
task_list.append(task)
await asyncio.gather(*task_list)
end = time.time()
print(F 'Send 100 requests, time:{end - start}')
asyncio.run(main())
Copy the codeThe operating effect is shown in the figure below:
Sending 100 requests takes about 0.6 seconds in HTTPX asynchronous mode.
aiohttp
The test code is as follows:
import aiohttp
import random
import datetime
import asyncio
import time
async def request(client, body):
resp = await client.post('http://122.51.39.219:8000/query', json=body)
result = await resp.json()
print(result)
async def main(a):
async with aiohttp.ClientSession() as client:
start = time.time()
task_list = []
for _ in range(100):
now = datetime.datetime.now()
delta = random.randint(5.15)
ts = (now - datetime.timedelta(days=delta)).strftime('%Y-%m-%d %H:%M:%S')
req = request(client, {'ts': ts})
task = asyncio.create_task(req)
task_list.append(task)
await asyncio.gather(*task_list)
end = time.time()
print(F 'Send 100 requests, time:{end - start}')
asyncio.run(main())
Copy the codeThe operating effect is shown in the figure below:
Sending 100 requests takes about 0.3 seconds using AIOHTTP
Send 1000 requests
Since Request stays connected faster than it does not, we will only test it by staying connected. And do not print the returned results.
requests
The operating effect is shown in the figure below:
Sending 1000 requests took about 16 seconds
httpx
Synchronous mode
The operating effect is shown in the figure below:
Sending 1000 requests takes about 18 seconds in HTTPX synchronous mode
Asynchronous mode
The operating effect is shown in the figure below:
Send 1000 requests, HTTPX asynchronous mode takes about 5 seconds
aiohttp
The operating effect is shown in the figure below:
Aiohttp takes about 4 seconds to send 1000 requests
conclusion
If you only send a few requests. The code is easiest to use in Requests or HTTPX synchronization mode.
If you are sending a lot of requests, but some places are sending synchronous requests and some places are sending asynchronous requests, then using HTTPX is the easiest.
If you’re sending a lot of requests, and the faster the better, using AIOHTTP is the fastest.
This review article is a very shallow review that only considers the speed of requests. If you are going to use it in a production environment, then you can do more experiments to see if it is suitable for your actual use.
