This project is about benchmarking three different Python API frameworks based on simple image processing task. The basic concept is the APIs consumes either image file in the form of base64 encoding or multipart-data or just a image url, and returns the corresponding pure black and white image to the client.
uvicorn app:app --reload # --reload to automatically reflect code changes
uvicorn app:app --reload --log-level critical # to suppress the console log
python flask_client.py
pytest
Once you run the server, please goto http://127.0.0.1:5000/ in order to check the results visually. This API server retrieves random dog images using Dog API and converts it into pure black and white.
uvicorn app:app --reload # --reload to automatically reflect code changes
uvicorn app:app --reload --log-level critical # to suppress the console log
python fast_client.py
pytest
python -m app.py -H localhost -P 8000 package.module:init_func
python client_aiohttp.py
pytest
-
locust -f locustfile.py --headless --host http://127.0.0.1:5000 -u 2000 -r 50 -t 30s --csv reports/imageurl ImageUrlUser locust -f locustfile.py --headless --host http://127.0.0.1:5000 -u 2000 -r 50 -t 30s --csv reports/base64 Base64User locust -f locustfile.py --headless --host http://127.0.0.1:5000 -u 2000 -r 50 -t 30s --csv reports/multipart MultipartUser
or, just run ./benchmark.sh
for 5-trial run
Note: This currently works for imageurl
endpoint only
-
Flask API
wrk http://127.0.0.1:5000/magic/imageurl -t12 -c400 -d30s -s lua_scripts/post.lua
-
Fast API
wrk http://127.0.0.1:8000/magic/imageurl -t12 -c400 -d30s -s lua_scripts/post.lua
-
aioHTTP
wrk http://0.0.0.0:8080/magic/imageurl -t12 -c400 -d30s -s lua_scripts/post_aiohttp.lua
Initially, aLL API endpoints were tested for performance using wrk HTTP benchmarking tool.
wrk tool is based on C and requires in-depth knowledge of Lua to pass the base64 encoded images as multipart/data. I faced great difficulty in passing base64 encoding image or multipart/form-data as a payload through wrk. Hence, I switched to locust, which is a python based HTTP benchmarking tool. It made the task very much easier since it utilizes the requests
library. However, it should be noted that wrk is extremely fast compared to locust.
Parameters for load testing on locust*:
-
Number of users = 2000
-
User spawning rate = 50/second
-
Duration of test = 30s