Tuesday, September 18, 2012

Python Fastest Web Framework

What is the fastest web framework for Python? In this post we will examine a trivial 'Hello World!'. See also:
  1. Performance Benchmarks
  2. Code Quality
  3. Template Engines
The application (see source) is written for various Python web frameworks and deployed to uWSGI application container (version 1.9.6 on CPython 2.7.4/3.3.1) and gunicorn (version 0.15.0 on PyPy 1.9). Latest available versions as of this writing (March 15, 2013):
  1. bobo 1.0.0
  2. bottle 0.11.6
  3. cherrypy 3.2.4
  4. circuits 2.1.0
  5. django 1.5.1
  6. flask 0.9
  7. pyramid 1.4
  8. tornado 3.0.1
  9. turbogears 2.2.0
  10. web.py 0.37
  11. web2py 2.1.1
  12. wheezy.web 0.1.365
Let setup few prerequisites to be able run this in clean debian testing installation. 
apt-get install make python-dev python-virtualenv \
    mercurial unzip

# Up TCP connection limits
sysctl net.core.somaxconn=2048
sysctl net.ipv4.tcp_max_syn_backlog=2048
The source code is hosted on bitbucket, let clone it into some directory and setup virtual environment (this will download all necessary package dependencies per framework listed above). 
hg clone https://bitbucket.org/akorn/helloworld
cd helloworld/01-welcome && make env
The make file has a target for each framework and runs particular example in uWSGI, e.g. in order to run django application just issue make django.
The installation for PyPy has own target so issue the following: 
make pypy
In order to run things on PyPy issue the command like this one (this way you specify you need gunicorn server and use PyPy environment): 
make wheezy.web SERVER=gunicorn ENV=pypy-1.9
The throughtput (requests served per second) was captured using apache benchmark (concurrecy level 1K, number of requests 1M) for http://yourserver:8080/welcome: 
               cpython2.7  pypy 1.9  cpython3.3
               uwsgi       gunicorn       uwsgi
bobo                20736     20633           -
bottle              24366     22229       23882
cherrypy             6418      9179           -
circuits             5797      3837           -
django              16007     16848       15965
flask               12483     14399           -
pyramid             23360     20202       24367
tornado             15861     17265       13825
turbogears           2764      5808           -
web.py               5023         -           -
web2py               4065      2769           -
wheezy.web          24703     22323       24858
wsgi                24938     23272       24955
The benchmark results above 22K are not reliable due to hardware limitations.

Isolated Benchmark

In order to provide more reliable benchmark I get rid of application server and network boundary. As a result I simulated a valid WSGI request and isolated calls just to framework alone (the source code is here). Here are raw numbers: 
cpython 2.7
              msec    rps  tcalls  funcs
bobo*         9414  10622     116     65
bottle        2832  35308      65     32
cherrypy*    54320   1841     600    165
circuits*   130650    765     504    112
django       13484   7416     144     75
flask        18861   5302     207    106
pyramid       5595  17875      65     48
tornado      15068   6636     201     66
turbogears* 301980    331    1706    331
web.py*     595932    168    2191     65
web2py      153727    651     417    143
wheezy.web    1793  55786      25     23
wsgi           281 355255       8      8

pypy 1.9
              msec    rps  tcalls  funcs
bobo*         1884  53076     114     64
bottle         803 124559      63     32
cherrypy*    53630   1864     652    185
circuits*    89780   1114     509    112
django        3395  29456     138     73
flask        10273   9735     215    110
pyramid       1819  54990      91     53
tornado*      3465  28864     176     62
turbogears* 275830    363    1705    347
web.py*          -     29   12868     73
web2py      189691    527     562    177
wheezy.web     475 210341      26     24
wsgi           287 349001       8      8

cpython 3.3
              msec    rps  tcalls  funcs
bobo            not installed
bottle        4377  22848      79     41
cherrypy        not installed
django       13572   7368     141     74
flask           not installed
pyramid       6611  15125      87     53
tornado      18331   5455     220     74
turbogears      not installed
webpy           not installed
web2py          not installed
wheezy.web    1968  50818      27     25
wsgi           378 264275      10     10
msec - a total time taken in milliseconds, rps - requests processed per second, tcalls - total number of call made by corresponding web framework, funcs - a number of unique functions used.
ATTENTION: The web frameworks marked with * (asterisk) experience memory leaks in this test.

Environment Specification
  • Client: Intel Core 2 Quad CPU Q6600 @ 2.40GHz × 4, Kernel 3.2.41-2 i686
  • Server: Intel Xeon CPU X3430 @ 2.40GHz x 4, Kernel 3.2.41-2 amd64, uwsgi 1.9.6
  • Debian Testing, Python 2.7.4, LAN 1 Gb
Python has a number of web frameworks. A trivial application gives you an idea where particular web framework stands in terms of performance and internal effectivity.
Posted by Andriy Kornatskyy at 3:00 PM
Labels: , , , ,

40 comments :

  1. Philipp HagemeisterSeptember 25, 2012 at 2:22 PM

    Great benchmark! For someone who's not familiar with all the frameworks, it would be great if you could link the list items to the respective benchmarks.

    ReplyDelete
  2. UnknownSeptember 25, 2012 at 5:00 PM

    Something is wrong here. Can you post the web2py code you are running? Did you remove the scaffolding models (they do lots of authentication logic)? Did you run it with -N (to disable background cron processes, a web2py only feature)? Are you using a template (if so are you using a template for other frameworks)? In web2py session handling is always on, did you enable sessions in other frameworks? Can you post any code that can will enable to reproduce your results at least for web2py and one of the other frameworks?

    ReplyDelete
    Replies
    1. Andriy KornatskyySeptember 25, 2012 at 5:14 PM

      The link to the source is in the post, as well as how to run it.

      Delete
    2. Andriy KornatskyySeptember 25, 2012 at 5:22 PM

      The situation is actually even worse... during test I have noticed huge memory leak.

      Delete
    3. Andriy KornatskyySeptember 26, 2012 at 4:16 PM

      I have rebuild environment and re-run web2py test. The problem is gone, no memory leaks. There were no need to change anything.

      Delete
    4. UnknownOctober 21, 2012 at 7:18 PM

      Thank you Andriy for updating the benchmarks. For readers out there, the web2py benchmark still includes session handling (session is created but although sessions is not saved) and internationalization handling (lookup per-request translation file to match requested accept-language).

      Delete
    5. Jimmy KaneFebruary 20, 2014 at 10:43 PM

      Sorry but is the chart updated for web2.py?

      Delete
    6. Andriy KornatskyyFebruary 21, 2014 at 8:07 AM

      The benchmark raw results and chart correspond to latest version of each web framework at the time the post was updated.

      Delete
  3. UnknownSeptember 25, 2012 at 9:21 PM

    Hello Andriy. We are looking into this. You have at least one problem. In the web2py example you must add session.forget() otehrwise your app is creating a session file at every request. This mans you have a huge number of session files and access gets slower and slower. Disk access alone can account for the numbers you get. We are discussing the memory leak since many of us have never seen the problem. A user believe it to be a uwsgi configuration issue (https://groups.google.com/d/msg/web2py/Yrtrj3BSFl4/_06tIzqJNzQJ).

    ReplyDelete
    Replies
    1. Andriy KornatskyySeptember 25, 2012 at 9:31 PM

      You have to be kidding, right? I am using a `hello world` application per examples provided by corresponding frameworks... including web2py. Let collaborate this via email or supply a patch so it doesn't bounce back and forward. Thanks.

      Delete
    2. Andriy KornatskyySeptember 25, 2012 at 9:38 PM

      uWSGI configuration is taken from here http://projects.unbit.it/uwsgi/wiki/Example. That most likely a source of all web2py deployments using uWSGI...

      Delete
    3. UnknownSeptember 26, 2012 at 4:55 PM

      Thank you. :-)

      Delete
    4. _sithu_November 17, 2013 at 2:13 AM

      +1 Massimo - nice way to handle a rather rude/ defensive response.

      Delete
  4. UnknownSeptember 25, 2012 at 9:54 PM

    For readers out there. A "hello world" app in web2py is not a "hello word" app since web2py does not follow "explicit is better then implicit". Out of the box it is optimized for rapid prototyping and not for speed. Web2py does lots of stuff out of the box whether you like it or not. Anyway, the main problem here is probably saving sessions, and there is a way to disable it.

    ReplyDelete
    Replies
    1. Andriy KornatskyySeptember 25, 2012 at 10:02 PM

      and you understand the risk web2py application (that use sessions) experience in production...

      Delete
    2. FisaOctober 17, 2012 at 4:05 AM

      Your argument isn't valid Massimo. You want web2py to be benchmarked disabling those features because it scores low when they are enabled, but django has those features enabled too and scores high anyway.
      For a "hello world" benchmark the framework should be tested with no fine tunning. That's the purpose of this kind of benchmarks.

      Delete
    3. Andriy KornatskyyOctober 17, 2012 at 9:42 AM

      If you check the source you will notice that I tried to pick a minimal possible application stack for given web framework: turned off debug, disabled logging, removed `unnecessary` middleware, etc. If I missed anything... please just let me know.

      Delete
  5. UnknownSeptember 26, 2012 at 6:26 AM

    Yes. DoS. In fact, old web2py was always creating a session file but new web2py only saves session if it changes and is not empty. Let's find out why your benchmarks do not agree with yours. Perhaps is not the problem.

    ReplyDelete
  6. AlecTaylorSeptember 27, 2012 at 7:07 PM

    I think you mean bobo 0.2?

    ReplyDelete
  7. Daniel SokolowskiOctober 4, 2012 at 4:11 PM

    Andriy, Thank you for doing the research and sharing this. It is very interesting to see pypy not living up to it's benchmarking hype. Possible extension to this test if you inclined to do so would be to include Apache + mod_wsgi stack test, and other languages frameworks, Haskell; I know the latter is very unlikely :).

    ReplyDelete
    Replies
    1. Andriy KornatskyyOctober 5, 2012 at 7:27 PM

      I personally believe pypy is the future. If you compare gunicorn on cpython (make wsgi SERVER=gunicorn) vs pypy (make wsgi SERVER=gunicorn ENV=pypy-1.9) you will see cpython is not that good... In my believe uwsgi is the right application server to be used with cpython, while gunicorn for pypy (I haven't chance to see uwsgi+pypy since it is not a primary direction of its development team). What relates to apache mod_wsgi: it all way good except performance, so I stick with uwsgi to be able to see the difference. It is actually a challenge to benchmark anything across other languages since many factors applies... I tried answer it for python.

      Delete
  8. JamesOctober 17, 2012 at 10:31 AM

    You forgot circuits.web (1) :/

    cheers
    James

    [1] http://pypi.python.org/pypi/circuits/

    ReplyDelete
  9. dAnjouOctober 17, 2012 at 12:39 PM

    Sorry mate but these "benchmarks" are so damn pointless.

    ReplyDelete
    Replies
    1. Andriy KornatskyyOctober 17, 2012 at 1:01 PM

      These benchmarks give you an idea:
      (1) where web frameworks stand in terms of internal effectivity running a simple thing
      (2) how various python implementations handle it
      (3) all are good (and sufficient for their communities)

      Delete
  10. JamesOctober 17, 2012 at 3:36 PM

    I think these results are very useful. It's certanily given me some insight as to some aspects of circuits and circuits.web I can improve upon. It's also a good measure of the quality (or lack thereof) of some of the frameworks and server implementations (even tornado doesn't suffer from memory leaks).

    --JamesMills / prologic

    ReplyDelete
  11. Alessandro MolinaOctober 31, 2012 at 11:45 AM

    I would be curious to see how the recently released TurboGears2.2 behaves related to the other frameworks. Some of the performance improvements that has been done in the 2.3 branch have been backported to 2.2 so there has been a sensible improvement over the past versions.

    Let me know if you need any help setting up a benchmark for TG, I would be glad to help.

    ReplyDelete
    Replies
    1. Andriy KornatskyyOctober 31, 2012 at 2:01 PM

      accepted

      Delete
    2. Alessandro MolinaNovember 3, 2012 at 1:34 PM

      Thank your for your test!
      Results are interesting, I'm going to try to replicate them using your configurations, how did you check for the memory leaks?

      I saw that the benchmarks got conducted with minimal setups for the other frameworks, have you set full_stack=False in config/middlware.py and disabled in config/app_cfg.py most optional things like:

      base_config.use_toscawidgets2 = False
      base_config.use_toscawidgets = False
      base_config.i18n_enabled = False
      base_config.disable_request_extensions = True
      base_config.serve_static = False

      As the speedup work is happening all on 2.3 branch we didn't have any compared benchmark for 2.2 and I was curious to see how it behaved.

      Delete
    3. Andriy KornatskyyNovember 3, 2012 at 1:55 PM

      I have isolated turbogears and noticed constant residential memory grow.

      The Makefile target creates project using paster quistart, than few files are copied (take look at source code). The ini file was updated per production configuration comments.

      Delete
  12. Michael WatkinsApril 15, 2013 at 10:38 PM

    Not sure why this older article showed up in my feed reader this morning but its appearance caused me to finally install and run the benchmarks. The takeaway for me was... choose the framework that best fits your mind or task, hopefully both at the same time. They all perform about the same when run with uwsgi. I mostly use DurusWorks which is an evolutionary brother to Quixote, one of the older web frameworks out there. Neither has a large community of users.

    ReplyDelete
    Replies
    1. Andriy KornatskyyApril 15, 2013 at 10:52 PM

      The post has been updated due to community request for django python3. The difference between frameworks well seen in three others: routing, reverse urls and caching.

      Delete
  13. Michael WatkinsApril 15, 2013 at 10:52 PM

    PS, meant to thank you for putting together the bundle of apps and tests. Regardless of my conclusion it was still interesting to go through them and I appreciate the effort.

    ReplyDelete
  14. AnonymousMay 20, 2013 at 7:34 AM

    In the benchmarks what app server does wsgi refer to?

    ReplyDelete
    Replies
    1. Andriy KornatskyyMay 20, 2013 at 10:08 AM

      wsgi test (as well as all others) is hosted in uwsgi for python2.7/3.3 and gunicorn for pypy.

      Delete
  15. Alexander TAugust 7, 2013 at 11:54 AM

    Thank you so much for doing this benchmark. I found out the hard way after writing an app that the framework wouldn't scale above 500 RPS regardless of settings. Talk about dissapointment! It's crazy that so many frameworks are so far away from pure WSGI perf.

    ReplyDelete
  16. RickardNovember 12, 2013 at 2:32 PM

    Have you tested the new Falcon framework? It is said to be very fast. Would be interesting to see how it compares with the others.
    http://falconframework.org/

    ReplyDelete
    Replies
    1. AnonymousNovember 12, 2013 at 8:50 PM

      http://faruk.akgul.org/blog/python-web-frameworks-benchmark/

      Delete

Subscribe to: Post Comments ( Atom )