Question about performance

Issue #27 new
Anonymous created an issue

I have created a benchmark for two handlers:

class WelcomeHandler(BaseHandler):
    def get(self):
        response = HTTPResponse()
        response.write(json.dumps(['Hello World!']))
        return response


def welcome(request):
    response = HTTPResponse()
    response.write(json.dumps(['Hello World!']))
    return response

The 'class style' is 3 times slower than the 'method style'. Should this be expected?

Comments (14)

  1. PauloC
    # Handlers
    
    from wheezy.http import HTTPResponse
    from wheezy.web.handlers import BaseHandler
    import simplejson as json
    
    class WelcomeHandler(BaseHandler):
        def get(self):
            response = HTTPResponse()
            response.write(json.dumps(['Hello World!']))
            return response
    
    
    def welcome(request):
        response = HTTPResponse()
        response.write(json.dumps(['Hello World!']))
        return response
    
    # App
    
    from wheezy.http import WSGIApplication
    from wheezy.routing import url
    from wheezy.web.middleware import bootstrap_defaults
    from wheezy.web.middleware import path_routing_middleware_factory
    
    
    all_urls = [
        url('1', WelcomeHandler, name='default1'),
        url('2', welcome, name='default2'),
    ]
    
    options = {}
    main = WSGIApplication(
        middleware=[
            bootstrap_defaults(url_mapping=all_urls),
            path_routing_middleware_factory
        ],
        options=options
    )
    
    # Benchmark test
    
    from wheezy.core.benchmark import Benchmark
    from wheezy.http.functional import WSGIClient
    
    def makeRequest1():
        client = WSGIClient(main)
        assert 200 == client.get('/1')
    
    def makeRequest2():
        client = WSGIClient(main)
        assert 200 == client.get('/2')
    
    p = Benchmark((makeRequest1, makeRequest2), 20000)
    
    p.report('benchmark', baselines={
        'test_index': 1.0,
        'test_index': 1.0,
    })
    
  2. Andriy Kornatskyy repo owner

    You didn't publish numbers. Here are results (python2.7):

    MacOS

    baseline throughput change target
      100.0%   15122rps  +0.0% makeRequest1
      107.4%   16241rps  +0.0% makeRequest2
    

    Linux

    baseline throughput change target
      100.0%   24035rps  +0.0% makeRequest1
      110.6%   26585rps  +0.0% makeRequest2
    

    So here I see 7-10% overhead for class style. What extra it does you can see here, which downs to some initialization and dispatch per HTTP method.

    P.S. In your benchmark methods (makeRequest1 and makeRequest1) you can move initialization of WSGIClient above the method scope.

  3. Andriy Kornatskyy repo owner

    Here is what I got initially using pypy-2.5.0-osx64:

    benchmark: 2 x 20000
    baseline throughput change target
      100.0%   30124rps  +0.0% makeRequest1
      196.0%   59055rps  +0.0% makeRequest2
    

    and here is result when I increased a number of iterations ten times:

    benchmark: 2 x 200000
    baseline throughput change target
      100.0%   71613rps  +0.0% makeRequest1
      102.7%   73519rps  +0.0% makeRequest2
    

    So as you can see pypy (JIT) needs more cycles to "warm up".

  4. Andriy Kornatskyy repo owner

    Thanks. Which benchmarking tool did you use?

    I would advise take a look at wrk, wrk2 and/or ab. A tooltip in table says there were 10K requests, I would advise start from 100K at least given the numbers you have in the table. Network is important since it is always present in real world. In your tests you are referring to wheezy... there is wheezy.http and wheezy.web instead (avoid confusion). I think wheezy-py27-uwsgi-1w for concurrency 16 needs to be re-run... it is completely off the neighbors. It is recommended to extend benchmark to use all web servers per web framework (at least bjoern and meinheld since they seem to show best possible results).

    It is very similar to one here.

  5. PauloC

    Weighttp. ab is very old/slow.

    I ran again. Now with 50k instead of 10k (with a warmup of 200k). I ran these tests twice and get the best result.

    Network is always the main bottleneck. For that reason I decide to avoid this variable.

    I renamed to wheezy.http, thanks!

    Thanks for all good feedback, I will always try to improve the tests! Feel free to always send me feedback!

    Best,

  6. Andriy Kornatskyy repo owner

    There are a lot of errors reported for uwsgi. You can improve this by adding the following to configuration:

    add-header = Connection: close
    

    It is recommended to install cython before you install wheezy.*. Use pip, since there is known issue with easy_install.

    Can you update results for wheezy.* with bjoern and meinheld, please?

    Just curious to see how that would look like.

  7. PauloC

    Thanks again!

    • I ran with this new header and it fixes the uWSGI errors, but the performance decrease a lot (I did not upload the data yet). I believe if I configure Nginx + uWSGI all of these errors will be fixed, and then I can use the keep-alive again. Unless you tell me to always use "connection: close" for the Wheezy.http.

    • I believe I can get better results if I put Nginx with Gunicorn too.

    • I ran Gunicorn + Meinheld and it has a good increase of performance (compare with Gunicorn alone), as expected.

    • Wheezy.http + Bjoern was a good combination though! Check the new charts.

    Best,

  8. Andriy Kornatskyy repo owner

    That looks interesting especially a flat line from 128 concurrent requests and up... just curious to know when it breaks. Without it looks a bit "incomplete"... of cause due to network limitation you may never get that practically... so some sort of limit should give an idea of common sense to stop.

    It is surprisingly breaks at 384 for undertow-java case.

  9. Log in to comment