Clone wiki

nxweb / Benchmarks

NXWEB 3.0 Benchmarks

NXWEB v3 is even faster than v2.

Results are in thousands requests per second. F = failed. NA = not supported. Measured by httpress on 4-core CPU (without AES-NI).

TestNXWEBG-WANlibeventmicrohttpdmongoosenginx
1. hello 100 ka200 / 12114430 / 69132190141
2. hello 10051 / 424115 / 32133441
3. hello 1000 ka160 / 11513021 / 43130180124
4. hello 100046 / 383814 / 30123540
5. hello 10000 ka115 / 8410323 / 40116119108
5.1. real concurrency9500-100009500-1000010000600-10001500-17004000-7000
5.2. memory footprint28Mb105Mb---4x15Mb
6. hello 1000038 / 343314 / 2792029
7. file 2.3K ka133NANANA598
7.1. file 2.3K ka cached145120NANANANA
8. file 2.3K42NANANA1239
8.1. file 2.3K cached4333NANANANA
9. file 100K ka3615NANA3.632
10. file 100K2312NANA3.521
11. file 2.1M ka3.20.7NANA0.62.6
12. file 2.1M2.30.6NANA0.52.0

Test descriptions:

  1. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, keep-alive (httpress -c 100 -n 1000000 -t 4 -k)
  2. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, no keep-alive (httpress -c 100 -n 500000 -t 4)
  3. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, keep-alive (httpress -c 1000 -n 1000000 -t 4 -k)
  4. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, no keep-alive (httpress -c 1000 -n 500000 -t 4)
  5. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, keep-alive (httpress -c 10000 -n 1000000 -t 4 -k)
    1. httpress tool allows to calculate real concurrency, the number of actually active connections participating in test
    2. 10K concurrent connections take a lot of RAM. Here you can see how much
  6. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, no keep-alive (httpress -c 10000 -n 500000 -t 4)
  7. Disk file 2.3 KiB; 400 concurrent connections, keep-alive
    1. Disk file 2.3 KiB; 400 concurrent connections, keep-alive, using memory cache
  8. Disk file 2.3 KiB; 400 concurrent connections, no keep-alive
    1. Disk file 2.3 KiB; 400 concurrent connections, no keep-alive, using memory cache
  9. Disk file 100 KiB; 400 concurrent connections, keep-alive
  10. Disk file 100 KiB; 400 concurrent connections, no keep-alive
  11. Disk file 2.1 MiB; 400 concurrent connections, keep-alive
  12. Disk file 2.1 MiB; 400 concurrent connections, no keep-alive

SSL Benchmarks

SSL INFO: ECDHE_RSA_AES_256_CBC_SHA1
- Protocol: TLS1.0
- Key Exchange: ECDHE-RSA
- Ephemeral ECDH using curve SECP256R1
- Cipher: AES-256-CBC
- MAC: SHA1
- Compression: NULL
- Certificate Type: X.509
- Certificate Info: RSA key 2048 bits, signed using RSA-SHA1

All tests done with 100 concurrent connections. Numbers are requests per second.

Eg. httpress -c100 -n4000 -t4 -z 'NORMAL:-CIPHER-ALL:+AES-256-CBC:-VERS-TLS-ALL:+VERS-TLS1.0'

Testnxwebnginx
1. file 2.3K ka2600023000
2. file 2.3K490430
3. file 100K ka13001100
4. file 100K360330

Server notes:

  • NXWEB: first measurement is for inprocess handler, second is for inworker handler
  • G-WAN: v.3.1.24 x64. Performance of this server is good, it is also very convenient to use - it autocompiles on the fly all source files put in special directory; I found it quite unstable though: segfaults on some static files, hangs on its own sample scripts; another big disadvantage is that it is closed-source
  • libevent: standard implementation of evhttp is single-threaded; second measurement in table is received from custom multi-threaded version (several instances of evhttp launched in 4 threads, listening to the same socket); not sure if this hack affects stability
  • GNU microhttpd: MHD_USE_SELECT_INTERNALLY | MHD_USE_POLL, 4 thread-pool
  • mongoose: configured with 2500 threads in pool; uses thread per connection model, which is certainly not very efficient and takes a lot of memory
  • nginx: v.1.1.12 compiled from source; using optimized setup; sendfile turned on, aio/directio turned off; location /hello { return 200 '<p>Hello, world!</p>'; }

NXWEB 2.0 Benchmarks

NXWEB v2 does not depend on libev anymore. It's been significantly optimized for speed and memory usage over v1.

Results are in thousands requests per second. F = failed. NA = not supported. Measured by httpress on 4-core CPU.

TestNXWEBG-WANlibeventmicrohttpdmongoosenginx
1. hello 100 ka165 / 12111530 / 69132190132
2. hello 10051 / 413815 / 32133428
3. hello 1000 ka135 / 10211521 / 43130180120
4. hello 100046 / 383614 / 30123527
5. hello 10000 ka105 / 8410323 / 40116119100
5.1. real concurrency100009500-1000010000600-10001500-17004000-10000
6. hello 1000038 / 343314 / 2792022
7. file 2.3K ka108100NANA590
8. file 2.3K4238NANA1222
9. file 100K ka3431NANA3.631
10. file 100K2020NANA3.513
11. file 2.1M ka3.22.8NANA0.62.6
12. file 2.1M2.22.0NANA0.52.0

Test descriptions:

  1. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, keep-alive (httpress -c 100 -n 1000000 -t 4 -k)
  2. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, no keep-alive (httpress -c 100 -n 500000 -t 4)
  3. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, keep-alive (httpress -c 1000 -n 1000000 -t 4 -k)
  4. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, no keep-alive (httpress -c 1000 -n 500000 -t 4)
  5. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, keep-alive (httpress -c 10000 -n 1000000 -t 4 -k)
    1. httpress tool allows to calculate real concurrency, the number of actually active connections participating in test
  6. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, no keep-alive (httpress -c 10000 -n 500000 -t 4)
  7. Disk file 2.3 KiB; 400 concurrent connections, keep-alive
  8. Disk file 2.3 KiB; 400 concurrent connections, no keep-alive
  9. Disk file 100 KiB; 400 concurrent connections, keep-alive
  10. Disk file 100 KiB; 400 concurrent connections, no keep-alive
  11. Disk file 2.1 MiB; 400 concurrent connections, keep-alive
  12. Disk file 2.1 MiB; 400 concurrent connections, no keep-alive

Server notes:

  • NXWEB: first measurement is for inprocess handler, second is for inworker handler
  • G-WAN: v.2.10.15 x32. Performance of this server is good, it is also very convenient to use - it autocompiles on the fly all source files put in special directory; I found it quite unstable though: segfaults on some static files, hangs on its own sample scripts; another big disadvantage is that it is closed-source
  • libevent: standard implementation of evhttp is single-threaded; second measurement in table is received from custom multi-threaded version (several instances of evhttp launched in 4 threads, listening to the same socket); not sure if this hack affects stability
  • GNU microhttpd: MHD_USE_SELECT_INTERNALLY | MHD_USE_POLL, 4 thread-pool
  • mongoose: configured with 2500 threads in pool; uses thread per connection model, which is certainly not very efficient and takes a lot of memory
  • nginx: v.1.0.5 from Ubuntu repo; using optimized setup; sendfile turned on, aio/directio turned off; location /hello { return 200 '<p>Hello, world!</p>'; }

NXWEB 1.0 Benchmarks

Results are in thousands requests per second. F = failed. NA = not supported. Measured by httpress on 4-core CPU.

TestNXWEBG-WANlibeventmicrohttpdmongoosenginx
1. hello 100 ka136 / 9711530 / 69132190132
2. hello 10042 / 373815 / 32133428
3. hello 1000 ka115 / 9311521 / 43130180120
4. hello 100041 / 363614 / 30123527
5. hello 10000 ka90 / 7610823 / 40116119100
5.1. real concurrency100003000-500010000600-10001500-17009500-10000
6. hello 1000036 / 343314 / 2792022
7. file 2.3K ka87100NANA590
8. file 2.3K3838NANA1222
9. file 100K ka3131NANA3.631
10. file 100K2020NANA3.513
11. file 2.1M ka3.02.8NANA0.62.6
12. file 2.1M2.22.0NANA0.52.0

Test descriptions:

  1. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, keep-alive (httpress -c 100 -n 1000000 -t 4 -k)
  2. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, no keep-alive (httpress -c 100 -n 500000 -t 4)
  3. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, keep-alive (httpress -c 1000 -n 1000000 -t 4 -k)
  4. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, no keep-alive (httpress -c 1000 -n 500000 -t 4)
  5. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, keep-alive (httpress -c 10000 -n 1000000 -t 4 -k)
    1. httpress tool allows to calculate real concurrency, the number of actually active connections participating in test
  6. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, no keep-alive (httpress -c 10000 -n 500000 -t 4)
  7. Disk file 2.3 KiB; 400 concurrent connections, keep-alive
  8. Disk file 2.3 KiB; 400 concurrent connections, no keep-alive
  9. Disk file 100 KiB; 400 concurrent connections, keep-alive
  10. Disk file 100 KiB; 400 concurrent connections, no keep-alive
  11. Disk file 2.1 MiB; 400 concurrent connections, keep-alive
  12. Disk file 2.1 MiB; 400 concurrent connections, no keep-alive

Server notes:

  • NXWEB: first measurement is for inprocess handler, second is for inworker handler
  • G-WAN: performance of this server is good, it is also very convenient to use - it autocompiles on the fly all source files put in special directory; I found it quite unstable though: segfaults on some static files, hangs on its own sample scripts; another big disadvantage is that it is closed-source
  • libevent: standard implementation of evhttp is single-threaded; second measurement in table is received from custom multi-threaded version (several instances of evhttp launched in 4 threads, listening to the same socket); not sure if this hack affects stability
  • GNU microhttpd: MHD_USE_SELECT_INTERNALLY | MHD_USE_POLL, 4 thread-pool
  • mongoose: configured with 2500 threads in pool; uses thread per connection model, which is certainly not very efficient and takes a lot of memory
  • nginx: v.1.0.5; using optimized setup; sendfile turned on, aio/directio turned off; location /hello { return 200 '<p>Hello, world!</p>'; }

Old Benchmarks

The following measurements were made by ApacheBench (ab) tool. This tool is single-threaded, which imposes limits on maximum throughput.

Other notable difference was that nginx was not configured properly for multi-core hardware. Therefore demonstrated poor performance.

Results are in thousands requests per second. F = failed. NA = not supported.

TestNXWEBG-WANlibeventmicrohttpdmongoosenginx
1. hello 100 ka73 / 696430 / 75NA6037
2. hello 10017 / 171719 / 15131819
3. hello 1000 ka60 / 585822 / 44NA5133
4. hello 100014 / 141616 / 14121615
5. hello 10000 ka54 / 5355F / FNAFF
6. hello 100007 / 712F / FF7F
7. file 2.3K ka5455NANA1025
8. file 2.3K1416NANA1117
9. file 100K ka1115NANA615
10. file 100K88NANA58
11. file 2.1M ka0.60.6NANA0.30.6
12. file 2.1M0.60.6NANA0.30.6

Test descriptions:

  1. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, keep-alive (ab -c 100 -n 320000 -k)
  2. Minimal handler returning '<p>Hello, world!</p>'; 100 concurrent, no keep-alive (ab -c 100 -n 32000)
  3. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, keep-alive (ab -c 1000 -n 320000 -k)
  4. Minimal handler returning '<p>Hello, world!</p>'; 1000 concurrent, no keep-alive (ab -c 1000 -n 32000)
  5. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, keep-alive (ab -c 10000 -n 400000 -k)
  6. Minimal handler returning '<p>Hello, world!</p>'; 10000 concurrent, no keep-alive (ab -c 10000 -n 40000)
  7. Disk file 2.3 KiB; 400 concurrent connections, keep-alive
  8. Disk file 2.3 KiB; 400 concurrent connections, no keep-alive
  9. Disk file 100 KiB; 400 concurrent connections, keep-alive
  10. Disk file 100 KiB; 400 concurrent connections, no keep-alive
  11. Disk file 2.1 MiB; 400 concurrent connections, keep-alive
  12. Disk file 2.1 MiB; 400 concurrent connections, no keep-alive

I was under the impression that it was performance of benchmarking program (ApacheBench) that was limiting factor in the last two tests with large files. Results for NXWEB, G-WAN and nginx are just too close.

Server notes:

  • NXWEB: first measurement is for inprocess handler, second is for inworker handler
  • G-WAN: performance of this server is very good, it is also very convenient to use - it autocompiles on the fly all source files put in special directory; I found it quite unstable though: segfaults on some static files, hangs on its own sample scripts; another big disadvantage is that it is closed-source
  • libevent: standard implementation of evhttp is single-threaded; second measurement in table is received from custom multi-threaded version (several instances of evhttp launched in 4 threads, listening to the same socket); not sure if this hack affects stability
  • GNU microhttpd: don't seem to support keep-alive (at least with ab); awkward API
  • mongoose: configured with 2500 threads in pool; uses thread per connection model, which is certainly not very efficient and takes a lot of memory
  • nginx: v.1.0.5; using default setup; in "hello" tests request /.htaccess file, which gives an error [hopefully] without invlving filesystem; no aio/directio optimizations turned on; only sendfile turned on

Updated