Commits

Fred T-H committed d9b605a

Adding first benchmark results

Comments (0)

Files changed (1)

-= Chut Benchmarks =
-
-Two kinds of benchmarks need to be run. The first kind is a brutish stress test trying to see how well a loaded node can fare when being spammed with messages. The second kind is a more realistic benchmark. This second kind should instead focus on emulating real conversations - each user listening for longer periods of time and sending messages less frequently.
-
-== Stress Benchmark ==
-Insert data here.
-
-== Realistic benchmark ==
-Not yet done.
+== Chut Benchmarks ==
+
+Three  kinds of benchmarks need to be run. 
+# The first kind is a brutish stress test trying to see how well a loaded node without a server in front can fare when being spammed with messages.
+# The second kind is a more realistic benchmark, still without a server. This benchmark should instead focus on emulating real conversations - each user listening for longer periods of time and sending messages least frequently.
+# The third kind of benchmark should be a stress test with the server up to see how it impacts Chut, and to help configure it better.
+
+
+=== Stress Benchmark ===
+The [[http://bitbucket.org/ferd/chut/src/tip/src/benchmark_stress.erl|first benchmark]] is all about stress tests.
+
+Rather than using a regular benchmark where a sea of clients are connected and messaging anyone at random, this test intends to simulate what could be considered a 'social group': the whole user base is divided into //x// groups containing //y// users each. These users message each other for a given period of time.
+
+==== Results ====
+Ran on:
+* HP dv9625ca
+* Ubuntu 8.04 (hardy) 32 bits
+* 2GB Ram
+* AMD Turion 64 X2 Mobile Technology TL-60 (2GHz)
+
+Legend:
+* **G**: number of groups.
+* **CPG**: number of clients per group
+* **T**: time allowed to message and receive in seconds.
+
+The first few rounds are done using text messages. The text message is [[http://en.wikipedia.org/wiki/Speak_to_Me#Spoken_parts|//"I've been mad for fucking years—absolutely years"//
+]]
+{{{
+#!text
+
+
+G:10    CPG:5   T:10
+Sent: 528704
+Received: 506096
+Average sent per second: 52870.4
+Average received per second: 50609.6
+Average sent per group: 52870.4
+Average received per group: 50609.6
+Average sent per client: 10574.08
+Average received per client: 10121.92
+Average sent per client per second: 1057.408
+Average received per client per second: 1012.192
+}}}
+
+{{{
+#!text
+
+G:10    CPG:10  T:10
+Sent: 457730
+Received: 414082
+Average sent per second: 45773.0
+Average received per second: 41408.2
+Average sent per group: 45773.0
+Average received per group: 41408.2
+Average sent per client: 4577.3
+Average received per client: 4140.82
+Average sent per client per second: 457.73
+Average received per client per second: 414.08
+}}}
+
+{{{
+#!text
+
+G:100   CPG:10  T:10
+Sent: 564092
+Received: 420320
+Average sent per second: 56409.2
+Average received per second: 42032.0
+Average sent per group: 5640.92
+Average received per group: 4203.2
+Average sent per client: 564.092
+Average received per client: 420.32
+Average sent per client per second: 56.4092
+Average received per client per second: 42.032
+}}}
+
+{{{
+#!text
+
+G:100   CPG:15  T:10
+Sent: 611595
+Received: 440836
+Average sent per second: 61159.5
+Average received per second: 44083.6
+Average sent per group: 6115.95
+Average received per group: 4408.36
+Average sent per client: 407.73
+Average received per client: 293.8906666666667
+Average sent per client per second: 40.773
+Average received per client per second: 29.38906666666667
+}}}
+
+Here the laptop starts to swap with 7500 users:
+{{{
+#!text
+
+G:750   CPG:10  T:10
+Sent: 990737
+Received: 512803
+Average sent per second: 99073.7
+Average received per second: 51280.3
+Average sent per group: 1320.9826666666668
+Average received per group: 683.7373333333334
+Average sent per client: 132.09826666666666
+Average received per client: 68.37373333333333
+Average sent per client per second: 13.209826666666666
+Average received per client per second: 6.837373333333334
+}}}
+
+So I'll try it with a binary message instead... //term_to_binary({<<"I've been mad for fucking years—absolutely years">>,os:timestamp()})// (the timestamps avoids the same binary being referenced and it's copied instead).
+
+No swapping this time.
+{{{
+#!text
+
+G:750   CPG:10  T:10
+Sent: 1184530
+Received: 697726
+Average sent per second: 118453.0
+Average received per second: 69772.6
+Average sent per group: 1579.3733333333332
+Average received per group: 930.3013333333333
+Average sent per client: 157.93733333333333
+Average received per client: 93.03013333333334
+Average sent per client per second: 15.793733333333332
+Average received per client per second: 9.303013333333334
+}}}
+
+Analysis of the code shows the benchmark to be really focusing on the code that needs to be tested, except what would be the random selection of a peer. There's also a discrepancy between received and sent messages, which sounds a bit weird because it grows as there are more users when the difference grows. I'm not sure this is due to latency, dropped messages, random distribution or my code that tests stuff in a wrong way, but an eye should be kept out for this code.
+
+Otherwise, the benchmark shows a somewhat good load possible. At 7500 users, it's still possible to send a few dozen thousand messages per second. The bottleneck will probably be the web server, which means the code is //fast enough//, especially given real users aren't likely to be spamming that much. Most of the memory consumption (not shown here) appears to be from the messages sent and the history kept by each user (10 messages).
+
+=== Realistic benchmark ===
+Not yet done.
+
+=== Server benchmark ===
+Not done either.