Commits

Fred T-H  committed ec3335c

Added realistic benchmarks

  • Participants
  • Parent commits 88cf960

Comments (0)

Files changed (1)

File Benchmarks.wiki

 Otherwise, the benchmark shows a somewhat good load possible. At 7500 users, it's still possible to send a few dozen thousand messages per second. The bottleneck will probably be the web server, which means the code is //fast enough//, especially given real users aren't likely to be spamming that much. Most of the memory consumption (not shown here) appears to be from the messages sent and the history kept by each user (10 messages).
 
 === Realistic benchmark ===
-Not yet done.
+The second benchmark is the [[http://bitbucket.org/ferd/chut/src/tip/src/benchmark_real.erl|Realistic benchmark]].
+
+The realistic benchmarks operate in a manner similar to the stress benchmarks, but there is a 5 seconds delay between each message sent by any user. The idea is to see whether the number of messages sent/received on a average stay constant when we augment the number of users in a system. This should be a decent test of latency and of how many users can be supported in a more realistic scenario.
+
+{{{
+#!text
+G:10    CPG:5   T:60
+Sent: 550
+Received: 550
+Average sent per group: 55.0
+Average received per group: 55.0
+Average sent per client: 11.0
+Average received per client: 11.0
+}}}
+
+{{{
+#!text
+G:10    CPG:10  T:60
+Sent: 1100
+Received: 1141
+Average sent per group: 110.0
+Average received per group: 114.1
+Average sent per client: 11.0
+Average received per client: 11.41
+}}}
+
+{{{
+#!text
+G:100   CPG:10  T:60
+Sent: 11000
+Received: 11393
+Average sent per group: 110.0
+Average received per group: 113.93
+Average sent per client: 11.0
+Average received per client: 11.393
+}}}
+
+Here my guess about why I get more messages received than sent is that a sent message generates one 'sent' event, counted as a received message. What that means is that because the 'sent' event takes less time than the 'receive' event to show up (no lookup needed), when I send the termination signal, the synchronization between what was sent and received is always off.
+
+The proof of this rests in how there is not much variation in the rate of Sent/Received depending on how long the test runs:
+
+{{{
+#!text
+G:100   CPG:10  T:10
+Sent: 1996
+Received: 2107
+...
+
+G:100   CPG:10  T:30
+Sent: 5182
+Received: 5480
+...
+
+G:150   CPG:10  T:30
+Sent: 7500
+Received: 7771
+...
+}}}
+
+Where the ratios are of 0.947, 0.946 and 0.965, respectively. The difference in what's sent and received doesn't seem to depend on time, groups or clients. Thus the cutoff point must be the next rational error cause. Now let's carry on with the benchmarks...
+
+{{{
+#!text
+G:100   CPG:15  T:60
+Sent: 16516
+Received: 17154
+Average sent per group: 165.16
+Average received per group: 171.54
+Average sent per client: 11.010666666666667
+Average received per client: 11.436
+}}}
+
+And for 7500 users:
+{{{
+#!text
+G:750   CPG:10  T:60
+Sent: 84110
+Received: 86456
+Average sent per group: 112.14666666666666
+Average received per group: 115.27466666666666
+Average sent per client: 11.214666666666666
+Average received per client: 11.527466666666667
+}}}
+
+Still no change in the stats. Let's push it to 15 000 users (you need to raise the default process limit: we have 15000*3+750 processes (Users * supervisors + groups + ...) = 45 750 + ... Given the limit is 32 768 by default, we've gotta go higher. Restart the VM with 'erl +P 75000' and then it can be run:
+
+{{{
+#!text
+G:750   CPG:20  T:60
+Sent: 179621
+Received: 183794
+Average sent per group: 239.49466666666666
+Average received per group: 245.05866666666665
+Average sent per client: 11.974733333333333
+Average received per client: 12.252933333333333
+}}}
+
+The send/received ratios remain similar, and the average per client too.  The core of the chat server can thus theoretically handle over 15 000 users sending each other messages every 5 second without too much degradation in response time per user. Note that at that point, the shut down of processes became a bit hard on my laptop and user timeouts started appearing (after the results were done and clients disconnected). Real world trials would be needed to further show reliability of Chut's core, but so far I'm pretty satisfied with the results.
 
 === Server benchmark ===
-Not done either.
+Not done yet.