DISCLAIMER: This piece of software is currently alpha quality.
ViffBench is a set of scripts that makes it easy to write, execute, and visulize benchmarks of VIFF protocols.
- Allows benchmarking using arbitrary VIFF revisions.
- Based on ssh and thus benefits from key agents like ssh-agent.
- Supports benchmarking of distributed VIFF protocols as well as local VIFF code.
- Automatically stores benchmark results in central database.
- Stores benchmark results for every single run, e.g. no info is lost.
ViffBench requires these 3rd party packages
As an illustrating example, all you have to do in order to benchmark the Toft05 comparison:
1. Define the benchmark
This means implementing a subclass of VIFFBenchmark that specifies the runtime class to be used as well as what should be measured. The overridden protocol method must return the benchmark result as a deferred value. Here we simply measure the time it takes to compare two random shared integers using the Toft05 comparison protocol.
class BenchmarkToft05(VIFFBenchmark): def get_runtime_class(self): return Toft05Runtime def protocol(self, rt, attr): Zp = GF(int(attr['modulus'])) a = rt.prss_share_random(Zp) b = rt.prss_share_random(Zp) c = gather_shares([a,b]) c.addCallback(self.start_timer) d = (a <= b) d.addCallback(self.stop_timer) return d
2. Run the benchmark
This is done by setting up a benchmark suite that knows a database where it can store the results.
d = MasterDatabase(host="localhost", db="bench_db", user="bench_user", passwd="secret") s = Suite(revision="ec70dc5ac213", database=d, user="bench_user", work_dir="some/temporary/dir") s.add_benchmark(BenchmarkToft05( runs=100, n=3, threshold=1, modulus=2342342348883, use_ssl=True)) s.run()
In this case, the Toft05 comparison is run 100 times with 3 players and security threshold 1 using ssl connections between the players. Multiple benchmarks can be added to the suite before running it. The resulting timings are stored in the database. Any parameter that is specified when adding a benchmark will be available to the running benchmark, i.e. in the protocol method above. A list of predefined parameters is found here.
3. Visualize the results
Data can be fetched directly from the database or through some helper methods (not quite written yet...) and visualized using e.g. matplotlib. Each single result is stored, so statistical analysis such as computing confidence intervals, is possible.
That's it. If you want to benchmark something that does not involve a protocol, e.g. just the time it takes to do a local multiplication (or even something unrelated to VIFF), you can subclass Benchmark instead of VIFFBenchmark. It works almost the same way execpt that the benchmark is only run on one machine and no runtime is available.