extradoc / talk / pycon2014 / benchmark-abstract.rst

How to benchmark your programs


Benchmarking is an important topic for software development. There is a lot of research about how to do benchmarks carefully and correctly, however a lot of people lack knowledge in this area as well as basic statistics. This talk is aimed at sharing my knowledge about benchmarking the PyPy project but the lessons learned should really apply to any software.


People interested in benchmarking. People who have performance problems with their code.


Attendees will learn some basic statistics, how to write benchmarks, how to get reproducible data and how to avoid basic pitfalls with benchmarking.

Detailed abstract

Benchmarking is hard. This talk is a way to share my knowledge that I accumulated over the last few years while benchmarking various programs on top of the PyPy project. It'll include benchmarking general programs, using PyPy and CPython to get some data and postprocessing data. It'll also include a brief guide of stuff I use for visualization.


  1. Intro, PyPy, background, 5min
  2. Background about benchmarking in general 10min.
  3. Commonly used tools. 10min
  4. Questions 5min

Additional notes

PyPy benchmarking infrastructure is available under