This repository holds code for Living Labs for IR Evaluation (LL4IR).

In particular, this repository contains:

  • the LL4IR API
  • example clients for this API
  • a dashboard
  • all documentation.

For more information, consult:


If you use this code, please refer to this paper:

        title = {Head First: Living Labs for Ad-hoc Search Evaluation},
        author = {K. Balog, L. Kelly, A. Schuth},
        url = {},
        year = {2014},
        booktitle = {CIKM'14},


The following people have contributed to this code:

  • Anne Schuth
  • Peter Dekker
  • Krisztian Balog
  • Harrie Oosterhuis
  • David Bouček

If your name is missing, please let us know (or create a pull request).


This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this program. If not, see


The development of this code is partially supported by the EU FP7 project LiMoSINe ( and by the ESF ELIAS Research Networking Programme.