Our demo Lerot: an Online Learning to Rank Framework with Katja Hofmann, Shimon Whiteson and Maarten de Rijke was accepted for publication at the Living Labs for Information Retrieval Evaluation workshop at CIKM’13.
You can download Lerot from Bitbucket.

Abstract

Online learning to rank methods for IR allow retrieval systems to optimize their own performance directly from interactions with users via click feedback. In the software package Lerot, presented in this paper, we have bundled all ingredients needed for experimenting with online learning to rank for IR. Lerot includes several online learning algorithms, interleaving methods and a full suite of ways to evaluate these methods. In the absence of real users, the evaluation method bundled in the software packages is based on simulations of users interacting with the search engine. The software presented here has been used to verify findings of over six papers at major information retrieval venues over the last few years.