Living Labs

The Living Labs for IR Evaluation (LL4IR) is a new evaluation paradigm. I implemented an API for participants ( researchers) and sites (search engines) that take part in this Living Lab (which is also run as a CLEF lab). The API allows participants (researchers) to evaluate their ranking systems on real users of real sites (search engines). On the flip site, it allows sites (search engines) to benefit from the knowledge of the research community.

The LL4IR API can be used by researchers to perform several actions such as obtaining queries, documents and feedback and to update runs. The API is RESTful, that is, everything is implemented as HTTP request, and we use the request types GET, PUT and DELETE.

The source code is available from bitbucket. It has mainly been developed by Anne Schuth and Krisztian Balog.

Several of my publications relate to Living Labs.

Lerot: an Online Learning to Rank Framework

Lerot is a framework, designed to run experiments on online learning to rank methods for information retrieval. It has mainly been developed by Katja Hofmann and Anne Schuth.
The source code of Lerot is available from bitbucket.
A paper describing Lerot is published in the Living Labs Workshop at CIKM’13: