Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015

Anne Schuth and Krisztian Balog and Liadh Kelly. In Proceedings of CLEF'15, 2015.
40 citations

Abstract

In this paper we report on the first Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab. Our main goal with the lab is to provide a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environments. For this first edition of the challenge we focused on two specific use-cases: product search and web search. Ranking systems submitted by participants were experimentally compared using interleaved comparisons to the production system from the corresponding use-case. In this paper we describe how these experiments were performed, what the resulting outcomes are, and conclude with some lessons learned.

Links

Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015
Google Scholar

Related Talks

Living Labs for Search - Workshop: Towards networked Research Infrastructures for behavioral studies. Brussels, Belgium. May 7, 2015.
Living Labs for IR Evaluation (LL4IR) Overview - CLEF'15. Toulouse, France. Sep 8, 2015.
LL4IR Introduction - CLEF LL4IR'15. Toulouse, France. Sep 10, 2015.

Bib

@inproceedings{schuth2015overview,
  title = {Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015},
  author = {Anne Schuth and Krisztian Balog and Liadh Kelly},
  year = {2015},
  booktitle = {Proceedings of CLEF'15}
}