Living Labs for IR Evaluation (LL4IR) Overview
CLEF'15, Toulouse, France. Sep 8, 2015.
Summary
This presentation provides an overview of Living Labs for IR Evaluation, a framework designed to enable realistic IR evaluation by exposing experimental systems to real users performing authentic tasks with genuine queries. The system features an open-source API that facilitates communication between researchers and search engines, handling queries, documents, runs, and clicks while making historical usage data available for analysis. This approach addresses the critical gap between laboratory-based IR evaluation and real-world search system performance, enabling more ecologically valid assessment of retrieval methods.
Slides
Links
Related Publications
Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015
Anne Schuth and Krisztian Balog and Liadh Kelly.
In Proceedings of CLEF'15, 2015.