Living Labs for Search
Workshop: Towards networked Research Infrastructures for behavioral studies, Brussels, Belgium. May 7, 2015.
Summary
This presentation introduces Living Labs for Search, a networked research infrastructure that addresses the challenge of validating information retrieval systems through real-world experimentation with actual search engines and users. The approach overcomes traditional limitations of controlled experiments in search research—which rely on expensive and unreliable document annotations—by enabling researchers to test their algorithms with live user interactions and click data from participating search engines including major industry partners like Microsoft, Yahoo, and others. The initiative represents a significant methodological advancement in information retrieval research, providing publicly available tools, APIs, and documentation while fostering collaboration between academic researchers and industry partners across 12 countries, thereby bridging the gap between laboratory-based evaluation and real-world search system performance.
Slides
Links
Related Publications
Overview of the Living Labs for Information Retrieval Evaluation (LL4IR) CLEF Lab 2015
Anne Schuth and Krisztian Balog and Liadh Kelly.
In Proceedings of CLEF'15, 2015.