Living Labs for Search

Summary

This presentation introduces Living Labs for Search, a networked research infrastructure that addresses the challenge of validating information retrieval systems through real-world experimentation with actual search engines and users. The approach overcomes traditional limitations of controlled experiments in search research—which rely on expensive and unreliable document annotations—by enabling researchers to test their algorithms with live user interactions and click data from participating search engines including major industry partners like Microsoft, Yahoo, and others. The initiative represents a significant methodological advancement in information retrieval research, providing publicly available tools, APIs, and documentation while fostering collaboration between academic researchers and industry partners across 12 countries, thereby bridging the gap between laboratory-based evaluation and real-world search system performance.