Living Labs for IR Evaluation (LL4IR) Overview

Summary

This presentation provides an overview of Living Labs for IR Evaluation, a framework designed to enable realistic IR evaluation by exposing experimental systems to real users performing authentic tasks with genuine queries. The system features an open-source API that facilitates communication between researchers and search engines, handling queries, documents, runs, and clicks while making historical usage data available for analysis. This approach addresses the critical gap between laboratory-based IR evaluation and real-world search system performance, enabling more ecologically valid assessment of retrieval methods.