Living Labs for Online Evaluation: From Theory to Practice

Anne Schuth and Krisztian Balog. In Proceedings of Tutorial at ECIR'16, 2016.

Abstract

Experimental evaluation has always been central to Information Retrieval research. The field is increasingly moving towards online evaluation, which involves experimenting with real, unsuspecting users in their natural task environments, a so-called living lab. Specifically, with the recent introduction of the Living Labs for IR Evaluation initiative at CLEF and the OpenSearch track at TREC, researchers can now have direct access to such labs. With these benchmarking platforms in place, we believe that online evaluation will be an exciting area to work on in the future. This half-day tutorial aims to provide a comprehensive overview of the underlying theory and complement it with practical guidance.

Links

Bib

@inproceedings{schuth2016lila,
  title = {Living Labs for Online Evaluation: From Theory to Practice},
  author = {Anne Schuth and Krisztian Balog},
  year = {2016},
  booktitle = {Proceedings of Tutorial at ECIR'16}
}