Eﬀects of Position Bias on Click-Based Recommender Evaluation

Katja Hofmann and Anne Schuth and Alejandro Bellogin and Maarten de Rijke. In Proceedings of ECIR'14, 2014.

Abstract

Measuring the quality of recommendations produced by a recommender system (RS) is challenging. Labels used for evaluation are typically obtained from users of a RS, by asking for explicit feedback, or inferring labels from implicit feedback. Both approaches can introduce significant biases in the evaluation process. We investigate biases that may affect labels inferred from implicit feedback. Implicit feedback is easy to collect but can be prone to biases, such as position bias. We examine this bias using click models, and show how bias following these models would affect the outcomes of RS evaluation. We find that evaluation based on implicit and explicit feedback can agree well, but only when the evaluation metrics are designed to take user behavior and preferences into account, stressing the importance of understanding user behavior in deployed RSs.

Links

Eﬀects of Position Bias on Click-Based Recommender Evaluation
https://doi.org/10.1007/978-3-319-06028-6_67

Bib

@inproceedings{hofmann2014,
  title = {Eﬀects of Position Bias on Click-Based Recommender Evaluation},
  author = {Katja Hofmann and Anne Schuth and Alejandro Bellogin and Maarten de Rijke},
  year = {2014},
  booktitle = {Proceedings of ECIR'14},
  doi = {10.1007/978-3-319-06028-6_67}
}