Probabilistic Multileave for Online Retrieval Evaluation

Summary

This presentation introduces probabilistic multileave, the first multileaving method that enables reuse of historical interaction data for online retrieval evaluation. Unlike previous multileaving approaches, probabilistic multileave allows ranker comparisons that require significantly less user interaction data by leveraging historical multileaved comparisons, while maintaining high sensitivity and remaining unbiased. The method addresses a critical limitation in multileaving evaluation by enabling data reuse and demonstrates superior scalability when the number of rankers increases, making it particularly valuable for large-scale online evaluation scenarios.