Unbiased offline recommender evaluation for missing-not-at-random implicit feedback
Research output: Contribution to journal › Conference article › Research › peer-review
Implicit-feedback Recommenders (ImplicitRec) leverage positive only user-item interactions, such as clicks, to learn personalized user preferences. Recommenders are often evaluated and compared offline using datasets collected from online platforms. These platforms are subject to popularity bias (i.e., popular items are more likely to be presented and interacted with), and therefore logged ground truth data are Missing-Not-At-Random (MNAR). As a result, the widely used Average-Over-All (AOA) evaluator is biased toward accurately recommending trendy items. In this paper, we (a) investigate evaluation bias of AOA and (b) develop an unbiased and practical offline evaluator for implicit MNAR datasets using the Inverse-Propensity-Scoring (IPS) technique. Through extensive experiments using four real-world datasets and four widely used algorithms, we show that (a) popularity bias is widely manifested in item presentation and interaction; (b) evaluation bias due to MNAR data pervasively exists in most cases where AOA is used to evaluate ImplicitRec; and (c) the unbiased estimator significantly reduces the AOA evaluation bias by more than 30% in the Yahoo! music dataset in terms of the Mean Absolute Error (MAE).
Original language | English |
---|---|
Journal | RecSys 2018 - 12th ACM Conference on Recommender Systems |
Pages (from-to) | 279-287 |
Number of pages | 9 |
DOIs | |
Publication status | Published - 27 Sep 2018 |
Externally published | Yes |
Event | 12th ACM Conference on Recommender Systems, RecSys 2018 - Vancouver, Canada Duration: 2 Oct 2018 → 7 Oct 2018 |
Conference
Conference | 12th ACM Conference on Recommender Systems, RecSys 2018 |
---|---|
Country | Canada |
City | Vancouver |
Period | 02/10/2018 → 07/10/2018 |
Sponsor | ACM Special Interest Group on Computer-Human Interaction (SIGCHI) |
Bibliographical note
Publisher Copyright:
© 2018 Association for Computing Machinery.
- Bias, Evaluation, Implicit feedback, Propensity, Recommendation
Research areas
ID: 301825505