Publication: Simulating simple user behavior for system effectiveness evaluation
Program
KU-Authors
KU Authors
Co-Authors
Kanoulas, Evangelos
Carterette, Ben
Publication Date
Language
Embargo Status
Journal Title
Journal ISSN
Volume Title
Alternative Title
Abstract
Information retrieval effectiveness evaluation typically takes one of two forms: batch experiments based on static test collections, or lab studies measuring actual users interacting with a system. Test collection experiments are sometimes viewed as introducing too many simplifying assumptions to accurately predict the usefulness of a system to its users. As a result, there is great interest in creating test collections and measures that better model user behavior. One line of research involves developing measures that include a parameterized user model; choosing a parameter value simulates a particular type of user. We propose that these measures offer an opportunity to more accurately simulate the variance due to user behavior, and thus to analyze system effectiveness to a simulated user population. We introduce a Bayesian procedure for producing sampling distributions from click data, and show how to use statistical tools to quantify the effects of variance due to parameter selection.
Source
Publisher
ACM
Subject
Computer engineering
Citation
Has Part
Source
International Conference on Information and Knowledge Management, Proceedings
Book Series Title
Edition
DOI
10.1145/2063576.2063668