Publication:
Simulating simple user behavior for system effectiveness evaluation

Placeholder

Departments

School / College / Institute

Program

KU-Authors

KU Authors

Co-Authors

Kanoulas, Evangelos
Carterette, Ben

Publication Date

Language

Embargo Status

Journal Title

Journal ISSN

Volume Title

Alternative Title

Abstract

Information retrieval effectiveness evaluation typically takes one of two forms: batch experiments based on static test collections, or lab studies measuring actual users interacting with a system. Test collection experiments are sometimes viewed as introducing too many simplifying assumptions to accurately predict the usefulness of a system to its users. As a result, there is great interest in creating test collections and measures that better model user behavior. One line of research involves developing measures that include a parameterized user model; choosing a parameter value simulates a particular type of user. We propose that these measures offer an opportunity to more accurately simulate the variance due to user behavior, and thus to analyze system effectiveness to a simulated user population. We introduce a Bayesian procedure for producing sampling distributions from click data, and show how to use statistical tools to quantify the effects of variance due to parameter selection.

Source

Publisher

ACM

Subject

Computer engineering

Citation

Has Part

Source

International Conference on Information and Knowledge Management, Proceedings

Book Series Title

Edition

DOI

10.1145/2063576.2063668

item.page.datauri

Link

Rights

Copyrights Note

Endorsement

Review

Supplemented By

Referenced By

0

Views

0

Downloads

View PlumX Details