Assessing the predictive power of social media data-fed large language models on vote behavior

Abstract

This article investigates how large language models (LLMs) reflect human preferences and exhibit biases influenced by the diversity and nature of their input data. We used survey data related to Turkish presidential elections alongside tweets to assess the predictive performance and bias manifestations of LLMs under three different data inclusion strategies: (1) using only demographic information, (2) integrating demographic information with tweets, and (3) relying solely on tweets. Our findings reveal that prompts enriched with tweets typically achieve higher F1 Macro scores. However, this trend differs significantly when examining classes individually. While user-generated content significantly improves performance in predictions related to Recep Tayyip Erdogan, it does not show the same effect for Kemal Klllçdaroglu. This study shows that different models and prompting styles result in varied biases for each candidate, leading to mixed outcomes. These results underscore the importance of exploring how biases vary across different scenarios, models, and prompting strategies for each case.