Trust my gesture or my word: how do listeners choose the information channel during communication?

Abstract

Information can be conveyed via multiple channels such as verbal and gestural (visual) channels during communication. Sometimes the information from different channels does not match (e.g., saying right while pointing to the left). How do addressees choose which information to act upon in such cases? In two experiments, we investigated this issue by having participants follow instructions on how to move objects on the screen. Experiment 1 examined whether people's choice of channel can be altered by feedback favoring either the verbal or the gestural channel. In Experiment 2, there was no feedback and participants were free to choose either channel. We also assessed participants' verbal and visuospatial working memory capacities. Results showed that, when faced with contradicting information, there is a natural bias at the group level toward relying on the verbal channel, although this bias can be temporarily altered by probabilistic feedback. Moreover, when labels were shorter and of higher frequency, participants relied more on the verbal channel. In the absence of feedback, the capacity of individuals' visual, but not verbal, working memory determined reliance on one channel versus the other. Collectively, these results show that information selection in communication is influenced by group-level biases, as well as the properties of items and characteristics of individuals.