Use of affect context in dyadic interactions for continuous emotion recognition

Abstract

Emotional dependencies play a crucial role in understanding complexities of dyadic interactions. Recent studies have shown that affect recognition tasks can benefit by the incorporation of a particular interaction's context, however, the investigation of affect context in dyadic settings using neural network frameworks remains a complex and open problem. In this paper, we formulate the concept of dyadic affect context (DAC) and propose convolutional neural network (CNN) based architectures to model and incorporate DAC to improve continuous emotion recognition (CER) in dyadic scenarios. We begin by defining a CNN architecture for single-subject CER-based on speech and body motion data. We then introduce dyadic CER as a two-stage regression framework. Specifically, we propose two dyadic CNN architectures where cross-speaker affect contribution to the CER task is achieved by: (i) the fusion of cross-subject affect (FoA) or (ii) the fusion of cross-subject feature maps (FoM). Based on the preceding dyadic models, we finally propose a new Convolutional LSTM (ConvLSTM) model for the dyadic CER. ConvLSTM architecture captures local spectro-temporal correlations in speech and body motion as well as the long-term affect inter-dependencies between subjects. Our multimodal analysis demonstrates that modeling and incorporation of the DAC in the proposed CER models provide significant performance improvements on the USC CreativeIT database and the achieved results compare favorably to the state-of-the-art.