Researcher:
Fatima, Syeda Narjis

Profile Picture
ORCID

Job Title

PhD Student

First Name

Syeda Narjis

Last Name

Fatima

Name

Name Variants

Fatima, Syeda Narjis

Email Address

Birth Date

Filters

2017 - 201922020 - 20211

Advanced Search

Filter by

Settings

Researcher Of Publications: search.filters.isAuthorOfPublication.818045fa-2100-4ef0-b1f0-b92cb976f691

Search Results

Now showing 1 - 3 of 3
  • Placeholder
    PublicationMetadata only
    Cross-subject continuous emotion recognition using speech and body motion in dyadic interactions
    (International Speech Communication Association ( ISCA), 2017) N/A; N/A; Department of Computer Engineering; Fatima, Syeda Narjis; Erzin, Engin; PhD Student; Faculty Member; Department of Computer Engineering; Graduate School of Sciences and Engineering; College of Engineering; N/A; 34503
    Dyadic interactions encapsulate rich emotional exchange between interlocutors suggesting a multimodal, cross-speaker and cross-dimensional continuous emotion dependency. This study explores the dynamic inter-attribute emotional dependency at the cross-subject level with implications to continuous emotion recognition based on speech and body motion cues. We propose a novel two-stage Gaussian Mixture Model mapping framework for the continuous emotion recognition problem. In the first stage, we perform continuous emotion recognition (CER) of both speakers from speech and body motion modalities to estimate activation, valence and dominance (AVD) attributes. In the second stage, we improve the first stage estimates by performing CER of the selected speaker using her/his speech and body motion modalities as well as using the estimated affective attribute(s) of the other speaker. Our experimental evaluations indicate that the second stage, cross-subject continuous emotion recognition (CSCER), provides complementary information to recognize the affective state, and delivers promising improvements for the continuous emotion recognition problem.
  • Placeholder
    PublicationMetadata only
    Use of non-verbal vocalizations for continuous emotion recognition from speech and head motion
    (IEEE, 2019) N/A; Department of Computer Engineering; Fatima, Syeda Narjis; Erzin, Engin; PhD Student; Faculty Member; Department of Computer Engineering; Graduate School of Sciences and Engineering; College of Engineering; N/A; 34503
    Dyadic interactions are reflective of mutual engagement between their participants through different verbal and non-verbal voicing cues. This study aims to investigate the effect of these cues on continuous emotion recognition (CER) using speech and head motion data. We exploit the non-verbal vocalizations that are extracted from speech as a complementary source of information and investigate their effect for the CER problem using gaussian mixture and convolutional neural network based regression frameworks. Our methods are evaluated on the CreativeIT database, which consists of speech and full-body motion capture under dyadic interaction settings. Head motion, acoustic features of speech and histograms of non-verbal vocalizations are employed to estimate activation, valence and dominance attributes for the CER problem. Our experimental evaluations indicate a strong improvement of CER performance, especially of the activation attribute, with the use of non-verbal vocalization cues of speech.
  • Placeholder
    PublicationMetadata only
    Use of affect context in dyadic interactions for continuous emotion recognition
    (Elsevier, 2021) Erzin, Engin; N/A; Fatima, Syeda Narjis; PhD Student; Graduate School of Sciences and Engineering; N/A
    Emotional dependencies play a crucial role in understanding complexities of dyadic interactions. Recent studies have shown that affect recognition tasks can benefit by the incorporation of a particular interaction's context, however, the investigation of affect context in dyadic settings using neural network frameworks remains a complex and open problem. In this paper, we formulate the concept of dyadic affect context (DAC) and propose convolutional neural network (CNN) based architectures to model and incorporate DAC to improve continuous emotion recognition (CER) in dyadic scenarios. We begin by defining a CNN architecture for single-subject CER-based on speech and body motion data. We then introduce dyadic CER as a two-stage regression framework. Specifically, we propose two dyadic CNN architectures where cross-speaker affect contribution to the CER task is achieved by: (i) the fusion of cross-subject affect (FoA) or (ii) the fusion of cross-subject feature maps (FoM). Based on the preceding dyadic models, we finally propose a new Convolutional LSTM (ConvLSTM) model for the dyadic CER. ConvLSTM architecture captures local spectro-temporal correlations in speech and body motion as well as the long-term affect inter-dependencies between subjects. Our multimodal analysis demonstrates that modeling and incorporation of the DAC in the proposed CER models provide significant performance improvements on the USC CreativeIT database and the achieved results compare favorably to the state-of-the-art.