Interpretable machine learning for generating semantically meaningful formative feedback

Abstract

We express our emotional state through a range of expressive modalities such as facial expressions, vocal cues, or body gestures. However, children on the Autism Spectrum experience difficulties in expressing and recognizing emotions with the accuracy of their neurotypical peers. Research shows that children on the Autism Spectrum can be trained to recognize and express emotions if they are given supportive and constructive feedback. In particular, providing formative feedback, (e.g., feedback given by an expert describing how they need to modify their behavior to improve their expressiveness), has been found valuable in rehabilitation. Unfortunately, generating such formative feedback requires constant supervision of an expert. In this work, we describe a system for automatic formative assessment integrated into an automatic emotion recognition setup. Our system is built on an interpretable machine learning framework that answers the question of what needs to be modified in human behavior to achieve a desired expressive display. It propagates the desired changes to human-understandable attributes through explanation vectors operating on a shared low level feature space. We report experiments conducted on a childrens voice data set with expression variations, showing that the proposed mechanism generates formative feedback aligned with the expectations reported from a clinical perspective.