Publication: Investigating contributions of speech and facial landmarks for talking head generation
Files
Program
KU-Authors
KU Authors
Co-Authors
Publication Date
Language
Embargo Status
NO
Journal Title
Journal ISSN
Volume Title
Alternative Title
Abstract
Talking head generation is an active research problem. It has been widely studied as a direct speech-to-video or two stage speech-to-landmarks-to-video mapping problem. In this study, our main motivation is to assess individual and joint contributions of the speech and facial landmarks to the talking head generation quality through a state-of-the-art generative adversarial network (GAN) architecture. Incorporating frame and sequence discriminators and a feature matching loss, we investigate performances of speech only, landmark only and joint speech and landmark driven talking head generation on the CREMA-D dataset. Objective evaluations using the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and landmark distance (LMD) indicate that while landmarks bring PSNR and SSIM improvements to the speech driven system, speech brings LMD improvement to the landmark driven system. Furthermore, feature matching is observed to improve the speech driven talking head generation models significantly.
Source
Publisher
International Speech Communication Association (ISCA)
Subject
Photography, Image color analysis, Pipelines, Computer architecture, Network architecture, Noise measurement, Colored noise, Computational photography, Low-light imaging, Image denoising, Burst images
Citation
Has Part
Source
Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Book Series Title
Edition
DOI
10.21437/Interspeech.2021-1585