Publication:
Investigating contributions of speech and facial landmarks for talking head generation

Thumbnail Image

School / College / Institute

Organizational Unit

Program

KU Authors

Co-Authors

Publication Date

Language

Embargo Status

NO

Journal Title

Journal ISSN

Volume Title

Alternative Title

Abstract

Talking head generation is an active research problem. It has been widely studied as a direct speech-to-video or two stage speech-to-landmarks-to-video mapping problem. In this study, our main motivation is to assess individual and joint contributions of the speech and facial landmarks to the talking head generation quality through a state-of-the-art generative adversarial network (GAN) architecture. Incorporating frame and sequence discriminators and a feature matching loss, we investigate performances of speech only, landmark only and joint speech and landmark driven talking head generation on the CREMA-D dataset. Objective evaluations using the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and landmark distance (LMD) indicate that while landmarks bring PSNR and SSIM improvements to the speech driven system, speech brings LMD improvement to the landmark driven system. Furthermore, feature matching is observed to improve the speech driven talking head generation models significantly.

Source

Publisher

International Speech Communication Association (ISCA)

Subject

Photography, Image color analysis, Pipelines, Computer architecture, Network architecture, Noise measurement, Colored noise, Computational photography, Low-light imaging, Image denoising, Burst images

Citation

Has Part

Source

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

Book Series Title

Edition

DOI

10.21437/Interspeech.2021-1585

item.page.datauri

Link

Rights

Copyrights Note

Endorsement

Review

Supplemented By

Referenced By

0

Views

4

Downloads

View PlumX Details