Deep neural network augmentation: generating faces for affect analysis
Kollias, Dimitrios 
ORCID: https://orcid.org/0000-0002-8188-3751, Cheng, Shiyang, Ververas, Evangelos, Kotsia, Irene ORCID: https://orcid.org/0000-0002-3716-010X and Zafeiriou, Stefanos
(2020)
Deep neural network augmentation: generating faces for affect analysis.
International Journal of Computer Vision, 128
(5)
.
pp. 1455-1484.
ISSN 0920-5691
[Article]
(doi:10.1007/s11263-020-01304-3)
|
PDF
- Published version (with publisher's formatting)
Available under License Creative Commons Attribution 4.0. Download (5MB) | Preview |
Abstract
This paper presents a novel approach for synthesizing facial affect; either in terms of the six basic expressions (i.e., anger, disgust, fear, joy, sadness and surprise), or in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the emotion activation). The proposed approach accepts the following inputs:(i) a neutral 2D image of a person; (ii) a basic facial expression or a pair of valence-arousal (VA) emotional state descriptors to be generated, or a path of affect in the 2D VA space to be generated as an image sequence. In order to synthesize affect in terms of VA, for this person, 600,000 frames from the 4DFAB database were annotated. The affect synthesis is implemented by fitting a 3D Morphable Model on the neutral image, then deforming the reconstructed face and adding the inputted affect, and blending the new face with the given affect into the original image. Qualitative experiments illustrate the generation of realistic images, when the neutral image is sampled from fifteen well known lab-controlled or in-the-wild databases, including Aff-Wild, AffectNet, RAF-DB; comparisons with generative adversarial networks (GANs) show the higher quality achieved by the proposed approach. Then, quantitative experiments are conducted, in which the synthesized images are used for data augmentation in training deep neural networks to perform affect recognition over all databases; greatly improved performances are achieved when compared with state-of-the-art methods, as well as with GAN-based data augmentation, in all cases.
| Item Type: | Article |
|---|---|
| Keywords (uncontrolled): | Article, Special Issue on Generating Realistic Visual Data of Human Behavior, Dimensional, Categorical affect, Valence, Arousal, Basic emotions, Facial affect synthesis, 4DFAB, Blendshape models, 3DMM fitting, DNNs, StarGAN, GANimation, Data augmentation, Affect recognition, Facial expression transfer |
| Research Areas: | A. > School of Science and Technology > Computer Science |
| Item ID: | 30336 |
| Notes on copyright: | © The Author(s) 2020.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Useful Links: | |
| Depositing User: | Jisc Publications Router |
| Date Deposited: | 08 Jun 2020 15:46 |
| Last Modified: | 09 Feb 2022 10:37 |
| URI: | https://eprints.mdx.ac.uk/id/eprint/30336 |
Actions (login required)
 |
View Item |
Statistics
Additional statistics are available via IRStats2.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)