Humanoid Imitation of Facial Expressions through Deep Learning
A fine-grained emotion labelling framework that maps human facial motion to humanoid robot control and classifies each frame into 22 nuanced emotion categories.
Three-Stage Pipeline
Motion Transfer
LivePortrait transfers human facial motion onto a robot source image via keypoint warping and SPADE generation.
X2Control Regression
A ResNet-18 mapping network converts each synthesised frame into a 30-dimensional servo control vector.
Emotion Classification
Control vectors are mapped to AU intensities, then classified into 22 fine-grained emotions via a hybrid classifier.
22-Class Fine-Grained Emotion Taxonomy
A hierarchical taxonomy grounded in FACS Action Units, extending beyond the 7 basic emotions to capture nuanced compound expressions.
Action Units (AUs) are the building blocks of facial expressions defined by the Facial Action Coding System (FACS). Each AU corresponds to a specific facial muscle movement — for example, AU12 is the lip corner pull that produces a smile. Every emotion below is defined by its unique combination of AUs.
Tier 1 Basic Emotions (7)
Tier 2 Compound Emotions (10)
Combinations of two basic emotions, exhibiting AU patterns from both constituent emotions simultaneously.
Tier 3 Additional States (5)
Complex cognitive-emotional states that go beyond simple combinations.
Expression Gallery
Side-by-side comparisons: driving video (left) vs. synthesised robot output (right).
Acknowledgements
This project builds upon the X2CNet framework and the X2C dataset [1], which provides the three-stage pipeline for transferring human facial expressions to humanoid robots via 30-dimensional control vectors. Our fine-grained emotion taxonomy extends this pipeline with a semantic interpretation layer.
The emotion classification system is grounded in the Facial Action Coding System (FACS) [3], which defines the Action Unit vocabulary used to characterise each of our 22 emotion categories. The AU-based pattern matching stream of our hybrid classifier follows the conventions established by EmotioNet [2], which demonstrated that compound facial expressions can be reliably annotated through AU activation patterns at scale.