The School of Computer Science is pleased to present...
An Approach of Lip Synchronization in ECA for E-learning
MSc Thesis Defense by: Rajveer Singh
Date: 11 Jun 2024
Time: 2:00 pm
Location: Essex Hall, Room 122
Abstract:
Embodied Conversational Agents (ECAs) are becoming more and more incorporated into diverse sectors such as customer service, healthcare, education, and entertainment. They are utilised to improve user experiences and make interactions more efficient. Lip synchronization and human-Like expressions play a crucial role in enhancing user trust, satisfaction, and overall anthropomorphism of ECAs. By ensuring that the lips of an ECA correspond with its spoken words, lip synchronisation improves the naturalness and perceived realism of interactions. Through expression manipulation, ECAs may use their facial expressions to communicate intents and feelings, leading to more empathic and engaging conversations. When combined, these characteristics improve user engagement, understanding, and rapport in human-computer interactions by making ECAs more believable and successful overall. Their smooth integration improves the ECA's capacity to forge deep relationships and accomplish its communication objectives.
This study presents a novel approach to achieve accurate lip synchronisation and facial expression manipulation utilising machine learning and 3D animation. The algorithm uses a pre-trained multilingual machine learning model for real-time phoneme representations for Text-to-Speech generation and lip synchronization. 3D object files are used for illustrating phonemes derived from the machine learning model. These files are utilised to interpolate between different lip forms and to monitor and map the motions of the lips. This process ensures precise synchronisation of the avatar's lip movements with the produced speech. Additionally, the algorithm incorporates ECA’s facial expression manipulation enabling the ECA to express happiness, surprise, concern, etc. increasing audience engagement. Through the integration of the newly developed algorithm with the overall e-tutoring system architecture, this research contributes to the advancement of interactive online learning experiences.
Thesis Committee:
Internal Reader: Dr. O. Syrotkina
External Reader: Dr. L. Oriet
Advisor: Dr. X. Yuan
Chair: Dr. I. Saini