Abstract:
Multimodal emotion classification involves the analysis and identification of human emo
tions by integrating data from multiple sources, such as audio, video, and text. This
approach leverages the complementary strengths of each modality to enhance the accu
racy and robustness of emotion recognition systems. Audio data, for example, captures
vocal tone and pitch, which are crucial for detecting emotions like anger or joy, while
video data provides visual cues such as facial expressions and body language. Text data,
often extracted from spoken words or written content, adds context and semantic depth to
the emotion analysis. However, one significant challenge is effectively integrating these
diverse data sources, each with unique characteristics and levels of noise. Additionally,
the scarcity of large, annotated multimodal datasets in Bangla limits the training and
evaluation of models. In this work, we introduced a novel multimodal Bangla dataset
named MAViT-Bangla (Multimodal Audio Video Text Bangla dataset), which consists
of 1002 samples incorporating audio, video, and text modalities. This dataset includes
emotional categories such as anger, fear, joy, and sadness, providing a rich resource for
emotion recognition studies in the Bangla language. Each sample in MAViT-Bangla
was meticulously annotated to ensure high-quality labels, making it a valuable asset for
researchers working in this domain. Moreover, we developed a framework for emotion
recognition that utilizes a cross-modal attention mechanism among unimodal features.
This mechanism facilitates the interaction and fusion of features from different modalities,
enhancing the model’s ability to capture nuanced emotional cues. The proposed approach
demonstrated its effectiveness by achieving an F1 score of 0.64, showcasing significant
improvement over unimodal methods. This indicates that integrating multiple modalities
through a cross-modal attention mechanism can substantially enhance the performance
of emotion recognition systems, especially in the context of the Bangla language, where
resources have historically been limited. The MAViT-Bangla dataset and our framework
thus represent significant advancements in the field of multimodal emotion classification.
