Study of Human Emotion Using Multimodal Data

Abstract

Emotion analysis and classification problem is an emerging as well as challenging task in today’s world. It has an immense application on business, healthcare and educational industries. Precise emotion recognition of the user is pretty much difficult as the real set of possible emotional states are fuzzy in nature. Not only that, but also short utterances cannot be properly classified because of their dependency on the context of the dialogue. So, various machine learning and deep learning models can be deployed on multi modal data (text, emoji, audiovisual data etc.) in order to precisely classify and analyze various types of emotion specified by eminent researchers along with various evaluation metrics like confusion matrix, weighted f1 score, accuracy etc. so that predictive models can be compared and assessed. In this work, multi modal multi party dataset has been taken into consideration and this dataset has been accordingly preprocessed so that it can be annotated and features can be extracted from textual and audio data. Then, baseline algorithm like bcLSTM has been deployed on uni modal data and audio along with bimodal data and at last they are analyzed by some evaluation parameters relevant to this domain.