Effective text summarization:a multi-linguistic analysis

Abstract

In the digital age, the generation of vast amounts of multilingual data has necessitated the development of effective text summarization tools. This final year project proposes an automatic approach for multilingual extractive text summarization, with a focus on five languages: Russian, Portuguese, Hindi, Korean, and French. The task of summarizing text has been a focal point of research in the field of Natural Language Processing (NLP) for several years. However, the majority of this work has been conducted in English, with limited attention given to other languages. This has resulted in a significant gap in the literature, particularly for languages such as Russian, Portuguese, Hindi, Korean, and French. Furthermore, the challenge is even more pronounced for low-resource languages, where the availability of annotated data for training and evaluation is scarce. With the increasing globalization and digitalization of our world, the need for effective multilingual text summarization tools has become more apparent. These tools can aid in various applications, from news aggregation to academic research, making the handling of large volumes of text data more manageable and efficient. The necessity of this work is underscored by the fact that the majority of the world’s population does not speak English as their first language, and yet, the majority of the digital text data is in English. The proposed approach addresses the challenge of summarizing heterogeneous textual content across languages. It leverages high-dimensional sentence embeddings generated by the Sentence Transformers model. These embeddings serve as excellent representations of context, capturing the meaning of each sentence effectively from the original text documents. Clustering techniques are employed to identify semantically related sentences and group them into coherent clusters. From each cluster, the most prominent representative is selected, ensuring a high-quality summary that captures essential information. This methodology is particularly effective for low-resource languages, where traditional methods may struggle due to the lack of annotated data. The project yielded the best results for the French language, with a ROUGE-1 score of 0.64, a ROUGE-2 score of 0.29, and a ROUGE-L score of 0.41, indicating the highest quality of summaries generated by the proposed approach. This final year project contributes to the field of Natural Language Processing by providing a novel approach to multilingual text summarization. It opens up new avenues for future research and has potential applications in various domains where handling multilingual text data is necessary. The proposed approach demonstrates promising results, particularly for the French language, and sets a benchmark for future work in this area. The project also highlights the need for further research in low-resource languages to make the digital world more inclusive and accessible..