Bengali news recommendation system using topic modeling

Abstract

Natural Language Processing is the technology used to aid computer to understand human’s language. It is usually shortened as NLP, is a part of artificial intelligence that deals with the interaction between computers and humans using natural language. Topic Modeling is an unsupervised technique of Natural Language Processing, that find out topic of a document. It first cluster the words of the documents into specific number of topics and then studying those clusters it finds the distribution of topics for the document. A very popular topic modeling technique is Latent Dirichlet Allocation (LDA). A news recommendation system is a system which read a news article and gives us the articles from the corpus which have almost similar topic distribution as the given article. After removing stopwords, punctuations, foreign words from all documents of the corpus we have used Latent Dirichlet Allocation to apply topic modeling on the corpus of documents. By doing so, we have discovered topic distributions of all the documents of the corpus. Then the next is to discover topic distribution of the article, inputted by the user and then comparing that topic distribution with the previously discovered topic distributions of the documents of the corpus, will provide us the documents, having almost similar topic distribution as the inputted article. And thus, we have been given the recommended news article for the inputted article. Since this is a Bengali news recommendation system, the used corpus is composed of Bengali news documents and it will work for only Bengali articles.