Enhancing stock prediction: integrating historical data and twitter sentiment analysis

Abstract

The use of social media data, particularly tweets, has gained significant attention in the field of stock prediction. This report focuses on utilizing tweets from prominent figures, such as Donald Trump and Narendra Modi, along with historical stock data to predict future stock values. The project incorporates Natural Language Processing (NLP) techniques to clean and process the tweet data, followed by sentiment analysis using the FinBERT model. The first phase involves data collection, where a substantial dataset of tweets from Donald Trump and Narendra Modi is gathered. Additionally, stock data for various stocks, including CRUDE OIL, S&P500, VIX, HANG SENG, and GOLD, is collected for analysis. The tweet data is then subjected to a custom NLP cleaning pipeline, removing irrelevant information and preparing it for further analysis. The sentiment analysis stage is crucial in understanding the sentiment expressed in the tweets with respect to the financial market. FinBERT, a powerful language model specifically designed for financial sentiment analysis, is employed to calculate sentiment scores for the tweets. The sentiment scores include positive, negative, and neutral probabilities, providing insights into the sentiment expressed in the tweets. To incorporate the temporal aspect, a memory-based approach is implemented to determine how long the effect of a tweet lasts. By assigning different weights to tweets based on their recency within a specified memory window, a weighted average sentiment score is calculated for each day. Finally, a bi-directional LSTM (Long Short-Term Memory) model is employed to predict future stock values based on the sentiment scores and historical stock data. By considering the look-back period and training the model on the merged dataset, predictions about future stock values are made. Overall, this project aims to leverage the power of tweets, sentiment analysis, and historical stock data to provide insights and predictions for stock market trends.