Uncertainty estimation in machine learning based classification models

Abstract

Machine learning has demonstrated excellence at medical image analysis tasks like segmentation and classification for diagnosis of autonomous driving, cyber bullying detection, Twitter’s sentiment analysis, from music generation to augmenting or replacing radiologists to identify cancer in Computed Tomography scans. Deep learning is a component of many industrial and clinical solutions. Despite their success, these methods do not consider the output quality and are simply concerned with improving point forecast accuracy and for that it is essential to understand how trustworthy a forecast is. Deep learning algorithms are becoming simpler to employ as new ones are developed, but uncertainty estimates still poses a serious challenge for safety-critical applications. The ability to estimate uncertainty not only enables us to assess the dependability of a system’s decision, but it also enables us to take on nondeterministic tasks with multiple potential outcomes, such as future prediction, which, when fully characterised, is a crucial component of human intelligence. This thesis starts with a general presentation of the uncertainty principle and apply them in Twitter US Airline Sentiment data to analyze how travelers in February 2015 expressed their feelings on Twitter. First, I used different text classification models with predicting uncertainty in respective to every single model . Then I compare these classification techniques based on the results, to find which machine learning algorithms appears to be superior to the other algorithms for this Twitter aircraft sentiment analysis data-set provided by the United States. Finally, I choose the best models (models with lowest uncertainty) and ensemble them to see if we can improved our model accuracy of prediction using uncertainty as a parameter.