An automated framework to segment overlapping cells from confluent cancerous cell lines

Abstract

In this project, we have segment cancer overlapping cell nuclei from microscopy images. This deep learning segmentation model works with two stages, coarse segmentation, and fine segmentation. For coarse segmentation, I use Mask R-CNN, a deep learning instance segmentation technique for pixel-level segmentation, and for fine segmentation, I use the Euclidean distance transform method to generate the marker and then use the watershed segmentation method. Maximum nuclei are segmented in the coarse segmentation part, but some complex overlapping nuclei cannot be segmented well. For this, I use the fine segmentation stage. In fine segmentation for each clump nucleus, we use the Euclidean distance transform method to find the seed points for watershed segmentation. Nuclei that cannot be segmented by Mask R-CNN, by fine-segmenting those overlapping nuclei. For this experiment on microscopy images, I used the Kaggle 2018 Data Science Bowl dataset, which was taken using various magnifications and modalities. After comparing this method with other popular deep learning methods, this method archives a dice score of 83.3%, AJI of 70.03%, Precision of 89.80%, and Recall of 87.20%, respectively.