Segmentation of Oral Epithelial Dysplasias Employing Mask R-CNN and Color Normalization

Oral epithelial dysplasia is a common type of pre-cancerous lesion that can be categorized as mild, moderate and severe. The manual diagnosis of this type of lesion is a time consuming and complex task. The use of digital systems applied to microscopic image analysis can aid the decision making of specialists. In recent years, deep learning-based methods are getting more attention due to its improved results in nuclei segmentation tasks. In this paper, we propose a methodology for nuclei segmentation on images of dysplastic tissues using neural networks. Several optimization algorithms and color normalization methods were evaluated. The methodology was performed on a dataset of mice tongue images. The experimental evaluations showed that the Nadam optimizer in combination with images without the use of color normalization obtained the best results. The method was able to segment the images with an average accuracy of 0.887, the sensitivity of 0.762 and specificity of 0.942. The algorithm was compared to other segmentation methods and showed relevant results. These values indicate that the proposed method can be used as a tool to aid specialists in the nuclei analysis of histological images of the buccal cavity.