Machine learning models for predicting prostate cancer recurrence and identifying potential molecular biomarkers

Prostate cancer (PCa) recurrence affects between 20% and 40% of patients, being a significant challenge for predicting clinical outcomes and increasing survival rates. Although serum PSA levels, Gleason score, and tumor staging are sensitive for detecting recurrence, they present low specificity. This study compared the performance of three supervised machine learning models, Naive Bayes (NB), Support Vector Machine (SVM), and Artificial Neural Network (ANN) for classifying PCa recurrence events using a dataset of 489 patients from The Cancer Genome Atlas (TCGA). Besides comparing the models performance, we searched for analyzing whether the incorporation of specific genes expression in the predictor set would enhance the prediction of PCa recurrence, then suggesting these genes as potential biomarkers of patient prognosis. The models showed accuracy above 60% and sensitivity above 65% in all combinations. ANN models were more consistent in their performance across different predictor sets. Notably, SVM models showed strong results in precision and specificity, particularly considering the inclusion of genes selected by feature selection (NETO2, AR, HPN, and KLK3), without compromising sensitivity. However, the relatively high standard deviations observed in some metrics indicate variability across simulations, suggesting a gap for additional studies via different datasets. These findings suggest that genes are potential biomarkers for predicting PCa recurrence in the dataset, representing a promising approach for early prognosis even before the main treatment.