Robustness of a macroscopic computer-vision wood identification model to digital perturbations of test images

Abstract

Distribution shift, a phenomenon in machine learning characterized by a change in input data distribution between training and testing, can reduce the predictive accuracy of deep learning models. As operator and hardware conditions at the time of training are not always consistent with those after deployment, computer vision wood identification (CVWID) models are potentially susceptible to the negative impacts of distribution shift in the field. To maximize the robustness of CVWID models, it is critical to evaluate the influence of distribution shifts on model performance. In this study, a previously published 24-class CVWID model for Peruvian timbers was evaluated on images of test specimens digitally perturbed to simulate four kinds of image variations an operator might encounter in the field including (1) red and blue color shifts to simulate sensor drift or the effects of disparate sensors; (2) resizing to simulate different magnifications that could result from using different or improperly calibrated hardware; (3) digital scratches to simulate artifacts of specimen preparation; and (4) a range of blurring effects to simulate out-of-focus images. The model was most robust to digital scratches, moderately robust to red shift and smaller areas of medium-to-severe blur, and was least robust to resizing, blue shift, and large areas of medium-to-severe blur. These findings emphasize the importance of formulating and consistently applying best practices to reduce the occurrence of distribution shift in practice and standardizing imaging hardware and protocols to ensure dataset compatibility across CVWID platforms.