Data envelopment analysis for algorithm efficiency assessment in metamodel-based simulation optimization

Abstract

In the last years, the use of metamodel-based simulation optimization techniques to solve industrial problems stood out as a promising research field, mainly due to the advance of machine learning techniques. The number of metamodeling studies has grown considerably in recent years, but many academics and practitioners still have doubts about which metamodels to choose for their projects. In this way, some studies have compared the effectiveness of metamodeling algorithms. However, they have just analyzed the performance of one or more metrics separately; i.e., they did not analyze the overall efficiency of these metamodels. Basing the metamodels’ choice only on one or more metrics empirically might generate biases, causing distortions in decision-making. Therefore, we propose using the multi-criteria data envelopment analysis (MCDEA) model to systematically compare some of the main machine learning algorithms (support vector machine, artificial neural network, gradient-boosted trees, random forest, and Gaussian process). To evaluate the proposed approach, we developed discrete-event simulation models of three real case studies to obtain their input and output data. Moreover, we used machine learning algorithms to train and optimize the metamodels and, finally, new-MCDEA was adopted to compare the metamodels’ efficiency considering the associated error, fitting, training and prediction times, and response, among other metrics. Different from traditional comparison approaches, where different algorithms could be chosen depending on the decision-maker bias, the proposed work allowed a good balance between all metrics, and for all cases, the metamodels based on gradient-boosted trees were considered the most efficient.