European Journal of Operational Research 262 (2017) 673–681 Contents lists available at ScienceDirect European Journal of Operational Research journal homepage: www.elsevier.com/locate/ejor Decision Support Increasing the efficiency in integer simulation optimization: Reducing the search space through data envelopment analysis and orthogonal arrays Rafael de Carvalho Miranda a , ∗, José Arnaldo Barra Montevechi a , Aneirson Francisco da Silva b , Fernando Augusto Silva Marins b a Federal University of Itajubá (UNIFEI), Avenida BPS, 1303 – Caixa Postal: 50 – Itajubá, MG 37500-903, Brazil b Sao Paulo State University (UNESP). Av. Dr. Ariberto Pereira da Cunhsa, 333, Guaratinguetá, SP 12.516-410, Brazil a r t i c l e i n f o Article history: Received 29 December 2015 Accepted 5 April 2017 Available online 11 April 2017 Keywords: Data envelopment analysis Simulation Integer simulation optimization Super-efficiency a b s t r a c t The development of various heuristics has enabled optimization in simulation environments. Neverthe- less, this research area remains underexplored, primarily with respect to the time required for conver- gence of these heuristics. In this sense, simulation optimization is influenced by the complexity of the simulation model, the number of variables, and by their ranges of variation. Within this context, this pa- per proposes a method capable of identifying the best ranges for each integer decision variable within the simulation optimization problem, thereby providing a reduction in computational cost without loss of the quality in the response. The proposed method combines experimental design techniques, Discrete Event Simulation, and Data Envelopment Analysis. The experimental designs called orthogonal arrays are used to generate the input scenarios to be simulated, and super-efficiency analysis is applied in a Data Envelopment Analysis model with variable returns to scale to rank the input scenarios. The use of the super-efficiency concept enables to distinguish the most efficient input scenarios, which allows for the ranking of all the orthogonal array scenarios used. The values of the variables of the two input scenar- ios that present the highest values of super-efficiency are adopted as the new range of the optimization problem. To illustrate this method’s use and advantages, it was applied to real cases associated with inte- ger simulation optimization problems. Based on the results, the effectiveness of this approach is verified because it delivered considerable reductions in the search space and in the computational time required to obtain a solution without affecting the quality. © 2017 Elsevier B.V. All rights reserved. 1 b C t t & t s m w ( m f t a t n m i f t t a i h 0 . Introduction In the last 40 years, Discrete Event Simulation has increasingly een used by decision-makers in a wide range of fields ( Banks, arson II, Nelson, & Nicol, 2009; Law, 2015 ). The flexibility, versa- ility, and potential for analysis offered by this method have driven he adoption of simulation ( Jahangirian, Eldabi, Naseer, Stergioulas, Young, 2010; Xu, Huang, Chen, & Lee, 2015 ). Identified as one of he most commonly used research techniques in multiple sectors, imulation enables the study of complex systems in a more rapid, ore flexible, and more economic fashion than experimentation ith real systems, which consumes a greater amount of resources Law, 2015; Shen & Wan, 2009 ). ∗ Corresponding author. E-mail addresses: rafael.miranda@unifei.edu.br (R.d.C. Miranda), ontevechi@unifei.edu.br (J.A.B. Montevechi), aneirson@feg.unesp.br (A.F. da Silva), marins@feg.unesp.br (F.A.S. Marins). s 2 F 2 m ttp://dx.doi.org/10.1016/j.ejor.2017.04.016 377-2217/© 2017 Elsevier B.V. All rights reserved. One drawback to simulation is the fact that it provides statis- ical estimations for a single unique problem unless a sensitivity nalysis is applied ( Hillier & Lieberman, 2010 ). Without optimiza- ion software, analysts would be forced to simulate each input sce- ario individually (with different numerical and logical program- ing requirements that must be altered manually by the analyst n each simulation run) based on the real-world situation to search or the simulation model with variable arrangements that produces he most desirable results ( Law & McComas, 2002 ). With the aim of overcoming this hurdle, the concept of simula- ion optimization appeared in the 1990s as a viable option ( Fu et l., 20 0 0; Pasupathy & Ghosh, 2014 ) and has been widely accepted n the market, as evidenced by the availability of simulation oftware with integrated optimization routines ( Alrabghi & Tiwari, 015, Banks et al., 2009; Better, Glover, & Kochenberger, 2015; igueira & Almada-Lobo, 2014; Fu, 2015; Fu et al., 2014; Kleijnen, 015 ). Although advancements have occurred in the realm of opti- ization, a common criticism that remains is that when these http://dx.doi.org/10.1016/j.ejor.2017.04.016 http://www.ScienceDirect.com http://www.elsevier.com/locate/ejor http://crossmark.crossref.org/dialog/?doi=10.1016/j.ejor.2017.04.016&domain=pdf mailto:rafael.miranda@unifei.edu.br mailto:montevechi@unifei.edu.br mailto:aneirson@feg.unesp.br mailto:fmarins@feg.unesp.br http://dx.doi.org/10.1016/j.ejor.2017.04.016 674 R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 t B l i c c s o F e m i s f w t t 5 2 2 l t c v c l a a v d m v t a c l s m s w t i c r v s N t e t r c i commercial software packages are presented with more than one decision variable, the solution time usually grows significantly ( Harrel, Ghosh, & Bowden, 2004; Tenne & Goh, 2010 ). Furthermore, according to Hillier and Lieberman (2010) , despite advancements in optimization software, simulation is still considered a relatively slow and costly means of studying chaotic and dynamic systems. Complex random systems require great investment in terms of money and time for programming and analysis, without even con- sidering computational costs. In an attempt to move beyond this obstacle, many studies have concentrated on development of methods for optimization, i.e., metaheuristics, which increase optimization efficiency by seeking solutions within acceptable computational time ( Bachelet & Yon, 2007; Dellino, Kleijnen, & Meloni, 2010; Lin, Sir, & Pasupathy, 2013; Martins et al., 2013; Siegmund, Bernedixen, Pehrsson, Ng, & Deb, 2012; Willis & Jones, 2008 ). However, few studies have examined placing a limit on the optimization search space and thus working only with the most significant variables and the best range of val- ues for each variable. Among these investigations, factorial designs stand out in their attempt to identify the significant variables for problems and concentrate the optimization process on these vari- ables ( Besseris, 2012; Kleijnen, 1998; Kleijnen, 2017; Montevechi, Miranda, & Friend, 2012 ). Therefore, this study aims at defining the best variation ranges for each integer decision variable, thus reducing the search space and the computational time involved in the optimization process through simulation while reaching an optimal solution. In a way to meet its objective, this study proposes a method to integrate Or- thogonal Arrays, Discrete Event Simulation, and Data Envelopment Analysis (DEA) with variable returns to scale (DEA BCC) combined with the concept of super-efficiency. Orthogonal arrays are used in this study for a pre-assessment of the search space of the simulation optimization problems, be- cause these arrays are experimental designs covering several vari- ables with multiple levels; the orthogonal arrays are popular in Taguchian optimization; see Taguchi (1987) . Data Envelopment Analysis (DEA) consists of a methodology for a comparative measurement of the efficiency of Decision Mak- ing Units (DMUs) ( Cooper, Sieford, & Tone, 2007 ). The classic DEA models developed by Charnes, Cooper, and Rhodes (1978 ) consider a constant return to scale, it is named DEA CCR, referring to the first letters of the surnames of their authors. Banker, Charnes, and Cooper (1984 ) had modified this DEA model to include a variable return to scale, this model is named DEA BCC, referring to the first letters of the surnames of their authors. The DEA enables the identification of reference DMUs and gen- erate an efficient border. In classic DEA models, DMUs with effi- ciency indexes equal to one are considered efficient, while those with efficiency indexes lower than one are inefficient. However, several DMUs may have efficiency indexes equal to one. In this case and with a view to deal with such limitation, Andersen and Petersen (1993) have proposed the super-efficiency concept. The method promotes an alteration in the formulation of the DEA BCC model, allowing efficient DMUs to have efficiency indexes over one, and, this way, enabling to rank the most effi- cient DMUs. Therefore, based on the super-efficiency ranking, it is possible to reduce the search space of the optimization problem through the identification of the orthogonal array most efficient scenarios and, as consequence, incrementing the process efficiency. Recently, Miranda, Montevechi, Silva, and Marins (2014 ) pro- posed a new Fuzzy-DEA-BCC model to obtain optimum variation intervals for decision variables, to improve DEA’s discrimination power under occurrence of uncertainty and seeking for reduction in search space and computational solution times when compared to conventional simulation optimization techniques. They adopted orthogonal array to generate the necessary quantity of DMUs, and he output variables were generated by simulation. The Fuzzy-DEA- CC model was applied to two mono- and multi objective prob- ems, and it presented good results which confirmed its reliabil- ty and applicability. However, it can be difficult to apply this pro- edure to large-scale problems because it involves mathematical omplexity that requires advanced knowledge of Operations Re- earch techniques from the analyst involved in the simulation. Here, we proposed the classical DEA BCC model, also using rthogonal array, as simpler and innovative alternative than the uzzy DEA BCC model ( Miranda et al., 2014 ) to analyze the super- fficiency. The new procedure reduces search space in the opti- ization problems without compromising the solution quality, and t allows solving real, complex, and large-scale problems, with sub- tantial decrease in the associated computational time. To address the proposed objectives, this paper is organized into our more sections. Section 2 presents the literature review upon hich the study is based, Section 3 describes the proposed op- imization procedure, Section 4 applies the proposal to simula- ion models and discusses the obtained results, and finally, Section summarizes the study conclusions. . Literature review .1. Simulation optimization The overwhelming majority of simulation optimization prob- ems in such fields as manufacturing, logistical, and transport sys- ems; supply chains, medical fields, and other applications are too omplex to be modeled analytically. In these cases, simulation pro- ides a useful method for evaluating the performance of these omplex systems but can present deficiencies when applied in iso- ated situations. Thus, in many cases, the coupling of simulation nd optimization is necessary ( Lee, Chew, Teng, & Chen, 2008 ). According to Fu (2002) , Simulation Optimization can be defined s the process of seeking out the best combination of the decision ariables. Swisher et al . (20 0 0) goes on to state that the simulation model ecision variables domain may be both discrete, continuous, or a ixture of these two. The type of decision variable as well as its ariation range will influence the search space and the optimiza- ion method that will be used. Additionally, according to Nelson (2010) , many decision vari- bles in simulation optimization problems are discrete, which in- reases the difficulty in obtaining solutions and a gap exists in the iterature on this topic. Fu (1994) defines a traditional minimization problem with a ingle objective via simulation as: in f ( θ ) (1) ubject to: θ∈ � ith f( θ) = E [ ψ( θ, ω)] as the expected value for the estimated sys- em performance based on the simulation model, ψ j ( θ, ω) indicat- ng the observed performance values derived from the discrete or ontinuous decision variables belonging to the feasible set �, ω epresents the pseudorandom numbers (PRNs), θ is a controllable ector of p parameters, and � is the constraint set on θ. Kleijnen, Van Beers, and Van Nieuwenhuyse (2010) wrote that imulation optimization problems are generally difficult to solve or P-Hard and claim the disadvantages lie in the fact that simula- ion model output data originate from implicit functions and are xposed to noise caused by the PRNs. Furthermore, depending on he number of decision variables in the simulation model and the anges of the decision variables, the optimization process can be- ome complex due to the computational demands and the time nvolved in convergence of these algorithms. R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 675 2 u A a d m m fi d P s m s( u v w { p o f v fi x p p e D t m s ∑ ∑ u v o o g t f m s ∑ ∑ u v c o e m b m t t p t u D s t p m c p t g s t m e m ( n w i D a fi A e p l u l A ( e i o 0 .2. Data envelopment analysis (DEA) According to Cook and Seiford (2009) , DEA is a methodology sed to perform a comparison of the relative efficiencies of DMUs. nother important characteristic of the DEA models is that they llow consideration of incommensurability, that is, the presence of ifferent measurement units for the input and output matrix ele- ents ( Tamiz, Jones, & El-Darzi, 1995 ). The DEA models enable users to identify DMUs that are bench- arks for the other models under analysis, thus creating an ef- ciency boundary. The classic DEA CCR and DEA BCC models are efined as follows. In their original model, Charnes et al. (1978) used Fractional rogramming to obtain the input and output variable weights, as hown in ( 2 )–( 5 ): ax w 0 = s ∑ r=1 u r y r0 / m ∑ i =1 v i x i 0 (2) ubject to: s ∑ r=1 u r y r j / m ∑ i =1 v i x i j ) ≤ 1 , j = 1 , 2 , . . . , n (3) r ≥ 0 , r = 1 , 2 , . . . , s (4) i ≥ 0 , i = 1 , 2 , . . . , m (5) here j are the DMU indices; n is the total number of DMUs; r ∈ 1, … ,s } is the output index, where s is total the number of out- uts; i ∈ { 1, … ,m } is the input index, where m is the total number f inputs; y rj is the r th output for the j th DMU; x ij is the i th input or the j th DMU; u r is the weight associated with the r th output; i is the weight associated with the i th input; w o is the relative ef- ciency of DMU 0 , which is the DMU under evaluation; and y r0 and i0 are respectively the outputs and inputs for DMU 0 . It can be observed that when w 0 =1, DMU 0 is efficient com- ared with the other units considered in the model; w 0 < 1 im- lies that the DMU is inefficient. The model given by ( 2 )–( 5 ) is not linear; however, it can be lin- arized as shown in ( 6 )–( 10 ), thus generating the input-oriented EA CCR model ( Charnes et al., 1978 ) that presents a constant re- urn of scale: ax w 0 = s ∑ r=1 u r y r0 (6) ubject to: m i =1 v i x i 0 = 1 (7) s r=1 u r y r j − m ∑ i =1 v i x i j ≤ 0 , j = 1 , 2 , . . . , n (8) r ≥ 0 , r = 1 , 2 , . . . , s (9) i ≥ 0 , i = 1 , 2 , . . . , m (10) In a similar procedure, it is possible to formulate the output- riented DEA CCR model ( Charnes et al., 1978 ). For cases in which it is interesting to consider variable returns f scale, the input-oriented DEA BBC model ( Banker et al., 1984 ) iven by ( 11 )–( 16 ) was developed, and the main difference be- ween the DEA BCC and DEA CCR models is the addition of the ree variable c 0 : ax w 0 = s ∑ r=1 u r y r0 + c 0 (11) ubject to: m i =1 v i x i 0 = 1 (12) s r=1 u r y r j − m ∑ i =1 v i x i j + c 0 ≤ 0 , j = 1 , 2 , . . . , n (13) r ≥ 0 , r = 1 , 2 , . . . , s (14) i ≥ 0 , i = 1 , 2 , . . . , m (15) 0 f ree (16) In a similar procedure, it is possible to formulate the output- riented DEA BCC model ( Banker et al., 1984 ). Banker et al. (1984) commented that a DMU that is consid- red efficient by BCC also would be considered efficient by the CCR odel; however, the inverse of this statement will not necessarily e true. Furthermore, according Cooper et al. (2007) , the DEA BCC odel is invariant to the application of linear transformations to he inputs and/or outputs values involved, which does not occur in he DEA CCR model. The DEA BCC model was chosen for use with the methodology roposed herein, primarily due to the stochastic and non-linear na- ure of simulation and the multiple inputs and outputs in the sim- lation models. These inputs and outputs are the DMUs for the EA model, as shown in Section 4.1 , and have different returns of cale. Moving forward, due to its property of invariance to linear ransformations, the DEA BCC model allows the use of data that resent negative values, which can occur in stochastic simulation odels, e.g., in the case of a variable associated with profit that ould turn negative in the case of financial loss. The solution pro- osed in the literature for this case is to add a positive value to he inputs and/or outputs values of each DMU. Finally, according to Azadeh, Moghaddam, Asadzadeh, and Ne- ahban (2011) , if the output values are given in the form of ratios, uch as production, productivity and waste rates, and these are the ype of problems addressed in this work, the DEA BCC model is the ost frequently applied. For Banker, Charnes, Cooper, Swarts, and Thomas (1989) , to nable suitable discrimination of the DMUs in a traditional DEA odel, one should verify whether the number of DMUs meets golden rule): ≥ Max { ( ms ) , 3 ( m + s ) } (17) here n is the total number of DMUs; m is the total number of nputs and s is total the number of outputs. Classic DEA models consider DMUs with w 0 =1 as efficient and MUs with w 0 < 1 as inefficient. This situation occurs if there is possibility that multiple DMUs may have w 0 =1; i.e., it is dif- cult to differentiate between DMUs. To address this limitation, ndersen and Petersen (1993) proposed a concept known as super- fficiency, which determines the difference between DMUs that resent w 0 =1. For ( Bal et al., 2010 ) the use of super-efficiency al- ows an increase in the DMU discrimination. This concept will be sed in this study together with the techniques covered in the fol- owing sections. For use of the super-efficiency approach in a DEA BCC model, ndersen and Petersen (1993) suggest the removal of restriction 13 ) from the model but only for the DMU under analysis, thus nabling the DMUs to attain levels greater than 1, which may facil- tate the elaboration of an efficiency ranking for those DMUs that riginally presented w =1. 676 R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 Table 1 Taguchi’s orthogonal array and full factorial design. Orthogonal array L16 (4 4 ) Full factorial (2 4 ) A B C D A B C D 1 1 1 1 1 1 1 1 1 2 2 2 4 1 1 1 1 3 3 3 1 4 1 1 1 4 4 4 4 4 1 1 2 1 2 3 1 1 4 1 2 2 1 4 4 1 4 1 2 3 4 1 1 4 4 1 2 4 3 2 4 4 4 1 3 1 3 4 1 1 1 4 3 2 4 3 4 1 1 4 3 3 1 2 1 4 1 4 3 4 2 1 4 4 1 4 4 1 4 2 1 1 4 4 4 2 3 1 4 1 4 4 4 3 2 4 1 4 4 4 4 4 1 3 4 4 4 4 i t t T d m t 3 s d t s u F m p i 2.3. Orthogonal arrays Ballantyne, Van Oorschot, and Mitchell (2008) note that the process of simulation optimization for complex systems with mul- tiple input variables can require a considerable amount of time and consume extensive resources in terms of personnel and financial impacts. For systems with only a few levels, Antony (2006) recom- mends the use of factorial strategies, i.e., full, or fractional designs, which typically work with two levels, i.e., one lower and one upper level ( Montgomery, 2009 ). Complex processes involving four or more input variables with three or more levels require a larger number of experiments for thorough optimization. One alternative approach is the use of or- thogonal array ( Taguchi, 1987 ), which reduces the number of nec- essary experiments and makes it possible to analyze and deter- mine the most important input variables. Taguchi, Chowdhury, and Wu (2005 ) define the orthogonal ar- ray as a matrix of numbers arranged in lines and columns in a way that each pair of columns is orthogonal to the other. When used in an experiment, each line represents an experimental situation, and each column a specific factor or condition that may be altered. Ac- cording to Taguchi et al. (2005 ) an orthogonal matrix enables the evaluation of the impact of each parameter independently. In fact, this array allows for the consideration of k input vari- ables with l levels while testing each level of each variable in a balanced manner ( Ross, 1996 ). In these arrays, the number of ex- periments needed is recognized by the index that accompanies the L abbreviation (Latin Square). A L8 experiment, for instance, refers to an experiment with eight experiments, disregarding any repli- cations ( Ross, 1996 ). For Roy (2010) , orthogonal arrays usually de- fine other ratings that show the number of factors involved in each array. For instance, the orthogonal array L16 (4 4 ) foresees the per- formance of 16 experiments in 4 columns, being each column the combination of 4 levels. In this paper, orthogonal array is used as a pre-evaluation phase of the search space during optimization to generate a more diver- sified experimental matrix. The orthogonal arrays are selected as a function of the number of variables involved and their ranges, and the experimental matrix guides the running of the simulation in- put scenarios. The efficiency for each input scenario is determined by a DEA BCC model. As an example, Table 1 presents an experimental matrix for a L16 (4 4 ) orthogonal array and full factorial design (2 4 ) for four in- put variables (A, B, C, D) and four levels (1, 2, 3, 4). The example shows that the orthogonal array explores all four levels for all four input variables that compose the feasible region n a more diversified manner, whereas the full factorial experimen- al matrix examines only two levels for each input variable these wo levels are the levels 1 and 4 (excluding the levels 2 and 3). he discussion about the power of analysis of both experimental esigns is beyond the scope of this paper, since only the experi- ental arrays are used to represent the feasible region of the op- imization problem. . Proposed simulation optimization method The procedure proposed in this paper includes the following as- umptions: - The use of orthogonal arrays permits to explore the feasible re- gion of the simulation optimization problem. - By ranking of the most efficient DMUs, the application of DEA in the orthogonal array makes it possible to reduce the search space of the simulation optimization problem. - With the search space reduction, the computational time re- quired for solving the simulation optimization problem will be less than that associated with the full search space to obtain solutions that are statistically equivalent. The proposed method for effective reduction of search space uring simulation optimization in this work makes use of four echniques: (1) Orthogonal array is applied to select the experimental ma- trix and generate input scenarios. (2) Simulation is used to obtain the input/output data. (3) A DEA BCC model is adopted to analyze the efficiency of each generated input scenario (DMUs) and to rank the DMUs. (4) Simulation optimization is used to obtain the solution. Fig. 1 (adapted from Miranda et al., 2014 ) presents the proposed tep-by-step approach for optimization and assumes that the sim- lation model has been built, programmed, verified, and validated. urthermore, all decision variables are integers. The steps are described as follows: (1) Determine the decision variables ( x 1 , x 2 , x 3 , … , x n ) and the input variable ranges for each variable. (2) Determine the estimator of the objective to be optimized ( y ). (3) Select the Taguchi’s orthogonal array (L4, L9, L12, L16, L25, L32, L54, …) as a function of the number of decision vari- ables and their ranges as well as the minimum number of DMUs to be analyzed using DEA BCC with the purpose of meeting the basic rule for number of DMUs ( Banker et al., 1989 ). (4) Execute the chosen orthogonal array combining original sim- ulation inputs. (5) Execute the experiments and store the simulated data for analysis. (6) Determine each input scenario’s super-efficiency by applica- tion of DEA BCC for the simulated results. (7) Rank the most efficient DMUs. (8) Set new ranges of variation for each decision variable based on two most efficient DMUs. Remove from the optimization process, the variables that possess equal values in both most efficient DMUs. (9) Optimize the simulation model using the new ranges for the remaining decision variables. It should be noted that the first two steps of the proposed ethod are part of the definition of formulating the optimization roblem, while the remaining steps are part of the method of solv- ng the problem. In Fig. 1 , the activities established to the right of R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 677 Fig. 1. Proposed optimization method (adapted from Miranda et al., 2014 ). t i c a a 4 d p c A p l m l n w s ∏ w t p s p p f d p c 4 4 t l m s P a t a a a b he main flow (steps 0, 5 and 9) are performed in the simulator or n the optimization module coupled to the simulator. To exemplify the application of the procedure from Fig. 1 , real ases will be presented in the next section. In accordance with the ssumptions of this study, the simulation models were validated nd verified and thus are suitable for simulation optimization. . Application of the optimization procedure, results, and iscussion According to Ahuja, Magnanti, and Orlin (1993) , there are three ossible approaches for complexity analysis of the proposed pro- edure: Empirical Analysis, Average-Case Analysis, and Worst-Case nalysis. This work adopted the Empirical Analysis for testing the erformance of the proposed procedure on certain classes of prob- em instances. This section describes real cases in which the proposed opti- ization procedure was implemented. The cases are typical prob- ems that appear in the manufacturing area and present increasing umber of scenarios. Note that for design of experiments problems ith multiple, and different level for each decision variable, the as- ociated complexity is: n i =1 l i (18) here l i is the number of levels for ith decision variable, and n is he number of decision variables. This study evaluated three classes of simulation optimization roblems: (1) Integer Simulation Optimization problems with the same number of levels per decision variable. (2) Integer Simulation Optimization problems with a different number of levels per decision variable. (3) Integer Simulation Optimization problems where the num- ber of decision variables changes while the simulation model remains the same. The simulation models were implemented using the ProModel oftware ( ProModel, 2016 ). The case studies represent large-scale roblems in which discrete event simulation is often used to sup- ort decision-making. The cases exemplify the use of the method or reducing the search space. It should be noted the method eveloped herein can be applied to any simulation optimization roblem if the problem presents the same assumptions already ited. .1. First class of simulation optimization problems .1.1. Case study 1 The first case study is taken from a Brazilian telecom company hat produces transponders sold to several countries. The simu- ation model was effectively verified and validated such that the odel represents the system that both researchers and systems pecialists sought to optimize. The model was programed in the roModel software and consists of eight locations, seven entities nd eight resources. The Appendix brings the probability distribu- ions (adherence tests) for the times of the simulated processes nd the conceptual model for the case study. For this first case study, the decision variables were defined s number of workbenches (types 1 and 2) denoted by ( x 1 , x 2 ), nd the number of technical personnel (types 1, 2 and 3) denoted y ( x , x , x ), which perform different activities that make up 3 4 5 678 R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 Table 2 Decision variables for original search space - case study 1. Decision variables (integers) x 1 Number of workbenches, Type 1 1 ≤ x 1 ≤ 5 x 2 Number of workbenches, Type 2 1 ≤ x 2 ≤ 5 x 3 Number of technical personnel, Type 1 1 ≤ x 3 ≤ 5 x 4 Number of technical personnel, Type 2 1 ≤ x 4 ≤ 5 x 5 Number of technical personnel, Type 3 1 ≤ x 5 ≤ 5 t d t n v F a w p b t f o s y a p t n e t w s i e g 3 c w a e 3 c i e p t s d r t t a i ( c 4 o s s d p the production cell. The decision variables are integers. Table 2 presents this information. The optimization objective was to find the best combination of variables to maximize the estimator of the total production, de- noted by y . From Table 2 , there are five decision variables, each one with five levels, and the total level number calculated by ( 18 ) is: ∏ 5 i =1 l i = 3 , 125 scenarios. Orthogonal array L25 was chosen for the quantity of variables and the variation of the levels for each. This array covers out 25 experiments, which meets the minimum rule ( Eq. 17 ). Once the array was defined, an experimental matrix was gen- erated and is presented in Table 3 . The input scenarios from the experimental matrix were simulated in ProModel. The simulation run length for each input scenario is one month of simulated op- erations in the production cell, and the data for each estimator of the objective function were stored to calculate the super-efficiency. The number of necessary replications for each scenario was calcu- lated considering the minimum number of data needed for the ap- plication of the parametric tests (normality test and t -test) and the calculus proposed by Banks et al. (2009 , p. 4 45–4 49), and turned out to be 30. Simulations of the 25 input scenarios and their respective repli- cations were performed on a computer with an Intel Core 2 Duo 1.58-GigaHertz processor, 2 GigaByte of RAM, and the Windows (64-bit) operating platform. For the simulation of the 30 replica- tions of each of the 25 scenarios, common random numbers (CRN) were used. The entire process lasted slightly less than 25 minutes. The software General Algebraic Modeling – GAMS, version 23.6.5 ( Gams, 2015 ), was used to calculate the efficiency related to each DMU and adapted to calculate the super-efficiency. With these results, it is possible to relate the efficiency value of each DMU to each input scenario. These values are listed in Table 3 . The formulation of the DEA model for the case study is in the Appendix. The analysis of the super-efficiency values ( Table 3 ), ranked the most efficient DMUs, and it was shown that DMU 17 is the most efficient followed by DMU 1. Both DMUs are listed in Table 3 . With the identification of the two most efficient DMUs, a new interval for each decision variable can be redefined, thus leading to a re- duction of optimization search space. For instance, for the x 1 vari- able, the analysis of Table 3 showed that in the most efficient DMU, such variable was equal to 4, while the second most efficient DMU was equal to 1, therefore, according to the proposed method, this variable assumes a new variation range given by 1 ≤ x 1 ≤ 4. Table 3 shows that variable x 5 presented the same value for its most effi- cient DMUs; namely, the value 1, reducing the number of variables under evaluation from 5 to 4. The new intervals for the other de- cision variables are shown in Table 4 . With the reduction of the variation interval for each decision variable, the search space was reduced from 3125 to 120 scenar- ios, a reduction of 96.16%. The commercial optimizer SimRunner ( SimRunner User Guide, 2002 ) was used to test the efficiency of this proposal . SimRunner is a robust software package that is avail- able with ProModel and uses Genetic Algorithms associated with Evolution Strategies ( Banks et al., 2009 ). To confirm the results from case study 1, SimRunner was set to he same conditions and the same objectives and with the original ecision variable ranges (see Table 2 ). The results obtained with he optimization by reducing the search space and with the origi- al search space of the problem are presented in Table 5 . The optimal values estimated by the optimizer for the decision ariables were equal only for the variable x 3 ; namely, the value 4. or the other decision variables, SimRunner found different values, s shown by the fact that the problem with the original ranges ould require researchers to hire a greater number of technical ersonnel of type 2 and type 3 and purchase additional work- enches of type 2. In a way to assure the equality of results for the estimator of he objective function, the t -test ( Montgomery, 2009 ) was applied or the difference of the 30 responses regarding each replication f the solutions found by the optimizer, considering the original earch space and the reduced search space. To do so, we defined ( i, j ) as the optimum estimated by method i with i = 1, 2 (reduced nd original search space) in replication j with j = 1, …, 30. Com- uted the 30 differences d ( j ) = y (1, j ) - y (2, j ), the t -test was applied o these d ( j ). The p -value of the test was equal to 0.758, evidencing o difference of results, that is, they are equivalent in terms of the stimated optimal simulation output. Continuing the analysis, it can be observed that the optimiza- ion phase completed 69 experiments before converging while orking in a reduced range equivalent to 57.5% of the total search pace and consumed 2.4 hours of computational time. In compar- son, the solution that considered the original range required 243 xperiments, which is equivalent to only 7.7% of the feasible re- ion, and consumed approximately 8 hours. To validate the assumptions applied and the results produced, 125 input scenarios (related to the total search space) were exe- uted with 30 replications each, i.e., a total of 93,750 simulations ere performed. Approximately 106 hours were required to run ll the input scenarios and their replications. In this sequence, the fficiency analysis using DEA BCC model was carried out for the 125 input scenarios (number of DMUs) and obtained 1426 effi- ient DMUs. For comparison, the DEA BCC model was also applied to 120 nput scenarios limited by the procedure proposed. In fact, consid- ring the 120 input scenarios, 73 were efficient, i.e., the proposed rocedure led to a new search space for the simulation optimiza- ion problem with approximately 61% of the original efficient input cenarios. In other words, the proposed procedure provided a re- uction of the search space from 3125 input scenarios to 120 (a eduction of approximately 96%) without loss of solution quality. Furthermore, it was observed that with generation of 100% of he possible input scenarios, the optimal solution (the average of he 30 replications) involved production of 1399 products within n interval with 95% confidence given by (1395; 1402). This result s statistically equivalent to that obtained by the proposed method reduced search space): 1397 products within an interval with 95% onfidence given by (1393; 1400). .2. Other classes of problems The same steps, presented for case study 1, were applied to ther case studies, described in the Appendix of this paper. The re- ults found for all case studies, with respect to reduction of search pace and time, are shown in Table 6 . Applying the method to the case studies led to an average re- uction in search space of 97%, and an average reduction in com- utation time of 40%, with no loss of quality of response reported. R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 679 Table 3 Experimental matrix and results obtained for case study 1. DMU Decision variables Estimator of the objective function Super-efficiency Ranking x 1 x 2 x 3 x 4 x 5 ȳ 1 1 1 1 1 1 853.77 2.0 0 0 2 2 1 2 2 2 2 1392.97 1.630 6 3 1 3 3 3 3 1391.40 1.0 0 0 19 4 1 4 4 4 4 1393.53 1.370 8 5 1 5 5 5 5 1392.97 1.0 0 0 20 6 2 1 2 3 4 1252.93 1.159 15 7 2 2 3 4 5 1394.50 1.085 16 8 2 3 4 5 1 1393.47 1.278 10 9 2 4 5 1 2 851.73 1.0 0 0 21 10 2 5 1 2 3 1136.77 1.218 13 11 3 1 3 5 2 1394.30 1.345 9 12 3 2 4 1 3 851.57 1.0 0 0 22 13 3 3 5 2 4 1396.43 1.875 4 14 3 4 1 3 5 1125.00 1.0 0 0 18 15 3 5 2 4 1 1393.43 1.529 7 16 4 1 4 2 5 1395.57 1.917 3 17 4 2 5 3 1 1395.43 2.989 1 18 4 3 1 4 2 1121.83 1.182 14 19 4 4 2 5 3 1394.90 1.687 5 20 4 5 3 1 4 847.63 1.0 0 0 23 21 5 1 5 4 3 1394.83 1.039 17 22 5 2 1 5 4 1152.07 1.244 11 23 5 3 2 1 5 852.07 1.0 0 0 24 24 5 4 3 2 1 1306.07 1.224 12 25 5 5 4 3 2 1389.67 0.790 25 Table 4 Decision variables for reduced search space – case study 1. Decision variables (Integers) x 1 Number of workbenches, Type 1 1 ≤ x 1 ≤ 4 x 2 Number of workbenches, Type 2 1 ≤ x 2 ≤ 2 x 3 Number of technical personnel, Type 1 1 ≤ x 3 ≤ 5 x 4 Number of technical personnel, Type 2 1 ≤ x 4 ≤ 3 Table 5 Optimization results for case study 1. Decision variables Solution Reduced search space (Proposed method) Original search space x 1 3 1 x 2 2 5 x 3 4 4 x 4 3 4 x 5 1 5 Estimator of the objective function Responses Reduced search space (Proposed method) Original search space ȳ 1397 1398 Confidence interval for ȳ (95%) (1393–1400) (1394–1401) Computational time Reduced search space (Proposed method) Original search space (2.4 + 0.42) hours 8.0 hours 5 w t m n m w w c m c s i a s n t a S o t o t b t i t t s o o b t . Conclusions and recommendations for future studies The proposed simulation optimization method was integrated ith DEA BCC, adapted to the concepts of super-efficiency and or- hogonal array, and used to determine the best search space. This ethod produced excellent results for the problems investigated. The studied cases, which were associated with real input sce- arios were modeled using simulation. With the application of this ethod, reductions of nearly 97% of the search space were attained ithout loss of response quality. We also wish to point out that, ith the application of the method, average reductions of 40% in omputational time were reached in the simulation optimization. Furthermore, it should be noted that because it is not a para- etric analysis tool, DEA enables users to work with data asso- iated with non-normal probability distributions as well as small amples. In addition, we solve the problem of incommensurability; .e. in the input and output matrix represented by the orthogonal rrays, we could admit different measurement units and different cales between the variables of decision and responses without the eed of any data modification. The use of the SimRunner optimizer allowed for comparison of he optimization results with those from the original search space nd from the reduced space proposed in this paper. In all cases, imRunner reached equivalent solutions in terms of the estimated ptimal simulation output ( y ), thus demonstrating that the reduc- ion in search space leads to optimal solutions for the problem. An- ther highlight is that in all cases, the reduction in computational ime obtained with the optimization procedure was significant. The reduction of search space allows the feasible region to be etter explored by SimRunner. In the studied cases, the coverage of he feasible region after reduction of the search space was signif- cantly greater resulting in estimated values of the objective func- ion that often do not differ significantly to those found by the op- imizer when considered all the search space. In this manner, it can be assumed that, it is possible to repre- ent the feasible region of a simulation optimization problem using rthogonal array. Additionally, using the super-efficiency analysis f the results of this array, it is possible to reduce the search space y ranking the most efficient DMUs using a DEA BCC model. With respect to recommendations for future studies, the au- hors suggest: – Application of the proposed procedure in multi-objectives opti- mization problems. 680 R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 Table 6 Results for reducing search space and computational time. Class / Case study Original search space Reduced search space Reduction (%) Time SimRunner (Original search space) Hours Time method (Reduced search space) Reduction (%) 1 1 3125 120 96.16 8.00 (2.40 + 0.42) h 64.79 2 15,625 480 96.93 78.50 (45.50 + 3.00) h 38.22 3 10 0,0 0 0 1200 98.80 11.50 (7.40 + 0.88) h 27.97 2 4 4320 200 95.40 25.20 (9.60 + 8.5) h 28.17 5 2488,320 7776 99.70 10.12 (5.18 + 0.62) h 42.69 6 127,776 484 99.60 7.46 (4.70 + 1.10) h 22.25 3 7 107,811 15,309 85.80 5.15 (3.34 + 0.81) h 19.42 8 1331 33 97.50 4.75 (0.70 + 0.72) h 70.11 9 215,622 1764 99.20 4.72 (2.88 + 0.75) h 23.09 C C C D F F G G H H J K K K M – Investigation of other experimental arrays combined with a DEA in a way to improve the efficiency and the effectiveness of the proposed method. – Testing of the proposed procedure in problems involving con- tinuous variables. – Testing of the proposed procedure using other optimizers. – Testing of BiO-MCDEA models to replace the DEA BCC model ( Ghasemi et al., 2014 ). Acknowledgments The authors thank CAPES, CNPq (Process 445632/2014-3, 306214/2015-6, and 431758/2016-6), FAPEMIG (Process TEC-PPM- 00512-15, and APQ-01188-16) and FAPESP (Process 2015/12711-4, and 2015/24560-0) for supporting this research and the anony- mous referees for suggestions that contributed to substantial im- provements in the paper. Supplementary materials Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.ejor.2017.04.016 . References Ahuja, R. K. , Magnanti, T. L. , & Orlin, J. B. (1993). Network Flows . New Jersey: Prentice Hall . Alrabghi, A. , & Tiwari, A. (2015). State of the art in simulation-based optimisation for maintenance systems. Computers & Industrial Engineering, 82 , 167–182 . Andersen, P. , & Petersen, N. C. (1993). A procedure for ranking efficient units in data envelopment analysis. Management Science, 39 , 1261–1264 . Antony, J. (2006). Taguchi or classical design of experiments: A perspective from a practitioner. Sensor Review, 26 , 227–230 . Azadeh, A. , Moghaddam, M. , Asadzadeh, S. M. , & Negahban, A. (2011). An integrated fuzzy simulation-fuzzy data envelopment analysis algorithm for job-shop lay- out optimization: the case of injection process with ambiguous data. European Journal of Operational Research, 214 , 768–779 . Bachelet, B. , & Yon, L. (2007). Model enhancement: improving theoretical optimiza- tion with simulation. Simulation Modelling Practice and Theory, 15 , 703–715 . Bal, H. , Örkcü, H. , & Çelebioglu, S. (2010). Improving the discrimination power and weights dispersion in the data envelopment analysis. Computers & Industrial En- gineering, 37 , 99–107 . Ballantyne, K. N. , Van Oorschot, R. A. , & Mitchell, R. J. (2008). Reduce optimization time and effort: t aguchi experiment al design methods. Forensic Science Interna- tional: Genetics, 1 , 7–8 . Banker, R. D. , Charnes, A. , & Cooper, W. W. (1984). Some models for estimating tech- nical and scale inefficiencies in data envelopment analysis. Management Science, 30 , 1078–1092 . Banker, R. D. , Charnes, A. , Cooper, W. W. , Swarts, J. , & Thomas, D. A. (1989). An introduction to data envelopment analysis with some of its models and their uses. Research in Governmental and Nonprofit Accounting, 5 , 125–163 . Banks, J. , Carson II, J. S. , Nelson, B. L. , & Nicol, D. M. (2009). Discrete-event system simulation . Upper Saddle River: Pearson . Besseris, G. J. (2012). Profiling effects in industrial data mining by non-parametric DOE methods: an application on screening check weighing systems in packag- ing operations. European Journal of Operational Research, 220 , 147–161 . Better, M. , Glover, F. , & Kochenberger, G. (2015). Simulation optimization: improv- ing decisions under uncertainty. In Cox L. A. , Jr. (Ed.), Breakthroughs in decision science and risk analysis (pp. 59–82). New Jersey: John Wiley & Sons . harnes, A. , Cooper, W. W. , & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2 , 429–4 4 4 . ook, W. D. , & Seiford, L. M. (2009). Data Envelopment analysis (DEA) - thirty years on. European Journal of Operational Research, 192 , 1–17 . ooper, W. W. , Sieford, L. M. , & Tone, K. (2007). Data envelopment analysis: A com- prehensive text with models, application, references and DEA-solver software . New York: Springer Science + Business Media . ellino, G. , Kleijnen, J. P. C. , & Meloni, C. (2010). Robust optimization in simulation: taguchi and response surface methodology. International Journal of Production Economics, 125 , 52–59 . Figueira, G. , & Almada-Lobo, B. (2014). Hybrid simulation-optimization methods: a taxonomy and discussion. Simulation Modelling Practice and Theory, 46 , 118–134 . Fu, M. C. (1994). Optimization via simulation: a review. Annals of Operations Re- search, 53 , 199–247 . u, M. C. (2002). Optimization for simulation: theory vs. practice. Journal on Com- puting, 14 , 192–215 . Fu, M. C. (2015). Handbook of simulation optimization . New York: Springer . Fu, M. C. , Andradóttir, S. , Carson, J. S. , Glover, F. , Harrel, C. R. , Ho, Y. C. , Kelly, J. P. , & Robinson, S. M. (20 0 0). Integrating optimization and simulation: research and practice. In J. A. Joines, R. R. Barton, K. Kang, & P. A. Fishwick (Eds.), Proceedings of the 20 0 0 Winter Simulation Conference (pp. 610–616) . u, M. C. , Bayraksan, G. , Henderson, S. G. , Nelson, B. L. , Powell, W. B. , Ryzhov, I. O. , & Thengvall, B. (2014). Simulation optimization: a panel on the state of the art in research and practice. In A. Tolk, S. Y. Diallo, I. O. Ryzhov, L. Yilmaz, S. Buck- ley, & J. A. Miller (Eds.), Proceedings of the 2014 Winter Simulation Conference (pp. 3696–3706) . AMS. The General Algebraic Modeling. (2015). http://www.gams.com/ Accessed 11.11.15. hasemi, M. R. , Ignatius, J. , & Emrouznejad, A. (2014). A bi-objective weighted model for improving the discrimination power in MCDEA. European Journal of Operational Research, 233 (3), 640–650 . arrel, C. R. , Ghosh, B. K. , & Bowden, R. (2004). Simulation using promodel . New York: McGraw-Hill . illier, F. S. , & Lieberman, G. J. (2010). Introduction to operations research . New York: McGraw-Hill . ahangirian, M. , Eldabi, T. , Naseer, A. , Stergioulas, L. K. , & Young, T. (2010). Simula- tion in manufacturing and business: a review. European Journal of Operational Research, 203 , 1–13 . leijnen, J. P. C. (1998). Experimental design for sensitivity analysis, optimization, and validation of simulation models. In J. Banks (Ed.), Handbook of simulation (pp. 173–223). New-York: Wiley . leijnen, J. P. C. (2017). Regression and Kriging metamodels with their experimental designs in simulation: a review. European Journal of Operational Research, 256 , 1–16 . Kleijnen, J. P. C. (2015). Design and analysis of simulation experiments . New York: Springer . leijnen, J. P. C. , Van Beers, W. , & Van Nieuwenhuyse, I. (2010). Constrained opti- mization in simulation: a novel approach. European Journal of Operational Re- search, 202 , 164–174 . Law, A. M. (2015). Simulation modeling and analysis . Boston: McGraw-Hill . Law, A. M. , & McComas, M. G. (2002). Simulation-based optimization. In P. A. Far- rington, H. B. Nembhard, D. T. Sturrock, & G. W. Evans (Eds.), Proceedings of the 1999 Winter Simulation Conference (pp. 56–59) . Lee, L. H. , Chew, E. P. , Teng, S. , & Chen, Y. (2008). Multi-objective simulation-based evolutionary algorithm for an aircraft spare parts allocation problem. European Journal of Operational Research, 189 , 476–491 . Lin, R. C. , Sir, M. Y. , & Pasupathy, K. S. (2013). Multi-objective simulation optimiza- tion using data envelopment analysis and genetic algorithm: specific application to determining optimal resource levels in surgical services. Omega – The Inter- national Journal of Management Science, 41 , 881–892 . Martins, M. S. R. , Fuchs, S. C. , Pando, L. U. , Lüders, R. , & Delgado, M. R. (2013). PSO with path relinking for resource allocation using simulation optimization. Com- puters & Industrial Engineering, 65 , 322–330 . iranda, R. C. , Montevechi, J. A. B. , Silva, A. F. , & Marins, F. A. S. (2014). A new approach to reducing search space and increasing efficiency in simulation opti- mization problems via the Fuzzy-DEA-BCC. Mathematical Problems in Engineer- ing, 2014 , 1–15 . http://dx.doi.org/10.1016/j.ejor.2017.04.016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0001 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0001 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0001 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0001 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0001 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0002 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0002 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0002 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0002 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0003 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0003 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0003 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0003 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0004 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0004 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0005 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0006 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0006 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0006 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0006 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0007 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0007 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0007 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0007 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0007 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0008 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0008 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0008 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0008 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0008 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0009 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0009 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0009 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0009 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0009 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0010 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0011 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0012 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0012 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0013 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0013 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0013 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0013 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0013 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0014 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0014 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0014 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0014 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0014 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0015 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0015 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0015 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0015 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0016 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0017 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0017 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0017 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0017 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0017 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0018 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0018 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0018 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0018 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0019 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0019 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0020 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0020 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0021 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0021 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0022 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0023 http://www.gams.com/ http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0024 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0024 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0024 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0024 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0024 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0025 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0025 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0025 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0025 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0025 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0026 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0026 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0026 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0026 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0027 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0028 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0028 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0029 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0029 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0030 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0030 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0031 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0031 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0031 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0031 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0031 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0032 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0032 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0033 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0033 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0033 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0033 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0034 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0035 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0035 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0035 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0035 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0035 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0036 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0037 R.d.C. Miranda et al. / European Journal of Operational Research 262 (2017) 673–681 681 M M N P P R R S S S S T T T T W X ontevechi, J. A. B. , Miranda, R. C. , & Friend, J. D. (2012). Sensitivity analysis in discrete event simulation using design of experiments. In E. W. C. Lim (Ed.), Discrete event simulations - development and applications (pp. 63–102). Rijeka In- Tech . ontgomery, D. C. (2009). Design and analysis of experiments (7th ed). Hoboken: Wiley . elson, B. L. (2010). Optimization via simulation over discrete decision variables. In J. J. Hasenbein (Ed.), Tutorials in operations research: Risk and optimization in an uncertain world (pp. 193–207). Hanover: INFORMS . asupathy, R. , & Ghosh, S. (2014). Simulation optimization: a concise overview and implementation guide. In H. Topaloglu (Ed.), Tutorials in operations research: The- ory driven by influential applications (pp. 122–150). Catonsville: Informs . roModel. ProModel Corporation. (2016). https://www.promodel.com/ Accessed 12.12.16. oss, P. J. (1996). Taguchi techniques for quality engineering (2nd ed., p. 329). New York: McGraw-Hill . oy, R. K. (2010). A primer on the Taguchi method . Dearborn: Society of Manufactur- ing Engineers . hen, H. , & Wan, H. (2009). Controlled sequential factorial design for simulation factor screening. European Journal of Operational Research, 198 , 511–519 . iegmund, F. , Bernedixen, J. , Pehrsson, L. , Ng, A. H. C. , & Deb, K. (2012). Refer- ence point-based evolutionary multi-objective optimization for industrial sys- tems simulation. In C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, & A. M. Uhrmacher (Eds.), Proceedings of the 2012 Winter Simulation Conference (pp. 1–11) . imrunner User Guide. (2002). ProModel Corporation. Orem: ProModel. wisher, J. R. , Hyden, P. D. , Jacobson, S. H. , & Schruben, L. W. (20 0 0). A survey of simulation optimization techniques and procedures. In J. A. Joines, R. R. Barton, K. Kang, & P. A. Fishwick (Eds.), Proceedings of the 20 0 0 Winter Simulation Con- ference (pp. 119–128) . aguchi, G. (1987). System of experimental design: Engineering methods to optimize quality and minimize costs . Dearborn: UNIPUB/Kraus International Publications . aguchi, G. , Chowdhury, S. , & Wu, Y. (2005). Taguchi’s quality engineering handbook . New Jersey: John Wiley & Sons, Inc . amiz, M. , Jones, D. F. , & El-Darzi, E. (1995). A review of goal programming and its applications. Annals of Operations Research, 58 (1), 39–53 . enne, Y. , & Goh, C-K. (2010). Computational intelligence in expensive optimization problems . Berlin: Springer . illis, K. O. , & Jones, D. F. (2008). Multi-objective simulation optimization through search heuristics and relational database analysis. Decision Support Systems, 46 , 277–286 . u, J. , Huang, E. , Chen, C-H. , & Lee, L. H. (2015). Simulation optimization: A review and exploration in the new era of cloud computing and big data. Asia-Pacific Journal of Operational Research, 32 (3), 1–34 . http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0038 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0038 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0038 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0038 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0038 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0039 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0039 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0040 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0040 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0041 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0041 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0041 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0041 https://www.promodel.com/ http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0042 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0042 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0043 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0043 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0044 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0044 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0044 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0044 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0045 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0046 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0047 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0047 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0048 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0048 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0048 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0048 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0048 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0049 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0049 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0049 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0049 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0049 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0050 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0050 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0050 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0050 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0051 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0051 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0051 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0051 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 http://refhub.elsevier.com/S0377-2217(17)30349-1/sbref0052 Increasing the efficiency in integer simulation optimization: Reducing the search space through data envelopment analysis and orthogonal arrays 1 Introduction 2 Literature review 2.1 Simulation optimization 2.2 Data envelopment analysis (DEA) 2.3 Orthogonal arrays 3 Proposed simulation optimization method 4 Application of the optimization procedure, results, and discussion 4.1 First class of simulation optimization problems 4.1.1 Case study 1 4.2 Other classes of problems 5 Conclusions and recommendations for future studies Acknowledgments Supplementary materials References