S-Race: A Multi-Objective Racing Algorithm

This paper presents a multi-objective racing algorithm, S-Race, which efficiently addresses multi-objective model selection problems in the sense of Pareto optimality. As a racing algorithm, S-Race attempts to eliminate candidate models as soon as there is sufficient statistical evidence of their inferiority relative to other models with respect to all objectives. This approach is followed in the interest of controlling the computational effort. S-Race adopts a non-parametric sign test to identify pair-wise domination relationship between models. Meanwhile, Holm's Step-Down method is employed to control the overall family-wise error rate of simultaneous hypotheses testing during the race. Experimental results involving the selection of superior Support Vector Machine classifiers according to 2 and 3 performance criteria indicate that S-Race is an efficient and effective algorithm for automatic model selection, when compared to a brute-force, multi-objective selection approach.