Stepwise Regression Ensembling

Abstract:

Ensembling is a powerful approach to model complex relations in high-dimensional data and yield accurate predictions. However, it is not obvious which models are best combined in an ensemble. Standard approaches are to use a predefined set of models or to use some sort of randomness to build models. We propose a data-driven approach in which the different models are grown simultaneously. The candidate variables are added to the models in a stepwise manner. Variables are combined in a single model if they work well together. Otherwise they are assigned to different models. The resulting ensemble likely overfits the data. Therefore, we regularize each of the models using lasso or elastic net penalties. These models are then combined in an ensemble to obtain the final fit. We show the performance of the method using simulations and real data examples and compare it to existing approaches.

Joint work with Anthony Cristidis and Ruben Zamar