Extremal Random Forests

Abstract
Classical methods for quantile regression fail in cases where the quantile of interest is extreme and only few or no training data points exceed it. Asymptotic results from extreme value theory can be used to extrapolate beyond the range of the data, and several approaches exist that use linear regression, kernel methods or generalized additive models. Most of these methods break down if the predictor space has more than a few dimensions or if the regression function of extreme quantiles is complex. We propose a method for extreme quantile regression that combines the flexibility of random forests with the theory of extrapolation. Our extremal random forest (ERF) estimates the parameters of a generalized Pareto distribution, conditional on the predictor vector, by maximizing a local likelihood with weights extracted from a quantile random forest. Under certain assumptions, we show consistency of the estimated parameters. Furthermore, we penalize the shape parameter in this likelihood to regularize its variability in the predictor space. Simulation studies show that our ERF outperforms both classical quantile regression methods and existing regression approaches from extreme value theory. We apply our methodology to extreme quantile prediction for U.S. wage data.

About Nicola Gnecco
Nicola Gnecco is a postdoctoral researcher at the Copenhagen Causality Lab at the University of Copenhagen. His research interests are causal inference, distribution generalization, and extreme value theory. He is currently working on distributional generalization problems from a causal perspective to develop predictive methods that generalize to unseen environments. He is supervised by Jonas Peters and Niklas Pfister. He received his PhD from the University of Geneva under the supervision of Sebastian Engelke and a master’s degree in Statistics from ETH Zurich under the supervision of Nicolai Meinshausen.