Variational splines and stochastic processes: From kernel methods (and Norbert Wiener) to ReLU networks

The Wiener (MMSE) estimator of a Brownian motion (a.k.a. Wiener process) from noisy samples is a linear spline. Similarly, a variational reconstruction problem with second-order total variation regularization yields an adaptive linear spline solution. Deep ReLU networks can also be interpreted as spline functions, albeit in higher-dimensional spaces. In this talk, I will show how these seemingly disparate results are unified through a common variational framework. I will further relate this perspective to stochastic modeling: kernel methods naturally correspond to Gaussian processes, while neural network approaches can be interpreted in terms of sparse stochastic processes.