Optimization for machine learning: algorithms, convergence, and applications

My research concerns the design and theoretical analysis of optimization algorithms motivated by machine learning. In this talk, I will give an overview of several directions of this work, connected by a common question: how can we obtain reliable algorithms when computations are large scale, noisy, or inexact? I will start with proximal and stochastic methods for convex optimization. This includes randomized block-coordinate algorithms, stochastic subgradient methods under heavy-tailed noise, and accelerated proximal methods. I will also present a recent work on adaptive methods, examining both a new adaptation rule based on successive gradient differences and a limitation of classical AdaGrad for composite objectives. This direction promises to reduce the burden of setting algorithmic parameters properly and, ultimately, to provide universal algorithms capable of working efficiently in several scenarios. The second part of the talk will concern computational optimal transport. I will briefly describe algorithms for computing Sinkhorn barycenters with free support and for solving regularized multimarginal transport problems, along with the convergence questions they raise. I will also mention some recent algorithmic approaches to unbalanced optimal transport. In the third part, I will discuss bilevel optimization, where an optimization problem or fixed-point equation defines part of a larger model. I will explain how to approximate derivatives of implicitly defined solutions, including in nonsmooth and stochastic settings. Finally, if time permits, I will discuss kernel methods beyond the Hilbert space setting. In particular, I will describe how regularized learning in Banach spaces leads to generalized representer theorems and how, for suitable l^p-type regularization, tensor kernels provide a computational framework analogous to classical kernel methods.