First-Order Optimization in p-Norm Geometry: Theory Inspired by Modern Deep Learning Optimizers

Abstract: Motivated by the growing popularity of the Muon optimizer for deep learning, this talk explores how norm geometry can guide the design and analysis of first-order optimization methods. I will first introduce Schattor, a family of Schatten-norm methods that unifies the Euclidean gradient updates of SGD with Muon-style spectral gradient updates. I will also present a martingale moment inequality for controlling the accumulation of stochastic errors. I will then turn to convex optimization and introduce an adaptive accelerated optimistic dual averaging method in p-norm geometry, which achieves accelerated convergence under local smoothness. Finally, I will present numerical results demonstrating the practical advantages of p-norm-based optimization methods.