Exploring Infinite-Width Limit of Deep Neural Networks
Abstract
Math Machine Learning seminar MPI MiS + UCLA
The Math Machine Learning seminar MPI MiS + UCLA is an online seminar series (via Zoom) organized by Guido Montúfar's research group at the Max Planck Institute for Mathematics in the Sciences and UCLA. The seminar is usually Thursdays (sometimes also Fridays) at 5 pm (CEST) (GMT+2; PDT: 8-9 am). The talks are about 50 minutes with time for questions and discussion.
In this talk, I will discuss our research on understanding the infinite-width limit of neural networks. In this limit, neural networks correspond to Neural Network Gaussian Processes (NNGPs) and Neural Tangent Kernels (NTKs). I will first describe our empirical study exploring the relationship between wide neural networks and neural kernel methods. Our study resolves a variety of open questions related to infinitely wide neural networks and opens up new interesting questions. In the second half of the talk, I will discuss our recent work on scaling up infinite-width neural kernel methods to millions of data points. There are unique challenges in scaling up neural kernel methods, and I will talk about our attempts to overcome them. If there is time, I will discuss some applications of the infinite-width limit of neural networks, such as dataset distillation, neural architecture search, uncertainty quantification, and neural scaling laws.
If you want to participate in this video broadcast please register using this special form. The (Zoom) link for the video broadcast will be sent to your email address one day before the seminar.