On the Banach Spaces Associated with Multi-Layer ReLU Networks: Function Representation, Approximation Theory and Gradient Descent Dynamics (CSIAM Transactions on Applied Mathematics, 2020)
We develop Banach spaces for ReLU neural networks of finite depth $L$ andinfinite width. The spaces contain all finite fully connected $L$-layer networks and their $L^2$-limiting objects under bounds on the natural path-norm. Under this norm, theunit ball in the space for $L$-layer networks has low Rademacher complexity and thusfavorable generalization properties. Functions in these spaces can be approximated bymulti-layer neural networks with dimension-independent convergence rates.The key to this work is a new way of representing functions in some form of expectations, motivated by multi-layer neural networks. This representation allows us todefine a new class of continuous models for machine learning. We show that the gradient flow defined this way is the natural continuous analog of the gradient descentdynamics for the associated multi-layer neural networks. We show that the path-normincreases at most polynomially under this continuous gradient flow dynamics.