Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks (CSIAM Transactions on Applied Mathematics, 2024)
In this work, we delve into the relationship between deep and shallow neural networks (NNs), focusing on the critical points of their loss landscapes. We discoveran embedding principle in depth that loss landscape of an NN “contains” all criticalpoints of the loss landscapes for shallower NNs. The key tool for our discovery is thecritical lifting that maps any critical point of a network to critical manifolds of anydeeper network while preserving the outputs. To investigate the practical implicationsof this principle, we conduct a series of numerical experiments. The results confirmthat deep networks do encounter these lifted critical points during training, leading tosimilar training dynamics across varying network depths. We provide theoretical andempirical evidence that through the lifting operation, the lifted critical points exhibitincreased degeneracy. This principle also provides insights into the optimization benefits of batch normalization and larger datasets, and enables practical applications likenetwork layer pruning. Overall, our discovery of the embedding principle in depthuncovers the depth-wise hierarchical structure of deep learning loss landscape, whichserves as a solid foundation for the further study about the role of depth for DNNs.