Dying ReLU and Initialization: Theory and Numerical Examples (Communications in Computational Physics, 2020)
The dying ReLU refers to the problem when ReLU neurons become inactiveand only output 0 for any input. There are many empirical and heuristic explanationsof why ReLU neurons die. However, little is known about its theoretical analysis. Inthis paper, we rigorously prove that a deep ReLU network will eventually die in probability as the depth goes to infinite. Several methods have been proposed to alleviatethe dying ReLU. Perhaps, one of the simplest treatments is to modify the initialization procedure. One common way of initializing weights and biases uses symmetricprobability distributions, which suffers from the dying ReLU. We thus propose a newinitialization procedure, namely, a randomized asymmetric initialization. We showthat the new initialization can effectively prevent the dying ReLU. All parameters required for the new initialization are theoretically designed. Numerical examples areprovided to demonstrate the effectiveness of the new initialization procedure.