Convergence of Online Gradient Method with Penalty for BP Neural Networks (Communications in Mathematical Research, 2021)
Online gradient method has been widely used as a learning algorithm fortraining feedforward neural networks. Penalty is often introduced into the trainingprocedure to improve the generalization performance and to decrease the magnitudeof network weights. In this paper, some weight boundedness and deterministic convergence theorems are proved for the online gradient method with penalty for BPneural network with a hidden layer, assuming that the training samples are suppliedwith the network in a fixed order within each epoch. The monotonicity of the errorfunction with penalty is also guaranteed in the training iteration. Simulation resultsfor a 3-bits parity problem are presented to support our theoretical results.