DropConnect Regularization Method with Sparsity Constraint for Neural Networks

DropConnect is a recently introduced algorithm to prevent the co-adaptation of feature detectors. Compared to Dropout, DropConnect gains state-of-the-art results on several image recognition benchmarks. Motivated by the success of DropConnect, we extended this algorithm with the ability of sparse feature selection. In DropConnect algorithm, the dropping masks of weights are generated using Bernoulli gating variables that are independent of the weights and activations. We introduce a new strategy to generate masks depending on the outputs of previous layer. Using this method, neurons which are promising to produce sparser features will be assigned a bigger possibility to keep active in the forward and backward propagations. We then evaluate such sparsity constrained DropConnect on MNIST and CIFAR datasets in comparison with ordinary DropConnect and Dropout method. The results show that our new method improves the sparsity of features significantly, while not degrading the precision.