Dilution is a regularization technique for reducing overfitting in artificial neural networks by preventing complex co-adaptations on training data. It is an efficient way of performing model averaging with neural networks. The term dilution refers to the thinning of the weights. The term dropout refers to randomly "dropping out", or omitting, units during the training process of a neural network. Both the thinning of weights and dropping out units trigger the same type of regularization, and often the term dropout is used when referring to the dilution of weights.
Types and uses
Dilution is usually split in weak dilution and strong dilution. Weak dilution describes the process in which the finite fraction of removed connections is small, and strong dilution refers to when this fraction is large. There is no clear distinction on where the limit between strong and weak dilution is, and often the distinction is meaningless, although it has implications for how to solve for exact solutions. Sometimes dilution is used for adding damping noise to the inputs. In that case, weak dilution refers to adding a small amount of damping noise, while strong dilution refers to adding a greater amount of damping noise. Both can be rewritten as variants of weight dilution. These techniques are also sometimes referred to as random pruning of weights, but this is usually a non-recurring one-way operation. The network is pruned, and then kept if it is an improvement over the previous model. Dilution and dropout both refer to an iterative process. The pruning of weights typically does not imply that the network continues learning, while in dilution/dropout, the network continues to learn after the technique is applied.
Output from a layer of linear nodes, in an artificial neural net can be described as This can be written in vector notation as This previous two equations are crucial for the interpretation of the following ones, as they refer to the same variables.
Weak dilution
During weak dilution, the finite fraction of removed connections is small, giving rise to a tiny uncertainty. This edge-case can be solved exactly with mean field theory. In weak dilution the impact on the weights can be described as The interpretation of probability can also be changed from keeping a weight into pruning a weight. In vector notation this can be written as where the function imposes the previous dilution. In weak dilution only a small and fixed fraction of the weights are diluted. When the number of terms in the sum goes to infinite it is still infinite, thus mean field theory can be applied. In the notation from Hertzet al. this would be written as There are some assumptions for this to hold, which are not listed here.
Strong dilution
When the dilution is strong, the finite fraction of removed connections is large, giving rise to a huge uncertainty.
Dropout
Dropout is a special case of the previous weight equation, where the aforementioned equation is be adjusted to remove a whole row in the vector matrix, and not only random weights Because dropout removes a whole row from the vector matrix, the previous assumptions for weak dilution and the use of mean field theory are not applicable. The process by which the node is driven to zero, whether by setting the weights to zero, by “removing the node”, or by some other means, does not impact the end result and does not create a new and unique case. If the neural net is processed by a high-performance digital array-multiplicator, then it is likely more effective to drive the value to zero late in the process graph. If the net is processed by a constrained processor, perhaps even an analog neuromorph processor, then it is likely a more power-efficient solution is to drive the value to zero early in the process graph.
Google's patent
Although there have been examples of randomly removing connections between neurons in a neural network to improve models, this technique was first introduced with the name dropout by Geoffrey Hinton, et al. in 2012. Google currently holds the patent for the dropout technique.
