An Empirical study of Binary Neural Networks' Optimisation

Gradient clipping stops gradient flow if the weight's magnitude is larger than 1.
This effectively means gradients are computed with respect to hard tanh function.

Weight clipping is applied to weights after gradients have been applied to keep them within a range.

(1) history-free optimisers such as mini-batch SGD that do not take previous jumps or gradients into account, 
(2) momentum optimisers that maintain and use a running average of previous jumps such as Momentum
  and Nesterov (Sutskever et al., 2013), 
(3) Adaptive optimisers that adjust learning rate for each parameter separately such as AdaGrad 
(Duchi et al., 2011) and AdaDelta (Zeiler, 2012), and finally, 
(4) optimisers that combine elements from categories above such as ADAM which combines momentum with 
adaptive learning rate.

Our first observation is that vanilla SGD generally fails in optimising binary models using STE. We note that 
reducing SGD's stochasticity (by increasing batch size) improves performance initially. However, it still fails 
to obtain the best possible accuracy. SGD momentum and Nesterov optimisers perform better than SGD when they are
carefully fine-tuned. However, they perform significantly slower compared to optimsing non-binary models and have 
to be used for many more epochs than normally used for CIFAR-10 and MNIST datasets. Similar to SGD, increasing 
momentum rate improves training speed significantly but results in worse final model accuracy.

A possible hypothesis is that early stages of training binary models require more averaging for the
optimiser to proceed in presence of binarisaton operation. On the other hand, in the late stages of the
training, we rely on noisier sources to increase exploration power of the optimiser. This is reinforced
by our observation that binary models are often trained long after the training or validation accuracy
stop showing improvements. Reducing the learning rate in these epochs does not improve things
either. Yet, the best validations are often found in these epochs. In other words, using early stopping
for training binary models would terminate the training early on and would result in suboptimal
accuracies.

Reducing the momentum rate in BN can help to cancel the effect of long training. The effect is small 
but consistent.

In BinaryConnect Courbariaux et al. (2015) propose scaling
learning rates of each convolutional or fully connected layer by the inverse of Xavier initialisation's
variance value. The same value is also used as the range in weight clipping after gradient update.

While weight and gradient clipping help achieve better accuracy, our hypothesis is that they are only required 
in the later stages of training where the noise added by clipping weights and gradients increases the exploration 
of the optimiser.

while we can quickly get to the point where training and validation accuracies stagnate, there is a small gap 
between the achieved accuracy and the best possible one. This gap can only be filled by continuing training for 
many epochs.

last mile of model performance has little dependence on the STE's capability and mostly relies on a 
stochastic exploration of the parameter space