4.

6 DEBUGGING STRATEGIES

 When a machine learning system performs poorly, it is usually difficult to tell

 Whether the poor performance is intrinsic to the algorithm itself or

 Whether there is a bug in the implementation of the algorithm.

 Machine learning systems are difficult to debug for a variety of reasons. In most cases
we do not know a priori what the intended behavior of the algorithm is.

 In fact the entire point of using machine learning is that it will discover useful
behavior that we were not able to specify ourselves.

 If we train a neural network on a new classification task and it achieves 5% test error,
we have no straightforward way of knowing if this is the expected behavior or sub-
optimal behavior.

 A further difficulty is that most machine learning models have multiple parts that are
each adaptive. If one part is broken, the other parts can adapt and still achieve roughly
acceptable performance.

For example :-

 Suppose that we are training a neural net with several layers parametrized by weights
W and biases b.

 Suppose further that we have manually implemented the gradient descent rule for
each parameter separately, and we made an error in the update for the biases:

b←b–α

 where α is the learning rate. This erroneous update does not use the gradient at all. It
causes the biases to constantly become negative throughout learning, which is clearly
not a correct implementation of any reasonable learning algorithm.

 The bug may not be apparent just from examining the output of the model
though.

 Depending on the distribution of the input, the weights may be able to adapt to
compensate for the negative biases.
  Most debugging strategies for neural nets are designed to get around one or both of
these two difficulties.

 Either we design a case that is so simple that the correct behavior actually can
be predicted or

 we design a test that exercises one part of the neural net implementation in
isolation. Some important debugging tests include:

Visualize the model in action :

 When training a model to detect objects in images, view some images with the
detections proposed by the model displayed superimposed on the image.

 When training a generative model of speech, listen to some of the speech samples it
produces.

 Quantitative performance measurements like accuracy or log-likelihood.

 Directly observing the machine learning model performing its task will help to
determine whether the quantitative performance numbers it achieves seem reasonable.

 Evaluation bugs can be some of the most devastating bugs because they can mislead
you into believing your system is performing well when it is not.

Visualize the worst mistakes :

o Most models are able to output some sort of confidence measure for the task they
perform.

o For example: Classifiers based on a softmax output layer assign a probability to

each class.

o The probability assigned to the most likely class thus gives an estimate of the
confidence the model has in its classification decision.

o Typically maximum likelihood training results in these values being overestimates

rather than accurate probabilities of correct prediction, but they are somewhat useful
in the sense that examples that are actually less likely to be correctly labeled receive
smaller probabilities under the model.

o By viewing the training set examples that are the hardest to model correctly, one can
often discover problems with the way the data has been preprocessed or labeled.
Reasoning about software using train and test error:

o It is often difficult to determine whether the underlying software is correctly

implemented. Some clues can be obtained from the train and test error.

o If training error is low but test error is high, then it is likely that that the training
procedure works correctly, and the model is overfitting for fundamental algorithmic
reasons.

o An alternative possibility is that the test error is measured incorrectly due to a

problem with saving the model after training then reloading it for test set evaluation,
or if the test data was prepared differently from the training data.

o If both train and test error are high, then it is difficult to determine whether there is a
software defect or whether the model is underfitting due to fundamental algorithmic
reasons.

Fit a tiny dataset:

 If you have high error on the training set, determine whether it is due to genuine
underfitting or due to a software defect.

 Usually even small models can be guaranteed to be able fit a sufficiently small
dataset.

 For example:-

o A classification dataset with only one example can be fit just by setting the
biases of the output layer correctly.

o Usually if you cannot train a classifier to correctly label a single example

 An autoencoder to successfully reproduce a single example with high

fidelity, or

 A generative model to consistently emit samples resembling a single

example, there is a software defect preventing successful optimization on
the training set. This test can be extended to a small dataset with few
examples.
Compare back-propagated derivatives to numerical derivatives:

o If you are using a software framework that requires you to implement your own
gradient computations, or if you are adding a new operation to a differentiation library
and must define its bprop method, then a common source of error is implementing
this gradient expression incorrectly.

o One way to verify that these derivatives are correct is to compare the derivatives
computed by your implementation of automatic differentiation to the derivatives
computed by a finite differences. Because

 The perturbation size must chosen to be large enough to ensure that the perturbation is
not rounded down too much by finite-precision numerical computations.

 Usually, we will want to test the gradient or Jacobian of a vector-valued function g :

R m→ R n .

 Finite differencing only allows us to take a single derivative at a time.

o We can either run finite differencing mn times to evaluate all of the partial
derivatives of g, or

o we can apply the test to a new function that uses random projections at both
the input and output of g.
  For example:-

o we can apply our test of the implementation of the derivatives to f(x) where f
(x) = u T g(vx), where u and v are randomly chosen vectors.

o Computing f ‘(x) correctly requires being able to back-propagate through g

correctly, yet is efficient to do with finite differences because f has only a
single input and a single output.

 If one has access to numerical computation on complex numbers, then there is a very
efficient way to numerically estimate the gradient by using complex numbers as input
to the function.

 The method is based on the observation that

Monitor histograms of activations and gradient:

 It is often useful to visualize statistics of neural network activations and gradients,

collected over a large amount of training iterations (maybe one epoch).

 The pre-activation value of hidden units can tell us if the units saturate, or how often
they do.

 In a deep network where the propagated gradients quickly grow or quickly vanish,
optimization may be hampered.

 Finally, it is useful to compare the magnitude of parameter gradients to the magnitude

of the parameters themselves.