Skip to content

15. Training Neural Networks

Jump to bottom
Antonio Erdeljac edited this page Mar 31, 2019 · 1 revision

Training Neural Networks

Topic: Training Neural Networks

Course: GMLC

Date: 31 March 2019   

Professor: Not specified

Resources

Key Points

  • Backpropagation

    • Algorithm used to train Neural Networks

    • Performs gradient descent

    • Steps:

      • Calculates outputs of all nodes in a forward pass

      • Calculates error rate using derivatives of the error with each parameter

  • Failure cases

    • Vanishing gradients

      • Gradients vanish towards 0 and train very slowly or not at all

      • Rectified Linear Unit activation function can prevent the problem

    • Exploding gradients

      • Large weights can cause the lower layers of the network to involve too many products, thus they get too large to converge

      • Lowering the learning rate or batch normalization prevents the problem

    • Dead ReLU units

      • Weighed sum of ReLU activation function can fall below 0, thus not contributing to neural network at all

      • Lowering the learning rate prevents dead ReLU units

  • Dropout regularization

    • Works by randomly dropping out unit activations 

Check your understanding

  • Explain how backpropagation works

  • Be aware of NN failure cases

Summary of Notes

  • Neural networks are trained using  backpropagation algorithm which sums the outputs of each node and runs them through derivatives for error calculation & through activation functions for learning
Clone this wiki locally