Adam: A method for stochastic optimization

[standing-o]

Adam: A method for stochastic optimization

• Authors : Kingma, Diederik P and Ba, Jimmy
• Journal : arXiv preprint arXiv
• Year : 2014
• Link : https://arxiv.org/pdf/1412.6980.pdf

Abstract

• Adam is...
  ➔ Algorithm for first-order gradient-based optimization of stochastic objective functions based on adaptive estimates of lower-order moments.
  ➔ Straightforward to implement, computationally efficient, little memory requirements, invariant to diagonal rescaling of the gradients, well suited for problems that are large in terms of data or params.
  ➔ Appropriate for non-stationary objectives and problems with very noisy and sparse gradients
  ➔ Its hyper params have intuitive interpretations and typically require little tuning.

Introduction

• If an objective function requiring maximization or minimization with respect to its parameters is differentiable, gradient descent is a relatively efficient optimization method, since the computation of first-order partial derivatives is of the same computational complexity as just evaluating the function.
• Adam computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients.
  ➔ designed to combine the advantages of AdaGrad and RMSProp.
  ➔ Advantages : magnitudes of param updates are invariant to rescaling of the gradient, its stepsizes are approximately bounded by the stepsize hyperparameters.

Adam algorithm pseudo-code

Adam algorithm

• The algorithm updates exponential moving averages of the gradient (m) and The squared gradient (v) where the hyper-params β₁, β₂ control the exponential decay rates of these moving averages.
• These moving averages are initialized as 0, leading to moment estimates that are biased towards zero, especially during the initial timesteps, and especially when the decay rates are small (i.e β are close to 1)
  

Initialization bias correction

• Let us initialize the exponential moving average as v₀ = 0, then v_t can be written as a function of the gradients at all previous timesteps:
  
  
  where ζ = 0 if the true second moment E[g_i²] is stationary (can be kept small).
• we divide by (1-β₂^t to correct the initialization bias.

Experiment

• Logistic regression training negative log likelihood on MNIST images
  
• Training of multilayer neural networks on MNIST images
  
• Convolutional neural networks training cost on CIFAR10