Deep Learning

Overview

• Deep Learning is a newer architecture of neural networks that has come in to favor as of recent years (starting with Hinton 2006) . A Deep Network consists of (usually) 3 or more layers that each have a binomial sampling component as a way of doing feature extraction followed by a Logistic Regression Softmax Layer (there are other examples that will be described later). See Denoising AutoEncoders or Restricted Boltzmann Machines as the typical examples. Their multilayer counterparts counterparts are Stacked Denoising AutoEncoders and Deep Belief Networks respectively.

• Training - 2 phases * Pretraining: feature extraction via sampling by minimizing reconstruction error. Each layer is trained with a supervised objective function of the input minimizing reconstruction error.

  * Finetuning: From there stochastic gradient descent/LBFGS/Conjugate Gradient are run as a way of updating the weights on the final Logistic Regression output layer. The Logistic Regression layer in this case is learning from the features extracted via unsupervised means in the previous layers. From there you can run a classifier.
  

Resources

• Hinton Guide on Training RBMs: http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
• Stanford Papers: http://ufldl.stanford.edu/?papers
• DeepLearning.net: http://deeplearning.net/reading-list/
• UFLDL @ Stanford: http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial
  • NN Vectorization: http://ufldl.stanford.edu/wiki/index.php/Neural_Network_Vectorization
• Bengio Book: http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf
• Tutorials on deep learning: http://deeplearning.net/tutorial/
• Neural Networks Class by Hinton: https://class.coursera.org/neuralnets-2012-001/lecture

Use Cases

• Recommendation
  • http://www.machinelearning.org/proceedings/icml2007/papers/407.pdf
• Classification
• Regression
• Pretraining
  • http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/35536.pdf
• Text
  • http://nlp.stanford.edu/courses/NAACL2013/
• Training Strategies
  • http://machinelearning.wustl.edu/mlpapers/paper_files/jmlr10_larochelle09a.pdf