Deep-Learning-for-NLP-Resources

List of resources to get started with Deep Learning for NLP. (Updated incrementally)

Deep Learning (general + NLP) links:

1. https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7TqghAJ6NAPrNmUBH : This lecture series has very good introduction to Neural Network and Deep Learning.

2. https://www.coursera.org/course/neuralnets : This lecture series is from Geof Hinton. The concepts explained are bit abstract, concepts are hard to understand in first go. Generally people recommend these lectures as starting point but I am skeptical about it. I would suggest going through 1st one before this.

3. https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu : Deep Learning Lectures from Oxford University

4. https://www.iro.umontreal.ca/~lisa/pointeurs/TR1312.pdf : This is a short book on Deep Learning written by Yoshua Bengio. It deals with theoritical aspects related to Deep Architectures. Great book though.

5. http://www.deeplearningbook.org/ : This web page has a book draft written by Yoshua Bengio and Ian Goodfellow. Later person is author of Theano library. This is holy bible on Deep Learning.

6. http://cs231n.stanford.edu/ : Deep Learning for Vision by Stanford. Good lectures by Andrej Karpathy on introduction to DL (some initial lectures)

7. http://videolectures.net/yoshua_bengio/ : Video Lectures By Yoshua Bengio on Theoritical Aspects of Deep Learning. They are counterparts of resource [4].

8. http://videolectures.net/geoffrey_e_hinton/ : Video Lectures by the GodFather Geoffrey Hinton on introduction to Deep Learning and some advanced stuff too.

9. https://github.com/ChristosChristofidis/awesome-deep-learning : Good collection of resources.

10. http://deeplearning.net/reading-list/ : Reading resources

11. http://www.cs.toronto.edu/~hinton/csc2515/deeprefs.html : Reading list by Hinton

12. http://videolectures.net/mlss05us_lecun_ebmli/ : Intro to Energy based model by Yann Lecunn.

13. http://videolectures.net/kdd2014_bengio_deep_learning/?q=ICLR# : Yoshua Bengio's lecture series recorded in KDD' 14.

14. http://videolectures.net/nips09_collobert_weston_dlnl/ : Ronan Collobert lecture (it's quite old new, from 2008 but I think it is still useful).

15. https://www.youtube.com/watch?v=eixGKz0Asr8 : Lecture series by Chris Manning and Richard Socher given at NAACL 2013

16. https://www.youtube.com/watch?v=AmG4jzmBZ88 : Lecture series for DL4NLP with some practical guidelines.

17. https://blog.wtf.sg/2014/08/24/nlp-with-neural-networks/ : Blogpost on some DL applications.

18. http://lamda.nju.edu.cn/weixs/project/CNNTricks/CNNTricks.html : Some useful tricks for training Neural Networks

19. http://cs224d.stanford.edu/lectures/CS224d-Lecture11.pdf : Short notes on backprop and word embeddings

20. http://cilvr.nyu.edu/doku.php?id=courses:deeplearning2014:start : A course by Yann Lecunn on Deep Learning taught at NYU.

21. http://cs224d.stanford.edu/ : Course Specifically designed for DEEP LEARNING FOR NLP

22. https://devblogs.nvidia.com/parallelforall/understanding-natural-language-deep-neural-networks-using-torch/#.VPYhS2vB09E.reddit : NLP using Torch

23. http://www.kyunghyuncho.me/home/courses/ds-ga-3001-fall-2015 : Natural Language Understanding with Distributed Representations

24. http://mlwave.com/kaggle-ensembling-guide/ : ENSEMBLING guide. Very useful for designing practical ML systems

25. http://joanbruna.github.io/stat212b/ : TOPIC COURSE IN DEEP LEARNING by Joan Brune, UC Berkley Stats Department

26. https://medium.com/@memoakten/selection-of-resources-to-learn-artificial-intelligence-machine-learning-statistical-inference-23bc56ba655#.s5kjy7bgo : LIST of Deep Learning Talk

Deep Learning for Information Retrieval Links:

There are two very good survey papers on using Deep Learning for Information Retrieval. There reference section in these articles is an exhaustive list (I think) for IR using DL.

1. https://arxiv.org/abs/1611.03305 : Getting started with Neural Models for Semantic Matching in Web Search

2. https://arxiv.org/pdf/1611.06792.pdf : Neural Information Retrieval: A Literature Review

3. http://www.slideshare.net/BhaskarMitra3/neural-text-embeddings-for-information-retrieval-wsdm-2017 : WSDM'17 Tut. on Deep Learning for IR

Word Embeddings related articles

1. https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/index.html : Tensorflow tutorial on word2vec

2. http://textminingonline.com/getting-started-with-word2vec-and-glove : Intro to word2vec and glove

3. http://rare-technologies.com/deep-learning-with-word2vec-and-gensim/ : Getting starting with word2vec and gensim.

4. http://www.lab41.org/anything2vec/ : Great explaination of word2vec and it's relation to neural networks

5. http://www.offconvex.org/2015/12/12/word-embeddings-1/ : Intuition on word embedding methods

6. http://www.offconvex.org/2016/02/14/word-embeddings-2/ : Explains the mathy stuff behind word2vec and glove (Also contains some links pointing to some other good articles on word2vec)

7. http://textminingonline.com/getting-started-with-word2vec-and-glove-in-python : Getting started with glove and word2vec with python

8. http://www.foldl.me/2014/glove-python/ : Glove implementation details in python

9. http://videolectures.net/kdd2014_salakhutdinov_deep_learning/ : Tutorial by Ruslan

10. http://www.openu.ac.il/iscol2015/downloads/ISCOL2015_submission25_e_2.pdf : Comparing various word embedding models

11. http://clic.cimec.unitn.it/marco/publications/acl2014/baroni-etal-countpredict-acl2014.pdf : Comparision between word2vec and glove

12. https://levyomer.files.wordpress.com/2014/09/neural-word-embeddings-as-implicit-matrix-factorization.pdf : word2vec as matrix factorization

13. http://research.microsoft.com/pubs/232372/CIKM14_tutorial_HeGaoDeng.pdf : Tutorial by Microsoft on DL for NLP at CIKM '14

14. http://blog.aidangomez.ca/2016/04/17/Backpropogating-an-LSTM-A-Numerical-Example/ : How backprop works in LSTM's (the so-called BPTT (back prop. through time)

RNN related stuff

1. http://www.neutronest.moe/2015-11-15-LSTM-survey.html

2. http://www.kdnuggets.com/2015/06/rnn-tutorial-sequence-learning-recurrent-neural-networks.html

3. http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/ : Series of posts explaining RNN with some code

4. http://colah.github.io/posts/2015-08-Understanding-LSTMs/ : Great post explaining LSTMs

5. https://www.reddit.com/r/MachineLearning/comments/2zkb3b/lstm_a_search_space_odyssey_comparison_of_lstm/ : Comparision of various LSTM architectures

6. http://www.fit.vutbr.cz/~imikolov/rnnlm/ : RNN based language modelling toolkit by Tomas Micholov

7. http://www.fit.vutbr.cz/~imikolov/rnnlm/char.pdf : A new technique in solving sequence tasks which I belive will be point of interest in few years : subword based language models. Usually good at handling OOV, spelling error problems

8. https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition : Human activity recognition using TensorFlow on smartphone sensors dataset and an LSTM RNN (predict time series).

9. https://github.com/guillaume-chevalier/seq2seq-signal-prediction : Learn and practice seq2seq in TensorFlow on time series data for signal prediction

Solving NLP tasks using Deep Learning

1. http://eric-yuan.me/ner_1/ : Named Entity Recognition using CNN

2. http://arxiv.org/pdf/1511.06388.pdf : Word Sense Disambiguation using Word Embeddings

3. http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow : CNN for Text Classification

4. http://research.microsoft.com/en-us/projects/dssm/ : Deep Learning Models for learning Semantic Representation of text(document, paragraph, phrase) which can be used to solve variety of tasks including Machine Translation, Document ranking for web search etc.

5. http://www.aclweb.org/anthology/P15-1130 : Sentiment Analysis using RNN (LSTMs)

6. http://ir.hit.edu.cn/~dytang/paper/emnlp2015/emnlp2015.pdf : Sentiment Analysis using Hierarchical RNN's (GRU)

7. https://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-with-gpus/ : Machine translation using RNN's

8. http://neon.nervanasys.com/docs/latest/lstm.html : Practical example of using LSTM for sentiment analysis

9. https://cs224d.stanford.edu/reports/HongJames.pdf : Again Sentiment Analysis using LSTMs

10. arxiv.org/pdf/1412.5335 : ICLR '15 paper on using ensembles of NN + Generative models (Language model, Naive bayes) for solving Sentiment prediction task

11. http://research.microsoft.com/pubs/214617/www2014_cdssm_p07.pdf : Extension of paper mentioned in [4] which used Convolution and max-pooling operations to learn low-dimensional semanti c representation of text

Optimization for Neural Networks

1. http://cs231n.github.io/neural-networks-3/#update

2. http://nptel.ac.in/courses/106108056/10 : JUMP TO SECTION : Uncontstrained optimization. Has tutorials on Non-convex optimization essential in deep Learning.

3. http://online.stanford.edu/course/convex-optimization-winter-2014 : Has more convex optimization part, contains basics of Optimization

4. http://videolectures.net/deeplearning2015_schmidt_smooth_finite/ : Deep Learning Summer School optimization lecture

Datasets

1. https://bigquery.cloud.google.com/table/fh-bigquery:reddit_comments.2015_08?pli=1 : Reddit comments dataset

2. https://code.google.com/archive/p/word2vec/ : Links to unlabelled english corpus

3. http://github.com/brmson/dataset-sts : Variety of datasets wrapped in Python with focus on comparing two sentences, sample implementations of popular deep NN models in Keras

4. http://www.mpi-sws.org/~cristian/Cornell_Movie-Dialogs_Corpus.html : Conversation dataset (for learning seq2seq models possible leading to a chatbot kind of application)

5. https://github.com/rkadlec/ubuntu-ranking-dataset-creator : Ubuntu Dialog Corpus 5.1 : http://arxiv.org/pdf/1506.08909v3.pdf : Accompanying paper for Ubuntu dataset

6. http://www.aclweb.org/anthology/P12-2040 : Another Dialogue corpus

7. http://www.lrec-conf.org/proceedings/lrec2012/pdf/1114_Paper.pdf : yet another dialogue corpus

8. http://www.cs.technion.ac.il/~gabr/resources/data/ne_datasets.html : NER resources

9. http://linguistics.cornell.edu/language-corpora : List of NLP resources

10. https://github.com/aritter/twitter_nlp/blob/master/data/annotated/ner.txt : Annotated twitter corpus

11. http://schwa.org/projects/resources/wiki/Wikiner

12. https://www.aclweb.org/anthology/W/W10/W10-0712.pdf : Paper describing annotation process for NER on large email data (could not find any link, if anyone finds out please feel free to send a PR)

13. http://www.cs.cmu.edu/~mgormley/papers/napoles+gormley+van-durme.naaclw.2012.pdf : Annotated gigawords

14. http://jmcauley.ucsd.edu/data/amazon/ : Amazon review dataset (LARGE CORPUS)

15. http://curtis.ml.cmu.edu/w/courses/index.php/Amazon_product_reviews_dataset : Amazon product review dataset (available only on request)

16. http://times.cs.uiuc.edu/~wang296/Data/ : Amazon review dataset

17. https://www.yelp.com/dataset_challenge : Yelp dataset (review + images)

Practical tools for Deep Learning

1. Deep Learning libraries

   1.1. theano

   1.2. torch

   1.3. tensorflow

   1.4. keras

   1.5. lasagne

   1.6. blocks and fuel

   1.7. skflow

   1.8. scicuda

2. (Automatic Differentiation tool in python)[https://github.com/HIPS/autograd]

3. (Spearmint : Hyperparamter optimization using Bayesian optimization)