#66 overall
CS224d: Deep Learning for Natural Language Processing
Stanford University
9 recommenders · 10 threads · recommended in 3 years, 2015–2017 · score 7.6
Stanford course on deep learning architectures applied to natural language processing, including recurrent and recursive neural networks, LSTMs and GRUs, and word vector representations. Lecture notes and materials are published on the course site, and commenters note lectures were recorded and mirrored on YouTube.
Commenters call it a great course on NLP with deep learning, recommended as a companion to cs231n for going deeper into neural networks for language.
Machine Learning & AIadvancedfreevideo lectures#deep learning#nlp#recurrent neural networks#word vectors#socher
Recommendations by year
20062026
Awesome!
P.s. Richard Socher, one of the authors on the paper taught a great Stanford course 'CS224d: Deep Learning for Natural Language Processing' with videos and notes available here:
http://cs224d.stanford.edu/
For general introductory material in this style from Stanford, CS231n (fairly general but specialization in vision) and CS224d (specialization in DL for NLP) are great. The material for both of these are online for free and the video lectures (taken down due to legal challenges regarding accessibility) are available if you look hard enough ;)
If you're particularly after unsupervised deep learning, I'd recommend you do one or both of the above (or equivalent) and then read relevant recent papers.
http://cs231n.stanford.edu
http://cs224d.stanford.edu
(also, sorry, the speech stuff was using HMM+GMM models, and now mostly use HMM+DNN models [so, we replaced the gaussian mixture model with a DNN, but kept the HMM], although some people are moving over to RNN models which doesn't use HMMs (connectionist temporal classification))
This is a great course, about NLP generally:
http://cs224d.stanford.edu/syllabus.html
A typical NLU system might be:
an intent classifier to predict intent (e.g., "play music"). This can be pretty much anything (SVM, logistic regression, RNN, I don't care, try stuff and see what works for you)
a segmentation model or slot-filling model (so label every word as either junk or "album name" or "song name" or whatever). This requires a sequence model, e.g., HMMs, CRFs, LSTM-RNNs.
Here's a nice example:
https://www.microsoft.com/en-us/research/wp-content/uploads/...
In addition to cs231n which someone recommended, I would also suggest cs224d (stanford) and university of waterloo's course by ali ghodsi STAT 946
check out cs224d and cs231n (both stanford) , there's another course by university of waterloo https://uwaterloo.ca/data-science/deep-learning , you will find lecture videos for all on youtube
The possibility of deep NLP suggest that there are tremendous opportunities for building a personal assistant. For Deep NLP, I would suggest course CS224D at Stanford on YouTube (going on now but also delivered last year: https://www.youtube.com/watch?v=kZteabVD8sU
For Deep NLP, you will need to be solid on linear algebra and machine learning. For introduction to machine learning, check out Andrew Ng at Coursera: https://www.coursera.org/learn/machine-learning and my favorite talks on Linear Algebra are the ones done by Gilbert Strang: http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-...
For web crawling, there are plenty of open source libraries. If you are not familiar with it, check out the Common Crawl: http://commoncrawl.org/ This is a great source of data to crawl.
If you focus solely on NLP and data from the Common Crawl (or even Wikipedia), then you will see where you stand as the smoke clears and you feel comfortable with the state of the art techniques.
Ignore all the naysayers. The good news is that it has never been easier to get started in deep NLP. Once you have the experience of training a model and seeing how well it works, you can decide on the next steps. Perhaps, you will find a niche that is not well-served that you can go after as a first step of a personal assistant.
Good luck.
There are tons of ways to evaluate word vector quality! Word analogy tasks, word similarity tasks, contextual prediction tasks, etc.
This link contains a bunch of relevant evaluation datasets and benchmarks obtained using word2vec, GloVe, etc. You can evaluate your RNN-learned vectors and compare them to a traditionally trained word2vec-trained vectors. Link here: http://www.bigdatalab.ac.cn/benchmark/bm/Domain?domain=Word%...
For more background on evaluating word vectors check out these pretty great lecture notes from Socher's NLP class: http://cs224d.stanford.edu/lecture_notes/LectureNotes2.pdf
Also, here's the original papers from a few years ago that introduced many of these datasets and evaluation standards:
https://papers.nips.cc/paper/5021-distributed-representation...
http://www.cs.cmu.edu/~mfaruqui/papers/acl14-vecdemo.pdf
Yeh there is a lot of work going on in this area... check out all these lectures and the papers they cite.
http://cs224d.stanford.edu/syllabus.html
There is a race to create a virtual assistant and do question answering.. the big players are IBM Watson, Apple w/ Siri, Google, Microsoft w/ cortana, facebook M, and Amazon Echo.
This is fantastic! I've been meaning to make an RNN to generate EDM.
For begineers, I'd like to summarize some excellent resources to start:
- Coursera course by Andrew Ng (he explains everything magically. The best course I've done online) https://class.coursera.org/ml-003/lecture
- Neural Networks and Deep Learning neuralnetworksanddeeplearning.com/ (I highly recommend trying to write your own backprop and MNIST dataset classifier. I wrote in JS and gave me a lot of confidence)
- Oxford ML class (2015) https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearni... (perhaps the the most recent MOOC on ML online. Things are progressing so fast in deep learning that it's worthwhile to do multiple ML courses to get different perspectives)
- I also enjoyed http://karpathy.github.io/2015/05/21/rnn-effectiveness/ for inspiration and the video https://www.youtube.com/watch?v=xKt21ucdBY0 is a FANTASTIC summary. HN earlier pointed our the course videos are out http://cs224d.stanford.edu/syllabus.html
- Again for inspiration into big challenges, this talk from Andrew Ng is a good one https://www.youtube.com/watch?v=W15K9PegQt0
Andrej is also a great lecturer; his CS231n class in the winter was both the most enjoyable and educational I've taken all year. All of the materials are available at cs231n.stanford.edu, although I can't seem to find the lecture videos online. It may not have been recorded.
As a bonus, there's an ongoing class on deep learning architectures for NLP which covers Recurrent (and Recursive) Neural nets in depth (as well as LSTM's and GRU's). Check out cs224d.stanford.edu for lecture notes and materials. The lectures are definitely being recorded, but I don't think they're publicly available yet.
With all due respect, this is a quite random reading list, and appears more a result of the 'if "deep learning" in post.title: post.upvote()' trend on Hacker News ...
I will pick on two under the "Classics" section simply as I know the author and have used their work, so am not in any way saying the work isn't useful (it certainly can be in the right spot!), but it isn't "classic" or what I'd recommend for early readers at all. "Uncertainty in Deep Learning" and "Dropout as a Bayesian Approximation" were both published within the last year and a half and is a PhD thesis + paper on interpretations of neural networks in a Bayesian fashion. "Classic" for a paper + thesis less than two years old is quite a stretch even for the fast moving field of deep learning. The same holds true for many of the other papers in "Classics" such as batch norm which is (a) recent and (b) certainly not the only of such techniques (see layer norm, recurrent batch norm, ...) and (c) has complications in implementation[1].
As the simplest example, why is the original dropout paper[7] not under classics? It's an elegant paper, fundamentally important for current neural networks, and is more classic than "Dropout as a Bayesian Approximation" or "Dropout Rademacher Complexity of Deep Neural Networks" which are both listed.
I'm also highly dubious of the noted neuroscience connection - most deep learning researchers use very little from neuroscience.
Again, this list may be helpful to the creator of the repo and tailored toward their specific research direction but it is not useful for readers from Hacker News or those aiming to get their start in deep learning. Why so many upvotes? Zero comments? Zero discussion?
If you want a book, check out the Deep Learning book[2]. If you want a course for RNNs, check out CS224d[3]. If you want a course for CNNs, check out CS231n[4]. If you want to get down and dirty in a practical software engineering way, check out Fast AI[5]. If you want summaries of select recent deep learning papers in GitHub format, check out Denny Britz's notes[8]. There are many other starting points but those are my default suggestions.
If you really want to start learning, this isn't the right list for you and I'd really like to suggest a more sane and potentially tailored path. Seriously. If you reply with what you want, I'll do my best to suggest a starting point.
Background: I'm a deep learning researcher who publishes papers and articles[6].
[1]: http://www.alexirpan.com/2017/04/26/perils-batch-norm.html
[2]: http://www.deeplearningbook.org/
[3]: http://cs224d.stanford.edu/
[4]: http://cs231n.github.io/
[5]: http://course.fast.ai/
[6]: http://smerity.com/articles/2016/google_nmt_arch.html
[7]: https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf
[8]: https://github.com/dennybritz/deeplearning-papernotes
Find a problem to work on in a domain you find interesting. By reading published papers and trying to attack the problem, you'll be forced to pick up a lot of other knowledge not commonly discussed like: feature extraction and selection, dimensionality reduction, dealing with sparsity, common metrics for that problem, recent work, etc.
I was forced to learn a massive amount in a short period of time for work, but I'd previously watched Andrew Ng's lectures, as well as majored in Math/CS. I can also generally recommend Hinton's NN lectures, Socher's Deep learning for NLP, Andrew Ng's Machine Learning, and a few books.