MIT's introductory deep learning course (6.S191) covering neural network fundamentals up through recurrent networks, transformers, and attention, delivered as video lectures. Commenters point to its lectures as a gentle on-ramp to transformers.

Commenters describe it as a very good, gentle introduction that builds from the ground up to transformers and attention.

What commenters said (4)

1. Strongly recommendshn_throwaway_992023in “Revealing example of self-attention, the building block of transformer AI models”

   That whole MIT course, Intro to Deep Learning, is great and a really nice introduction to the field.

2. Recommendssenko2023in “Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?”

   Not really an ELI5, but it you really want to understand the topic, I've gathered the best resources I could find to help me do the same: https://blog.senko.net/learn-ai

   If you're a programmer, start with Karpathy's video series. For a somewhat gentler intro, take a look at the MIT intro lectures first to build up on the fundamentals.

   Then you're ready for The Illustrated Transformer, and afterwards, if you're brave enough, the Annotated Transformer.

   It's a fascinating subject, more so when you have a solid grasp! And you'll be able to quickly spot people who kinda stumble they way through but have big gaps in understanding.

3. Recommendsdetrites2023in “What are transformer models and how do they work?”

   This hour-long MIT lecture is very good, it builds from the ground up until transformers. MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: https://youtube.com/watch?v=ySEx_Bqxvvo

4. MentionsbonoboTP2020in “Deep Learning from Scratch Part 3: Generalizing Gradient Descent”

   I urge beginners not to learn these things from blog posts, even if it seems more digestible. Take a reputable textbook and you will be better off. Or take a MOOC, like Andrew Ng's or courses from MIT and Stanford on Youtube. It may look harder, but it has all been carefully didactically structured. It's harder on the long run to learn from a non-expert.

   There's so much confusion out there, blog posts confusing cross-entropy with "softmax loss", gradient descent with backpropagation etc. etc.

Related courses

• Machine Learning (Andrew Ng)Coursera · 240 recommenders
• CS229 Machine LearningStanford University · 20 recommenders
• Neural Networks: Zero to HeroYouTube · 16 recommenders
• Practical Deep Learning for Codersfast.ai · 112 recommenders
• CS231n: Convolutional Neural Networks for Visual RecognitionStanford University · 39 recommenders
• Deep Learning Specialization (Andrew Ng)Coursera · 31 recommenders
• Neural Networks for Machine Learning (Geoffrey Hinton)Coursera · 25 recommenders
• Learning From DataCaltech · 20 recommenders