#34 overall
Neural Networks: Zero to Hero
YouTube
16 recommenders · 16 threads · recommended in 4 years, 2022–2025 · score 15.1
A series of YouTube video lectures by Andrej Karpathy that builds neural networks from the ground up, starting with a small autograd and backpropagation engine and working up to a simplified GPT-style language model. Lectures are followed along in a Jupyter notebook.
Commenters call it a clear, hands-on way to understand backpropagation and transformers, with Karpathy's explanations of attention singled out as especially accessible.
Machine Learning & AIbeginnerfreevideo lectures#neural networks#backpropagation#transformers#gpt#karpathy
Recommendations by year
20062026
I understand the sadness around not understanding it, it's fucking hard. however, there are more and more resources online getting published for how to get started understanding it that help with understanding the math at an abstraction that helps with learning how to build with it.
I would strongly strongly strongly recommend starting with karpathy's from 0 to hero neural networks youtube course - it starts with building a tensor library and back propagation, explaining it in a way that finally clicked for me https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...
jeremy howard also has a fantastic video that is more around how to use LLMs and such called a hacker's guide to language models - https://www.youtube.com/watch?v=jkrNMKz9pWU&t=607s
as i've dove more and more into it, i would strongly recommend trying to run things on your local machine too (llama.cpp, ollama, LM studio). that has helped me fight that feeling of like "are we all just going to be open AI developers in the end" and made me feel like you _can_ integrate these things into stuff you build by your self. I can't imagine how fucked we'd all be if llama was never opensourced. being old does not mean that you can't continue to grow, and remember that it's okay to feel overwhelmed about all this - many people are.
Try Andrej Karpathy’s zero to hero course. It’s very good. It’s 8 video lectures where you follow along in your own Jupyter notebook. Each lecture is 1-2 hours.
The best way to understand transformers is to take Andrej’s Karpathy course on youtube. With a keyboard and a lot of focus time.
Andrej’s course is brilliant and so nice to follow.
His explanation of attention is the most accessible I have ever seen.
Wow, this is great. I can't wait for the video lecture, transformers are an aspect of modern machine learning that I'm not completely clear on. Andrej's lectures are brilliant - super detailed, and really answer the detailed questions I always have. Great stuff!
From my other comment elsewhere. These resources helped me understand the topics better.
If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1].
It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics)
There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3].
There is also book for excellent caltech course[4].
Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5].
[1] https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4...
[2] https://chrispiech.github.io/probabilityForComputerScientist...
[3] https://work.caltech.edu/telecourse
[4] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...
[5] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
I took Andrew Ng's Coursera ML course but I think it's a bit out of date now. I've heard good things about https://karpathy.ai/zero-to-hero.html to learn AI/ML.
Most tech interviews will have a leetcode component, so I would also take a more advanced algorithms/data structures class like https://www.coursera.org/learn/algorithms-part1
I would also start doing problems on leetcode once you finish the course.
I took the Deep Learning course [1] by deeplearning.ai in the past, and their resources where incredibly good IMHO. Hence, I would suggest to take a look at their NLP specialization [2].
+1000 to "Neural networks: zero to hero" already mentioned as well.
[1] https://www.deeplearning.ai/courses/deep-learning-specializa... [2] https://www.deeplearning.ai/courses/natural-language-process...
I personally preferred Andrej Karpathy's CS231n taught by him and his private videos about neural nets in general and transformers in particular. He has a youtube vid where he builds one from scratch in Python!
3BlueOneBrown videos are a great complement to Karpathy's lectures to aid in visualising what is going on.
My master's was in Convolutional NNs for language processing. I had zero prior knowledge and my advisor recommended I watch Karpathy's lectures[1] to get up to speed
[1] https://youtube.com/playlist?list=PLkt2uSq6rBVctENoVBg1TpCC7...
If you haven't already, I highly suggest going through Andrej Karpathy's Neural Networks course. He covers some of the history and evolution of neural networks.
https://karpathy.ai/zero-to-hero.html
I really like Andrej Karpathys content
yeah its youtube i know... but, its hand on too
for gpt/llm/ml in general https://karpathy.ai/zero-to-hero.html
it starts with writing back prop from scratch and take you through writing everything you need and training a gpt2 equivalent model in the end
I also thought his lectures at standford, on youtube cs231n 2016, were really good. they cover GAN for generation, but I think after that you can read the papers that were the source for diffusion models dalle and midjouney use
I enjoyed the fast AI course: https://course.fast.ai/
and Karpathy's course: https://karpathy.ai/zero-to-hero.html
Not really an ELI5, but it you really want to understand the topic, I've gathered the best resources I could find to help me do the same: https://blog.senko.net/learn-ai
If you're a programmer, start with Karpathy's video series. For a somewhat gentler intro, take a look at the MIT intro lectures first to build up on the fundamentals.
Then you're ready for The Illustrated Transformer, and afterwards, if you're brave enough, the Annotated Transformer.
It's a fascinating subject, more so when you have a solid grasp! And you'll be able to quickly spot people who kinda stumble they way through but have big gaps in understanding.
You haven't specified in which way this is too advanced for you.
Still, I'll take a stab at recommending some other resources.
1: Practical Deep Learning by fast.ai - a (free) hands on course that's designed to get you to making something useful as quickly as possible.
2: Neural Networks Zero to Hero from Andrej Karpathy - A Series of Youtube Lectures that requires only basic python knowledge and takes you all the way up to a simplified implementation of the tech inside gpt models.
3: Neural Networks from Scratch by Sentdex. A book that's explicitly written to teach you the basics. Doesn't cover stuff like Transformers and other advanced concepts, but really takes its time with the basics.
Take a look at all three of them. They all have a different teaching style and I'd guess that at least one of them will gel with you.
Great question!
I seem to be in a similar situation as an experienced software engineer who has jumped into the deep end of ML. It seems most resources either abstract away too much detail or too little. For example, building a toy example that just calls gensim.word2vec doesn't help me transfer that knowledge to other use cases. Yet on the other extreme, most research papers are impenetrable walls of math that obscure the forest for the trees.
Thus far, I would also recommend Andrej Karpathy's Zero to Hero course (https://karpathy.ai/zero-to-hero.html). He assumes a high level of programming knowledge but demystifies the ML side.
--
P.S. If anyone is, by chance, interested in helping chip away at the literacy crisis (e.g., 40% of US 4th graders can't read even at a basic level), I would love to find a collaborator for evaluating the practical application of results from the ML fields of cognitive modeling and machine teaching. These seemingly simple ML models offer powerful insight into the neural basis for learning but are explained in the most obtuse ways.
I'm in the same boat kinda, but even more outdated on some parts. I had some AI specialization back in college, but that was before deep learning was even a thing, so we did stuff like self-organizing maps and evolutionary algorithms, but it wasn't really all that useful for much back then. Been following deep learning from the sidelines, but for work my AI work has been restricted to GOFAI until recently.
Some of the stuff I'm currently reading/watching or have recently
Practical Deep Learning, though it sounds like you may know this stuff already (https://course.fast.ai/)
Practical Deep learning part 2, more about diffusion models. Full course coming early next year (https://www.fast.ai/posts/part2-2022-preview.html)
Hugging Face course (https://huggingface.co/course/chapter1/1)
Diffusion models from hugging face https://huggingface.co/blog/annotated-diffusion https://huggingface.co/docs/diffusers/index
Andrej Karpathy's Neural Networks: Zero to Hero. He goes from the basics up to how GPT etc, so you can start wherever suits you (https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs...)
3blue1brown's videos. I've found all his videos on neural networks and math worth watching, even for stuff that I already know, he sometimes has some new perspectives and nice animations.
brilliant.org. Nice math refresher and the courses there are almost like fun little games.
I think you are severely underestimating the time required, unless you are quite experienced, know exactly what to look for, or the paper is just a slight variation on previous work that you are already familiar with.
Even seasoned researchers can easily spend 30+ hours on trying to reproduce a paper, because papers almost never contain all the details that went into the experiments. You are left with a lot of fiddling and iteration. Of course, if you only care about roughly reproducing what the authors did, and don't care about getting the same results, the time can be much shorter. If the code is available that's even better, but looking at it is cheating since wrestling with issues yourself is a big part of the learning process.
A few people here mentioned Andrej's lectures, and I also think they are amazing, but they are not a replacement for getting stuck and solving problems yourself. You can easily watch these lectures and think "I get it!" because everything is so well explained, but you'll probably still be stuck when you run into your own problems trying to reproduce papers from scratch. There's no replacement for the experience you gain by struggling :)
It's like watching a math lecture and thinking you get it, but then getting stuck at the exercise problems. The real learning happens when you force yourself to struggle through the exercises.