A video lecture by Andrej Karpathy explaining the attention mechanism used in transformers, covering queries, keys, values, and how their dot products measure affinity between tokens.

The commenter calls it the best video for learning attention, praising how it explains queries, keys, and values as message passing between vectors.

What commenters said (1)

1. Strongly recommendsNoobSaibot1352023in “Intro to Large Language Models [Video]”

   Here’s another awesome Karpathy lecture from Stanford:

   https://youtu.be/XfpMkf4rD6E?si=1_EmuYDFfi7RNEhz

   This video is the best for learning attention, specifically where he explains:

   Think of attention like a directed graph of vectors passing messages to each other

   Keys are what other tokens are communicating to you,

   Queries are what you are interested in,

   and Values are what you are projecting out yourself.

   When you matrix multiply the queries x keys(transposed), you measure the interestingness or affinity between the two.

Related courses

• Machine Learning (Andrew Ng)Coursera · 240 recommenders
• Practical Deep Learning for Codersfast.ai · 112 recommenders
• CS231n: Convolutional Neural Networks for Visual RecognitionStanford University · 39 recommenders
• Deep Learning Specialization (Andrew Ng)Coursera · 31 recommenders
• Neural Networks for Machine Learning (Geoffrey Hinton)Coursera · 25 recommenders
• CS229 Machine LearningStanford University · 20 recommenders
• Learning From DataCaltech · 20 recommenders
• CS188 Introduction to Artificial IntelligenceUC Berkeley · 18 recommenders