#21 overallClassic
Learning From Data
Caltech
20 recommenders · 23 threads · recommended in 10 years, 2012–2025 · score 29.7
Yaser Abu-Mostafa's Caltech course on the theoretical foundations of machine learning, covering generalization, VC dimension, regularization, validation, and linear and nonlinear models. Available as free video lectures on the instructor's site and, at times, on edX, with an accompanying book.
Commenters call it an outstanding, rigorous introduction to the theory behind machine learning, more mathematical than Andrew Ng's Coursera class and taught with exceptional clarity.
Machine Learning & AIintermediatefreevideo lectures#machine learning#learning theory#vc dimension#regularization#abu-mostafa
Recommendations by year
20062026
It's not on Coursera, but Caltech's Yaser Abu-Mostafa offers "Learning from Data" on his own website https://work.caltech.edu/telecourse.html and it's intermittently on EdX https://www.edx.org/course/learning-from-data-introductory-m...
This is far and away the best MOOC I've ever taken. The class is genuinely challenging. It's a real Caltech undergraduate course, and you can't get away with copy-pasting code or just keep resubmitting until you pass the grader. The course is focused on real understanding of what's going on mathematically, not just learning to use some library API.
To learn a solid theoretical foundation: Caltech's "Learning from data" [1]. It is one of those rare courses where the professor is so good that he manages to make tricky concepts seem almost trivial (like Paar's "Understanding cryptography").
Look elsewhere if all you want is to learn tools and start practical projects ASAP without really understanding what you are doing. Tools come and go and will serve you for a while, concepts are timeless and will serve you for life. You need both, of course, but I wouldn't skip the theory, specially when such an amazing course is on tap.
[1] https://work.caltech.edu/telecourse.html
Do Caltech's free online course "Learning from data" to learn ML in a principled way. You will get the answers to all those questions and then some. You basically need to learn some basic theory and concepts (linear models, non linear transformations, vc dimension, learning curves, regularization, validation, etc) much more than tools first to deal with those questions. The professor in the video lectures is excellent
I'll just throw in a plug here for Prof. Abu-Mostafa's Learning from Data course at Caltech, which is outstanding: http://work.caltech.edu/telecourse.html
Unlike Andrew Ng's Coursera Machine Learning class, it is a real, unadulterated Caltech class, and exactly as challenging as that implies. It delves much more deeply into the mathematics behind ML, and the homework assignments are quite time consuming.
I took it as part of an interactive session through EdX, and the professor himself was extremely active on the forums: responding to student questions, clarifying lecture points, and giving homework suggestions--seemingly at all hours of the day and night.
Prof Yaser S. Abu-Mostafa's Caltech course "Learning from Data" (http://work.caltech.edu/telecourse) is probably the best introductory course for really understanding the physics of how machine learning works.
See Prof's Yaser's 1 min overview: http://www.youtube.com/watch?v=KlP0DpiM7Lw
The "Learning from Data Book" videos are online for free, and the book is on Amazon...
Videos: http://home.caltech.edu/lectures.html
Book: http://www.amazon.com/Learning-From-Data-Yaser-Abu-Mostafa/d...
The course is also availble on EdX: https://www.edx.org/course/caltechx/cs1156x/learning-data/11...
According to an e-mail sent to alumni like myself, this will be the last session offered.
My guess is is that they will switch to another platform (coursera ?), but I might be wrong.
I have personally taken more than a dozen MOOCs by now and I can say that this is hands down the best that I've taken so far. The content will feel more mathematical than Andrew Ng's ML class but Professor Abu-Mustafa's teaching style makes anything crystal clear. I encourage everyone interested in the subject to take the course (even if you took coursera's ML class before): you will learn a lot.
What exactly do you want? Do you want a resource that "clicks" with you that you can use to learn about machine learning, or do you want a witty piece of art like "_why's poignant guide to ruby"?
If its the former you are looking for, I have several recommendations. There are a plethora of online resources to learn about machine learning from. In video form, my favorite resource is Yaser Abu-Mostafa's (Caltech prof) video lectures, available here: http://work.caltech.edu/teaching.html
You can also actually enroll in the next version of CS 156 (Abu-Mostafa's class) and do the class online with problem sets very similar to that of Caltech students actually enrolled in the class. (It starts January 8th).
Coursera/Udacity contain several classes that involve machine learning; Andrew Ng's class is a good place to start.
If you (like me) prefer text to video, you can buy Abu-Mostafa's book (I hear it follows the course very closely, and the book costs way less than $500 :D ), read Andrew Ng's online lecture notes, or read one of several freely available ML books (such as "Elements of Statistical Learning" or "Bayesian Reasoning and Machine Learning", but I've found these to take more mathematical maturity that the other resources recommended).
Note: One reason that I've highly recommended Abu-Mostafa's resources (besides having taken the class at Caltech and loving it) is that his class is more focused on the theory of machine learning (ie hammering in over-arching principles like avoiding overfitting by regularizing) rather than just covering as many algorithms as possible. Also, I believe Mitchell of CMU has good online resources for his machine learning course.
Try the Learning From Data course : http://work.caltech.edu/telecourse.html A Fall run has just started (on the 2nd of oct.).
It's the same version as the course given at CalTech and is more in-depth than Andrew Ng's. There is no skeleton code for the programming assignments, answers are made through quizzes. I took the summer session and learned a lot from it.
Slightly off topic, but I just completed the online learning from data course offered by Cal Tech (for free), and am pleased to say that 1) I understand this thread, and 2) have implemented an SVM starting with a quadratic programming package (not quite from scratch!). I highly recommend the class for anyone interested, a re-run of the course is starting soon: http://work.caltech.edu/telecourse.html
What Andy says is absolutely true. This machine learning class was easily the best class I took at Caltech. Prof. Abu-Mostafa got a standing ovation at the end of the course the term I took it. I wish I could have taken more of his classes.
It was also fairly difficult -- the assignments were hard, but at every step, you could look at what you'd done and say "I know why I'm doing this, and I can see how this works."
I remember at the end of the term he took several students' notes and made copies of them, so that he could compare the students' notes with what he was trying to convey, and could know if he wasn't teaching certain parts of the class well enough.
It's a shame that not all professors are as dedicated and responsive to their teaching obligations as Prof. Abu-Mostafa.
Oh, also, "introductory" in this context is meant to differentiate it from "graduate level". Every student (mostly juniors and seniors) in this class will have had several terms of math, theoretical CS, and practical programming classes.
From my other comment elsewhere. These resources helped me understand the topics better.
If anyone wants to understand fundamentals of machine learning, one of the superb resources I have found is, Stanford's "Probability for computer scientists"[1].
It goes into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. But, this is a primarily a probability course that discusses the fundamentals of machine learning. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familiarity with linear algebra topics)
There is a course reader for CS109 [2]. You can download pdf version of this. Caltech's learning from data was really good too, if someone is looking for theoretical understanding of ML topics [3].
There is also book for excellent caltech course[4].
Also, neural networks zero to hero is for understanding how neural networks are built from ground up [5].
[1] https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4...
[2] https://chrispiech.github.io/probabilityForComputerScientist...
[3] https://work.caltech.edu/telecourse
[4] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...
[5] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
The best books I would recommend are;
— The Master Algorithm by Pedro Domingos
— The Book of Why by Judea Pearl
— What Computers Still Can't Do by Hubert L. Dreyfus
— Society of Mind & The Emotion Machine by Marvin Minsky
— The Feeling Of What Happens by Antonio Damasio
— Language & Mind by Noam Chomsky
— A great mooc is also, the "Learning From Data" Telecourse from Caltech https://work.caltech.edu/telecourse
If you're looking for a bit of an unconventional entry point, I recommend the seminal text 'Elements of Information Theory' by T. Cover (skipping chapters like Network Information/Gaussian channel should be fine), paired with David MacKay's 'Information Theory, Inference and Learning Algorithms'. Both seem available online:
http://www.cs-114.org/wp-content/uploads/2015/01/Elements_of...
http://www.inference.org.uk/itprnn/book.pdf
They cover some fundamentals of what optimal inference looks like, why current methods work, etc (in a very abstract way by understanding Kolmogorov complexity and its theorems and in a more concrete way in MacKay's text). Another good theoretical partner could be the 'Learning from data' course, yet a little more applied: (also available for free)
https://work.caltech.edu/telecourse.html
Excellent lecturer/material (to give a glimpse, take lecture 6: 'Theory of Generalization -- how an infinite model can learn from a finite sample').
Afterward I would move to modern developments (deep learning, or whatever interests you), but you'll be well equipped.
Here are the resources I found useful: ========================================== Advices from Open AI, Facebook AI leaders
Courses You MUST Take:
Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine-learning) /// Class notes: (http://holehouse.org/mlclass/index.html)
Yaser Abu-Mostafa’s Machine Learning course which focuses much more on theory than the Coursera class but it is still relevant for beginners.
(https://work.caltech.edu/telecourse.html)
Neural Networks and Deep Learning (Recommended by Google Brain Team) (http://neuralnetworksanddeeplearning.com/)
Probabilistic Graphical Models (https://www.coursera.org/learn/probabilistic-graphical-model...)
Computational Neuroscience (https://www.coursera.org/learn/computational-neuroscience)
Statistical Machine Learning (http://www.stat.cmu.edu/~larry/=sml/)
From Open AI CEO Greg Brockman on Quora
Deep Learning Book (http://www.deeplearningbook.org/) ( Also Recommended by Google Brain Team )
It contains essentially all the concepts and intuition needed for deep learning engineering (except reinforcement learning). by Greg
2. If you’d like to take courses: Linear Algebra — Stephen Boyd’s EE263 (Stanford) (http://ee263.stanford.edu/) or Linear Algebra (MIT)
(http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebr...)
Neural Networks for Machine Learning — Geoff Hinton (Coursera) https://www.coursera.org/learn/neural-networks
Neural Nets — Andrej Karpathy’s CS231N (Stanford)
http://cs231n.stanford.edu/
Advanced Robotics (the MDP / optimal control lectures) — Pieter Abbeel’s CS287 (Berkeley)
https://people.eecs.berkeley.edu/~pabbeel/cs287-fa11/
Deep RL — John Schulman’s CS294–112 (Berkeley) http://rll.berkeley.edu/deeprlcourse/
Thank you! I already had Prof. Yaser Abu-Mostafa's course in there too (also amazing), but I will check out Prof. Blitzstein's probability course as well.
You already have Prof. Gilbert Strang's course in there(made me fall in love with linear algebra), I would like to suggest Prof. Joe Blitzstein's course (https://projects.iq.harvard.edu/stat110/home) on probability and Prof. Yaser Abu-Mostafa's course (https://work.caltech.edu/telecourse) on Machine Learning.
Yes, I did my research but there is no such interactive tutorial online like Treehouse or Codecademy. There are so many tutorials but none of it tells you the whole path.
Here are the resources I found useful:
========================================== Advices from Open AI, Facebook AI leaders
Courses You MUST Take: Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine-learning) /// Class notes: (http://holehouse.org/mlclass/index.html)
Yaser Abu-Mostafa’s Machine Learning course which focuses much more on theory than the Coursera class but it is still relevant for beginners.(https://work.caltech.edu/telecourse.html)
Neural Networks and Deep Learning (Recommended by Google Brain Team) (http://neuralnetworksanddeeplearning.com/)
Probabilistic Graphical Models (https://www.coursera.org/learn/probabilistic-graphical-model...)
Computational Neuroscience (https://www.coursera.org/learn/computational-neuroscience)
Statistical Machine Learning (http://www.stat.cmu.edu/~larry/=sml/)
From Open AI CEO Greg Brockman on Quora
Deep Learning Book (http://www.deeplearningbook.org/) ( Also Recommended by Google Brain Team )
It contains essentially all the concepts and intuition needed for deep learning engineering (except reinforcement learning). by Greg
2. If you’d like to take courses: Linear Algebra — Stephen Boyd’s EE263 (Stanford) (http://ee263.stanford.edu/) or Linear Algebra (MIT)(http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebr...)
Neural Networks for Machine Learning — Geoff Hinton (Coursera) https://www.coursera.org/learn/neural-networks
Neural Nets — Andrej Karpathy’s CS231N (Stanford) http://cs231n.stanford.edu/
Advanced Robotics (the MDP / optimal control lectures) — Pieter Abbeel’s CS287 (Berkeley) https://people.eecs.berkeley.edu/~pabbeel/cs287-fa11/
Deep RL — John Schulman’s CS294–112 (Berkeley) http://rll.berkeley.edu/deeprlcourse/
From Director of AI Research at Facebook and Professor at NYU Yann LeCun on Quora
In any case, take Calc I, Calc II, Calc III, Linear Algebra, Probability and Statistics, and as many physics courses as you can. But make sure you learn to program.
My guess is that you won't find any course that explains all the prerequisite math. It's probably more useful to build a solid foundation in probability theory (and therefore calculus) before going on.
For machine learning, a good place to start is Andrew Ng's course on Coursera:
https://www.coursera.org/course/ml
It's pretty light on math, while at the same time giving you experience in implementing and understanding these techniques.
From there, I might recommend Learning from Data and the associated video lectures:
https://work.caltech.edu/telecourse.html
It is a bit of a jump, but it is a great course in presenting the field of machine learning and explaining the mathematical and statistical underpinnings in a systematic way.
You can do a MOOC with very little watering down. Yaser Abu-Mostafa's "Learning From Data" from Caltech in an example. Here's an article written by Abu-Mostafa about this [1].
[1] https://www.class-central.com/report/caltech-mooc-flipped-cl...
Caltech also has a fantastic open sourced machine learning course[1]. In fact, students taking the class are instructed to watch these lectures, and class-time is used for answering questions about lectures/homework. (disclaimer: caltech student)
[1] http://work.caltech.edu/lectures.html
I'd suggest going back to the original youtube[1] and course materials[2]. Coursera version is nothing but a hand-wavy watered down "feel good" version of the original class. I also really like the Caltech's take "Learning from data"[3]
[1] https://www.youtube.com/watch?v=UzxYlbK2c7E
[2] http://cs229.stanford.edu/
[3] http://work.caltech.edu/telecourse.html
It's more than that. The SVM kernel maps your multi-dimensional data into an infinite dimensional data. Because of the way the math works you can essentially learn from an infinite dimensional data without overfitting. The support vector is the data points that "support" the separating hyperplane, that is the points that meet the constraints. The other thing about SVMs is that they are computationally friendly.
Just as a simple example (stolen from the Caltech course which I highly recommend) if you look at points on a plane that form a circle and try to separate them with a line you're going to fail. I.e. your points are (x,y) and those in the circle are your fake dollar bill and those outside aren't. But you can take all these points and apply a non-linear transform, e.g. (x, y, xy, xxy, yyx, x+y, xx+y2), you get the idea... It turns out that now you can* separate the data into what's inside and outside the circle. The problem is you just increased your so called VC dimension of the model and you might overfit the data and not learn anything. SVMs let you get infinite numbers of combinations, without overfitting and with cheap calculations... Pretty neat.
I did that course and it's very different. They are complementary, the one in Coursera being more introductory.
I'd recommend the Caltech course too, which is also tougher.
I don't think it measures up as much. It's basically a dumbed down version of his actual Stanford course. If you have any experience in Math and Linear Algebra, you should take a more serious course like his lectures in Youtube or Caltech's course on Youtube. The ideal audience of the Coursera course is non-math people who want to get an idea of what machine learning is.
From, a comment I posted elsewhere for written versions.
There is a course reader for CS109 [1]. You can download pdf version of this.
There is also book[2] for excellent caltech course[3].
[1] https://chrispiech.github.io/probabilityForComputerScientist...
[2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...
[3] https://work.caltech.edu/telecourse
There is a course reader for CS109 [1]. You can download pdf version of this.
There is also book[2] for excellent caltech course[3].
[1] https://chrispiech.github.io/probabilityForComputerScientist...
[2] https://www.amazon.com/Learning-Data-Yaser-S-Abu-Mostafa/dp/...
[3] https://work.caltech.edu/telecourse
Here is a great caltech lecture about why "simple" models have better out of sample performance. Jump to 28:00, if you want to get straight to the point.
https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7...
Assuming that you consider Machine Learning to be either a subset of AI (as I do) or a sibling field, and want to learn aspects of ML, then consider Andrew Ng's Machine Learning course on Coursera. It's a great introduction and doesn't require a ton in terms of prerequisites. You'll see some multi-variable calculus and linear algebra, but he does the calculus derivations for you, and there's a pretty adequate review of the relevant parts of Linear Algebra.
In addition, if you don't already have a background in Calculus and Linear Algebra, then supplement the Ng course with the Khan Academy stuff on Calculus and Linear Algebra, or other courses you can pick up on Coursera or Edx or whatever.
If you get really interested in neural networks (which are all the rage these days) after the Ng class, there's a freely available book on Neural Network design that you could look at. It doesn't cover all the very latest techniques, but it would help you build the foundation of understanding.
http://hagan.okstate.edu/nnd.html
There's also a MOOC around the Learning From Data book that you could check out.
http://amlbook.com/
https://work.caltech.edu/telecourse.html
OTOH, if you're making a sharp distinction between "Classic AI" and "Machine Learning" and you really care mainly about the classical stuff, then you might want to start with the Berkeley CS188 class. You can take it through EdX (https://www.edx.org/course/artificial-intelligence-uc-berkel...) or just watch the videos and download the notes and stuff from http://ai.berkeley.edu/home.html
And if you just want to dive into reading some classic papers and stuff, check out:
http://publications.csail.mit.edu/ai/pubs_browse.shtml
and/or
http://ijcai.org/past_proceedings
Another good resource is
http://aitopics.org/
I think you're understating the degree to which modern machine learning algorithms are completely opaque, yes, to the everyday person, but also to the developers and researchers that use them.
Check out this section from Caltech's machine learning lectures about neural networks: https://www.youtube.com/watch?v=Ih5Mr93E-2c&hd=1#t=63m25s
"...sorry, we denied credit because lambda is less than 0.5. The fact that you are not able to interpret what happens in machine learning is very, very common."
Here's a comment which I had written earlier on another article. The context was about learning ML with Python - of course the objective of Hal is more generic but some parts of it apply here too.
https://news.ycombinator.com/item?id=5802968
The book details building ML systems with Python and does not necessarily teach ML per se. It is a good time to write a ML book in Python particularly keeping in mind efforts to make Python scale to Big Data [0].
What material you want to refer to is entirely dependent on What you want to do?. Here are some of my recommendations-
Q : Do you want to have an "Introduction to ML", some applications with Octave/Matlab as your toolbox?
A :Take up Andrew Ng's course on ML in Coursera [1].
Q : Do you want to have a complete understanding of ML with the mathematics, proofs and build your own algorithms in Octave/Matlab?
A : Take up Andrew Ng's course on ML as taught in Stanford; video lectures are available for free download [2]. Note - This is NOT the same as the Coursera course. For textbook lovers, I have found the handouts distributed in this course far better than textbooks with obscure and esoteric terms. It is entirely self contained. If you want an alternate opinion, try out Yaser Abu-Mostafa's ML course at Caltech [3].
Q : Do you want to apply ML along with NLP using Python ?
A : Try out Natural Language Tool Kit [4]. The HTML version of the NLTK book is freely available (Jump to Chapter 6 for the ML part) [5]. There is an NLTK cookbook available as well which has simple code examples to get you started [6].
Q: Do you want to apply standard ML algorithms using Python?
A : Try out scikit-learn [7]. The OP's book also seems to be a good fit in this category (Disclaimer - I haven't read the OP's book and this is not an endorsement).
[0] http://www.drdobbs.com/tools/us-defense-agency-feeds-python/....
[1] https://www.coursera.org/course/ml
[2] http://academicearth.org/courses/machine-learning/
[3] http://work.caltech.edu/telecourse.html
[4] http://nltk.org
[5] http://nltk.org/book/
[6] http://www.amazon.com/Python-Text-Processing-NLTK-Cookbook/d....
[7] http://scikit-learn.org
The book details building ML systems with Python and does not necessarily teach ML per se. It is a good time to write a ML book in Python particularly keeping in mind efforts to make Python scale to Big Data [0].
What material you want to refer to is entirely dependent on What you want to do?. Here are some of my recommendations-
Q : Do you want to have an "Introduction to ML", some applications with Octave/Matlab as your toolbox?
A :Take up Andrew Ng's course on ML in Coursera [1].
Q : Do you want to have a complete understanding of ML with the mathematics, proofs and build your own algorithms in Octave/Matlab?
A : Take up Andrew Ng's course on ML as taught in Stanford; video lectures are available for free download [2]. Note - This is NOT the same as the Coursera course. For textbook lovers, I have found the handouts distributed in this course far better than textbooks with obscure and esoteric terms. It is entirely self contained. If you want an alternate opinion, try out Yaser Abu-Mostafa's ML course at Caltech [3].
Q : Do you want to apply ML along with NLP using Python ?
A : Try out Natural Language Tool Kit [4]. The HTML version of the NLTK book is freely available (Jump to Chapter 6 for the ML part) [5]. There is an NLTK cookbook available as well which has simple code examples to get you started [6].
Q: Do you want to apply standard ML algorithms using Python?
A : Try out scikit-learn [7]. The OP's book also seems to be a good fit in this category (Disclaimer - I haven't read the OP's book and this is not an endorsement).
[0] http://www.drdobbs.com/tools/us-defense-agency-feeds-python/...
[1] https://www.coursera.org/course/ml
[2] http://academicearth.org/courses/machine-learning/
[3] http://work.caltech.edu/telecourse.html
[4] http://nltk.org
[5] http://nltk.org/book/
[6] http://www.amazon.com/Python-Text-Processing-NLTK-Cookbook/d...
[7] http://scikit-learn.org
What you do when a cluster has no points depends on the details of the problem you are trying to apply clustering to. If you don't actually care how many clusters you have, you could just drop such cluster centers from the final output.
If, on the other hand, you really want the number of clusters you asked for, one common approach is to abort the run if a cluster becomes entry and start over, with a new set of initial clusters.
I'm not sure if it really matters much how you pick the initial clusters. Random within the space, rather than random from among the points, can simply be easier. Suppose you've got 100 points and are trying to split into 10 clusters. If you pick from the points, you need to pick without replacement so that you won't pick the same point twice. This can complicate things a little, depending on what data structures you are using and what's in your language's library.
If, on the other hand, you are just picking 10 random points in the space, the chance of picking the same point twice is so low you can get away with not worrying about it.
If you are curious, another place K-means clustering is used is in choosing points to serve as the centers for modeling a dataset with radial basis functions. There is an excellent set of lectures and slides on this in the Caltech machine learning video library here: http://work.caltech.edu/library/