Thank you for this article; very thought provoking.
My nitpick:
> When each student was given a heavy book of MNIST database, hundreds of pages filled with endless hand-written digit series, 60000 total, written in different styles, bold or italic, distinctly or sketchy.
> ...
> So, are you going to say that was not the case?
I understand the point the author is making. Human brains are really good at taking limited examples and correctly extrapolating them to new cases. That is, of course, the goal of intelligence. Machine Learning has gotten better at this generalization, but has a long way to go. And ConvNets as they exist today will not achieve that, no matter how much training you perform on them.
This specific example is inaccurate though. Let us aggressively simplify and low-ball by saying that humans see at 24fps. Humans of course don't see in discrete frames, but this simplification doesn't detract from my argument and makes quantifying easier. So, if you give a human a single page of numbers, and they look at it for an hour, they have now seen >86k examples. That's 86k examples with twitching saccades, and from both eyes. That's in just an hour of looking at numbers.
Prior to being given that page of numbers, most children will have been alive for 4-5 years. That's 3 billion examples from a wide variety of subjects (we ignore sleeping cycles, because we're already low-balling this fps figure, and because the brain is still learning and visualizing during sleep).
And humans are born with a pre-built visual cortex. Edge detection, gradient detection, etc. are all already built for us. CNNs learn that from scratch.
The author's real point is still valid, though, don't get me wrong. I'm just nitpicking.
Humans don't need to do data augmentation over several thousands of frames just to learn to recognize an object or image.
In the words of Joshua Tenenbaum and coauthors, "human children learning names for object concepts routinely make strong generalizations from just a few examples".
You can check this out for yourself on the brilliant illustration that went with it: http://i.imgur.com/5axtXSo.png From Tenenbaum J.B. et al, "How to grow a mind: statistics, structure, and abstraction," March 2011, Science, DOI:10.1126/science.1192788.
Convnets can do this. Geoffrey Hinton has a wonderful lecture, where he trained a digit recognizer on everything but 7's and 8's (IIRC.) He then let another convnet tell it which numbers looked more or less like 7's and 8's. E.g. "that 9 looks kind of like an 8. That 1 looks kind of like a 7", etc.
And then it was able to correctly recognize 7's and 8's, despite never having actually seen one. I'm simplifying somewhat, but it was super cool.
I don't know why people are so focused on one-shot learning, or think that NNs can't do it. Neural networks learn features from lots of (possibly unlabelled) data. That's the whole point. Once you have those features, you can use them for all sorts of things. You can show it an image, and then measure how close other images are too it. Thereby learning from a single example.
That's a bunch of noisy pictures of very similar numbers. You can add noise to the pictures you're training your CNN with; it's not really going to help. It's still texture without structure or context.
You can show me novel symbols, with me only looking at a single example of each for a few seconds, and I can manage good categorization.
Maybe we need a different representation at the upper layers, to capture higher concepts. Perhaps cross domain learning, combining learning from text for example with learning from video/images would help a lot.
My nitpick:
> When each student was given a heavy book of MNIST database, hundreds of pages filled with endless hand-written digit series, 60000 total, written in different styles, bold or italic, distinctly or sketchy. > ... > So, are you going to say that was not the case?
I understand the point the author is making. Human brains are really good at taking limited examples and correctly extrapolating them to new cases. That is, of course, the goal of intelligence. Machine Learning has gotten better at this generalization, but has a long way to go. And ConvNets as they exist today will not achieve that, no matter how much training you perform on them.
This specific example is inaccurate though. Let us aggressively simplify and low-ball by saying that humans see at 24fps. Humans of course don't see in discrete frames, but this simplification doesn't detract from my argument and makes quantifying easier. So, if you give a human a single page of numbers, and they look at it for an hour, they have now seen >86k examples. That's 86k examples with twitching saccades, and from both eyes. That's in just an hour of looking at numbers.
Prior to being given that page of numbers, most children will have been alive for 4-5 years. That's 3 billion examples from a wide variety of subjects (we ignore sleeping cycles, because we're already low-balling this fps figure, and because the brain is still learning and visualizing during sleep).
And humans are born with a pre-built visual cortex. Edge detection, gradient detection, etc. are all already built for us. CNNs learn that from scratch.
The author's real point is still valid, though, don't get me wrong. I'm just nitpicking.