Generalization refers to your model's ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model.
Generalization
The Big Picture
- Goal: predict well on new data drawn from (hidden) true distribution.
- Problem: we don't see the truth.
- We only get to sample from it.
The Big Picture
- Goal: predict well on new data drawn from (hidden) true distribution.
- Problem: we don't see the truth.
- We only get to sample from it.
- If model h fits our current sample well, how can we trust it will predict well on other new samples?
How Do We Know If Our Model Is Good?
- Theoretically:
- Interesting field: generalization theory
- Based on ideas of measuring model simplicity / complexity
- Intuition: formalization of Ockham's Razor principle
- The less complex a model is, the more likely that a good empirical result is not just due to the peculiarities of our sample
How Do We Know If Our Model Is Good?
- Empirically:
- Asking: will our model do well on a new sample of data?
- Evaluate: get a new sample of data-call it the test set
- Good performance on the test set is a useful indicator of good performance on the new data in general:
- If the test set is large enough
- If we don't cheat by using the test set over and over
The ML Fine Print
Three basic assumptions in all of the above:
- We draw examples independently and identically (i.i.d.) at random from the distribution
- The distribution is stationary: It doesn't change over time
- We always pull from the same distribution: Including training, validation, and test sets