Avoiding Overfitting

• noise in data, number of training instances too small
• Given a hypothesis space , a hypothesis is said to overfit the training data if there exists some alternative hypothesis , such that has a smaller error than over the training examples, but has a smaller error than over the entire distribution of instances.