Prediction Accuracy

Training and Test Error as A Function of Model Complexity

Fit of a model improves with the complexity of the model, i.e. as more predictors are included in the model the R² value is expected to improve. If predictors truly capture the main features behind the data, then they are retained in the model.The trick to build an accurate predictive model is not to overfit the model to the training data.

Overfitting a Model

If a learning technique learns the structure of a training data too well then the model is applied to the data on which the model was built, it correctly predicts every sample value. In the extreme case the model in training data admits no error. In addition to learning the general patterns in the data, the model has also learned the characteristics of each training data point's unique noise. This type of model is said to be over-fit and will usually have poor accuracy when predicting a new sample. (Why?)