Machine Learning Model Evaluation

Model evaluation answers one question: how well does this model perform on data it has not seen before? Training performance tells you how well the model memorized the data. Evaluation performance tells you how well it generalizes.

A model that scores perfectly on training data but fails on new inputs has overfit. A model that performs poorly on both has underfit. Evaluation methods and metrics give you the tools to detect these failures and compare alternatives.

The fundamental problem

You want to estimate future performance, but you only have past data. The solution is to simulate the future by holding out data that the model never sees during training. Every evaluation strategy is a variation on this idea.

Train-test split

The simplest approach divides data into a training set and a test set. You train on one portion and measure performance on the other. This gives a single estimate of generalization performance.

A typical split uses 70-80 percent for training and 20-30 percent for testing. The test set must remain untouched until final evaluation. If you use it to tune hyperparameters, it stops being a reliable estimate of future performance.

Cross-validation

Cross-validation provides a more robust estimate by repeating the train-test process multiple times.