What is bias-variance trade-off?

  • What is error in a machine learning model?
  • What is bias?
  • What is variance?

A supervised machine learning model aims to train itself on the input variables(X) in such a way that the predicted values(Y) are as close to the actual values as possible. This difference between the actual values and predicted values is the error and it is used to evaluate the model. The error for any supervised Machine Learning algorithm comprises of 3 parts:

  1. Bias error
  2. Variance error
  3. The noise

While the noise is the irreducible error that we cannot eliminate, the other two i.e. Bias and Variance are reducible errors that we can attempt to minimize as much as possible.

What is bias?
Bias is the difference between the average prediction of our model and the correct value which we are trying to predict. Model with high bias pays very little attention to the training data and oversimplifies the model. It always leads to high error on training and test data.

What is variance?
Variance is the variability of model prediction for a given data point or a value which tells us spread of our data. Model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before. As a result, such models perform very well on training data but has high error rates on test data.

What is underfitting?
Underfitting happens when a model unable to capture the underlying pattern of the data. These models usually have high bias and low variance. It happens when we have very less amount of data to build an accurate model or when we try to build a linear model with a nonlinear data. Also, these kind of models are very simple to capture the complex patterns in data like Linear and logistic regression.

What is overfitting?
Overfitting happens when our model captures the noise along with the underlying pattern in data. It happens when we train our model a lot over noisy dataset. These models have low bias and high variance. These models are very complex like Decision trees which are prone to overfitting.

What is Bias-Variance trade-off?
If our model is too simple and has very few parameters then it may have high bias and low variance. On the other hand, if our model has large number of parameters then it’s going to have high variance and low bias. So we need to find the right/good balance without overfitting and underfitting the data.
This trade-off in complexity is why there is a trade-off between bias and variance. An algorithm can’t be more complex and less complex at the same time.

So, to build a good model, we need to find a good balance between bias and variance such that it minimizes the total error.

Look at the following graph.
The irreducible error denoted by red dashed line is constant.
Bias squared (yellow dashed line) and training error (black dashed line) decrease with complexity of the model whereas variance (blue dashed line) increases with complexity of the model.
We have to minimize the total error or generalization error, denoted by the green line.

To learn more about underfitting and overfitting check out: What is overfitting and underfitting in machine learning?

1 Like