Gradient Boosting Machine (GBM)

An ensembling technique where instead of averaging the predictions of all models, each successive model predicts the error of the previous model. The errors are then summed to obtain the final prediction.
  1. The first model produces a prediction.
  2. The difference between this prediction and the actual value is obtained.
  3. The difference is now set as the target.
  4. The next model now attempts to predict this difference.
  5. Repeat steps 2-4 for as many models as desired.
  6. Sum all obtained differences.

While this technique tends to produce better results, is more likely to overfit. This is because the machine is trying to minimize the difference between the predictions and actual values in the training set.

Back to top