Lecture 10: Ensemble Learning

Slides

Link for the slides.

Today’s Topics: Ensemble Learning

Ensemble learning is a simple idea. Instead of training one model, we train multiple models and combine the results. A popular ensemble learning algorithm is random forests that combines many decision trees that are trained on random subsets of the features. To build a classifier a majority vote is taken. There are various techniques to improve the system including boosting, bagging and gradient boosting. As you will see below these are not limited to random forests, but to other combinations of learning algorithms. There is a lot extensions and refinements of ensemble methods including AdaBoost. We will only cover gradient boosting and not go too much into the mathematics behind ensemble learning (that is reserved for a more advanced course). The aim of this lecture is to give you access to another powerful machine learning technique.

One thing that we did not cover in Lecture 8 was using decision trees for regression. Although the idea is not that complicated you should spend some time understand how Regression trees can be used for regression. Although you will not be examined on other algorithms for constructing decisions trees (we covered ID3 in Lecture 8 ) you should be aware that there are other algorithms.

Constructing the perfect decision tree is a computationally hard problem (in fact it is NP-complete). For machine learning we want fast and efficient algorithms for learning, and most decision tree algorithms in the literature are approximations.

Reading Guide

What should I know by the end of this lecture?

  • How do I use decision trees for regression?
  • What is ensemble learning?
  • How do random forest work?
  • What is boosting and bagging?
  • What is gradient boosting?