Boosting

My last blog I explained bagging and now it’s time to discuss boosting, another method used in ensemble methods. The difference between bagging and boosting methods is strong and weak learners. Random forests used strong learners. Strong learners have the goal of doing as well as possible. Weak learners, boosting, learn by training a single weak learner and figuring out what examples went wrong. Then it builds another weak learner focusing on where the last learner went wrong and continues this process until a set number of learners is trained or the model plateaus. In the end all of the weak learners are put together to create one strong predictor. The bagging method uses all of the end results from the trees to pool together a prediction, the boosting gives weight to certain trees that make harder decisions and uses the weights to decide what the prediction will be.

Boosting is used in decision tree methods but unlike bagging, which trains each tree independently, boosting trains one tree and uses the information it learned to train the next tree.

There are 2 main boosting models Adaboost and Gradient boost. Adaboost is similar to what I described above by basing the weight of each tree based on how hard it was to classify. The way it decides if a tree is hard is based on how many predictions it gets correctly in that tree. Gradient boost is similar but uses the residuals to see how wrong the answers are not just correct or incorrect.