Ensemble Methods: Random Forest & Boosting

Question 1

Ensemble methods improve performance by…

Accepted Answer

Combining models so their errors cancel out — Aggregating diverse models (voting/averaging) reduces variance and/or bias, usually outperforming any single member — the 'wisdom of crowds'.

Question 2

BAGGING (Bootstrap Aggregating) primarily reduces…

Accepted Answer

Variance, via bootstrap averaging — Each model trains on a random resample; averaging their predictions cancels out their independent errors, lowering variance (e.g. random forests).

Question 3

BOOSTING primarily reduces…

Accepted Answer

Bias, by sequentially fitting errors — Boosting builds learners in sequence, each focusing on examples the ensemble still gets wrong, steadily reducing bias.

Question 4

A Random Forest combines bagging with…

Accepted Answer

Random feature subsampling at each split — Besides bootstrapping rows, each split considers only a random subset of features, so trees disagree more — better variance reduction.

Question 5

In bagging, each base model is trained on…

Accepted Answer

A bootstrap sample — Sampling with replacement creates diverse training sets; about 63% of unique points appear in each, the rest forming out-of-bag validation data.

Question 6

Boosting trains its models…

Accepted Answer

Sequentially, correcting residual errors — Each new learner depends on the previous ones' mistakes, so boosting is inherently sequential — unlike bagging's parallel, independent trees.

Question 7

Gradient boosting fits each new tree to the…

Accepted Answer

Negative gradient — It performs gradient descent in function space: each tree predicts the residual errors, incrementally improving the ensemble. XGBoost/LightGBM are popular implementations.

Question 8

Ensemble base models are often 'weak learners', meaning each is…

Accepted Answer

Only slightly better than random guessing — Boosting especially combines many weak learners (e.g. depth-1 stumps) into a strong one; their diversity and sequential correction do the heavy lifting.

Question 9

The learning rate (shrinkage) in gradient boosting controls…

Accepted Answer

How much each new tree contributes — A low learning rate makes boosting take small, careful steps, reducing overfitting at the cost of needing more boosting rounds.

Question 10

Out-of-bag (OOB) error in a random forest provides…

Accepted Answer

A built-in validation estimate — Each tree's bootstrap omits ~37% of data; predicting those gives an honest, cross-validation-like error estimate without a separate hold-out set.

Model Type

Parameters

Data & View

Individual Tree Performance

Bagging or boosting?

Test your knowledge

Ensemble methods improve performance by…

Learn these first

Interview Questions

References & Further Reading

Love this visualization?