XGBoost Learning Rate · Eta, Shrinkage, and Tree Count
The learning rate (eta) is XGBoost’s most impactful hyperparameter, governing how much each tree contributes to the final prediction. Together with n_estimators, it forms the primary axis for controlling the bias-variance tradeoff.
The Shrinkage Principle
In gradient boosting, each tree $f_t$ is trained to predict the negative gradient of the loss. The model update is:
$$ \hat{y}^{(t)} = \hat{y}^{(t-1)} + \eta \cdot f_t(x) $$
Where $\eta$ (eta, learning rate) shrinks the contribution of each tree. This is Friedman’s shrinkage — the insight that taking smaller steps with more trees generalizes better than taking larger steps with fewer trees.
The Eta × n_estimators Tradeoff
For a given problem, the product $\eta \times n_estimators$ approximates a constant. Halving $\eta$ requires roughly doubling $n_estimators$ to achieve the same level of fit.
| Eta | Typical n_estimators | Total Product |
|---|---|---|
| 0.3 | 100 | 30 |
| 0.1 | 300 | 30 |
| 0.05 | 600 | 30 |
| 0.01 | 3000 | 30 |
This isn’t exact — very small $\eta$ (0.001) may require more than the product suggests because early trees waste capacity on large errors.
Diminishing Returns
Below $\eta$ ≈ 0.01, improvements in generalization become marginal while training time scales linearly. The sweet spot for most problems is $\eta \in [0.05, 0.1]$.
Empirical finding (Chen & Guestrin, 2016): On benchmark datasets, $\eta = 0.1$ with 300–500 rounds achieves 98%+ of the test performance of $\eta = 0.01$ with 3000+ rounds, at 6–10× lower training cost.
The Early Stopping Shortcut
Instead of tuning $\eta$ and $n_estimators$ independently: set $\eta = 0.1$, a large $n_estimators$ (1000), and use early stopping:
model = xgb.XGBRegressor(n_estimators=1000, learning_rate=0.1, early_stopping_rounds=50)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
print(f"Best: {model.best_iteration} trees, lr=0.1")The optimal $n_estimators$ is determined by validation performance. Then re-train with a smaller $\eta$ and $n_estimators_{new} \approx n_estimators_{old} \times \frac{\eta_{old}}{\eta_{new}}$.
Learning Rate Scheduling
XGBoost supports per-round learning rates via callback:
callbacks = [xgb.callback.LearningRateScheduler(lambda epoch: 0.1 * (0.99 ** epoch))]
model = xgb.train(params, dtrain, num_boost_round=500, callbacks=callbacks)Decaying the learning rate over rounds can improve convergence — early rounds use larger steps to capture broad patterns; later rounds use smaller steps for fine-grained refinement.
Practical Recommendation
- Set $\eta = 0.1$, large $n_estimators$ with early stopping
- Find best_iteration
- Optionally: reduce $\eta$ by factor of 2, multiply best_iteration by 2, retrain on full data
- Don’t go below $\eta = 0.01$ unless working with millions of rows and training time is not a constraint