Learning Rate and Number of Estimators in XGBoost: The Critical Tradeoff

The Learning Rate (eta) vs n_estimators Tradeoff in XGBoost

The learning rate eta and the number of boosting rounds n_estimators are the two most tightly coupled hyperparameters in XGBoost. They cannot be tuned in isolation. The core tradeoff is simple: a smaller eta requires more trees to achieve the same level of model complexity, but almost always yields better generalization — provided you can afford the computational cost.

What eta Actually Does

In gradient boosting, each new tree is trained to predict the negative gradient of the loss function with respect to the current ensemble’s predictions. The tree outputs raw leaf scores (also called terminal node weights). Instead of adding these scores directly to the ensemble, XGBoost multiplies them by eta before the additive update:

$$ \hat{y}_i^{(t)} = \hat{y}_i^{(t-1)} + \eta \cdot f_t(x_i) $$

where $\hat{y}_i^{(t)}$ is the prediction for instance $i$ after $t$ trees, and $f_t(x_i)$ is the raw output of the $t$-th tree. When eta = 1.0, each tree contributes its full fitted values. When eta = 0.1, each tree contributes only 10% of what it learned. This is called shrinkage.

Shrinkage does not change how trees are grown — splits are still determined by the same gain calculations — but it changes how much each tree’s output is trusted. The remaining 90% of the signal must be picked up by subsequent trees.

The Theory: Shrinkage as Regularization

The idea of shrinkage in gradient boosting comes directly from Jerome Friedman’s seminal work. In his 1999 paper “Greedy Function Approximation: A Gradient Boosting Machine” and the 2001 follow-up “Greedy function approximation: A gradient boosting machine” (Annals of Statistics), Friedman demonstrated that introducing a learning rate parameter consistently improved generalization performance. The insight is that gradient boosting performs a form of gradient descent in function space. In standard optimization, smaller step sizes with more iterations often find better minima. The same holds here: a smaller eta with more trees allows the ensemble to navigate the loss landscape more carefully, avoiding overshooting and settling into better-generalizing solutions.

Friedman’s empirical finding was striking: a smaller eta with a correspondingly larger number of trees almost always outperforms a larger eta with fewer trees, given sufficient computational budget. This has been replicated extensively in practice and is now a foundational principle of gradient boosting methodology.

The regularization interpretation is that eta controls the effective capacity of each individual tree. By shrinking each tree’s contribution, you force the ensemble to distribute learning across many weak learners, which reduces variance. This is analogous to the $\ell_2$ penalty in linear models, though the mechanism is different — here it’s a constraint on the optimization path rather than a penalty on the objective function.

The Computational Cost

The tradeoff is paid in three currencies:

  • Training time: Each additional tree requires a full pass over the data to compute gradients and hessians, sort features for split finding, and evaluate candidate splits. Doubling the number of trees roughly doubles training time.
  • Model size: More trees means a larger serialized model. For deployment scenarios with memory constraints or bandwidth limitations (e.g., edge devices, browser inference), this matters.
  • Inference latency: Prediction requires traversing every tree in the ensemble. More trees means more conditional branches and memory lookups per prediction. For batch inference this is often negligible due to vectorized implementations, but for single-row, low-latency serving it can be significant.

The rough heuristic is that halving eta requires approximately doubling n_estimators to maintain similar training loss. In practice the relationship is not perfectly linear — it depends on data complexity, tree depth, and subsampling rates — but the proportional scaling holds well enough for planning.

Empirical Guidance on eta Values

The following ranges are based on extensive community experience and the XGBoost documentation’s

  • eta = 0.3 (default): Fast convergence, coarse granularity. Good for quick baselines, exploratory analysis, and problems where you need a reasonable model in seconds. Expect 50-100 trees to be sufficient for many tabular datasets. The XGBoost parameter documentation lists this as the default, and it’s a reasonable starting point when you don’t yet know what performance level is achievable.

  • eta = 0.1: The most common default in practice. Provides a good balance between training speed and model quality. Typically requires 100-300 trees. This is the standard starting point recommended in the scikit-learn GradientBoostingClassifier documentation and is widely used in Kaggle competitions for initial submissions.

  • eta = 0.01 to 0.05: Fine-grained learning. Expect to need 500-2000+ trees. This range is where you go when you’ve already tuned other parameters and are squeezing out the last 0.5-1% of performance. The improvement over eta = 0.1 is often measurable but modest. Worth it for production systems where every basis point of AUC or every decimal of RMSE matters.

  • eta < 0.01: Diminishing returns. The number of trees required becomes prohibitive (often 5000+), and the marginal improvement in validation metrics is typically negligible. There are edge cases — extremely noisy data, very high-dimensional sparse features — where extremely small eta helps, but these are rare.

The eta × n_estimators Constant Heuristic

A useful mental model: the product eta × n_estimators is roughly constant for a given dataset and tree complexity. If you find that eta = 0.1 with 200 trees gives good results, then eta = 0.05 will likely need around 400 trees, and eta = 0.01 will need around 2000 trees. This is not a precise law — interactions with max_depth, subsample, and the inherent difficulty of the problem cause deviations — but it’s an excellent starting heuristic for budget planning.

How to Choose: Grid Search with Early Stopping

The most reliable method for joint tuning uses early stopping to let the data tell you how many trees are actually needed:

import xgboost as xgb
from sklearn.model_selection import GridSearchCV
import numpy as np

# Base model with high n_estimators and early stopping
base_model = xgb.XGBRegressor(
    n_estimators=2000,          # Set high; early stopping will reduce
    early_stopping_rounds=20,   # Stop if no improvement for 20 rounds
    objective='reg:squarederror',
    random_state=42
)

# Parameter grid — only vary eta initially
param_grid = {
    'eta': [0.01, 0.03, 0.05, 0.1, 0.2, 0.3],
    'max_depth': [3, 5, 7],
    'subsample': [0.8, 1.0]
}

# Grid search with cross-validation
# Note: eval_set must be passed via fit_params for early stopping
grid = GridSearchCV(
    base_model,
    param_grid,
    cv=5,
    scoring='neg_mean_squared_error',
    n_jobs=-1,
    verbose=1
)

# fit_params passes the validation set for early stopping
# In practice, use a separate holdout set or split within CV
grid.fit(X_train, y_train)

print(f"Best params: {grid.best_params_}")
print(f"Best score: {grid.best_score_:.4f}")

A more refined approach uses a separate validation set directly, which gives you visibility into the actual stopping iteration for each eta value:

import xgboost as xgb
from sklearn.model_selection import train_test_split

X_tr, X_val, y_tr, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42
)

results = {}
for eta in [0.01, 0.03, 0.05, 0.1, 0.2, 0.3]:
    model = xgb.XGBRegressor(
        n_estimators=3000,
        eta=eta,
        max_depth=5,
        subsample=0.8,
        early_stopping_rounds=30,
        random_state=42
    )
    model.fit(
        X_tr, y_tr,
        eval_set=[(X_val, y_val)],
        verbose=False
    )
    results[eta] = {
        'best_iteration': model.best_iteration,
        'best_score': model.best_score,
        'n_estimators_used': model.best_iteration
    }
    print(f"eta={eta:.3f}: stopped at iter {model.best_iteration}, "
          f"val score={model.best_score:.4f}")

The optimal eta is typically the smallest value whose validation loss curve doesn’t plateau prematurely (indicating underfitting) or start rising (indicating overfitting) before early stopping triggers. If eta = 0.01 stops at iteration 300 with a validation score worse than eta = 0.05 at iteration 800, the larger eta is preferable — it converged further within the patience window.

Interaction with Other Hyperparameters

eta does not operate in isolation. Several other parameters have regularization-like effects that can substitute for or complement shrinkage:

  • subsample: Stochastic gradient boosting, introduced by Friedman in his 2002 paper “Stochastic gradient boosting” (Computational Statistics & Data Analysis), randomly samples a fraction of training data for each tree. A subsample of 0.8 with eta = 0.1 often outperforms subsample = 1.0 with eta = 0.05 on the same tree budget, because the stochasticity itself acts as regularization. When using aggressive subsampling (< 0.8), you may need a larger eta to maintain convergence speed.

  • reg_lambda and reg_alpha: L2 and L1 regularization on leaf weights. These directly penalize large leaf outputs, which is a different mechanism than shrinkage but has a similar effect of constraining each tree’s contribution. Strong leaf regularization can sometimes allow slightly larger eta values.

  • max_depth: Deeper trees have higher variance per tree. With max_depth > 6, smaller eta values become more important to prevent overfitting. With very shallow trees (max_depth = 1 or 2), the ensemble behaves more like a pure additive model and eta can often be larger.

  • min_child_weight: This is effectively a regularization parameter based on instance weight (Hessian) sums. Higher values prevent splits on small groups of instances, which indirectly constrains tree complexity and interacts with the shrinkage dynamic.

The general tuning principle: tune the tree complexity parameters first (max_depth, min_child_weight) with a moderate eta (0.1), then jointly tune eta, subsample, and regularization parameters.

FAQ

Should I always use a small eta if I have time?

Not necessarily. Extremely small eta values (< 0.01) can lead to models that are so conservative they never fully fit the signal within a practical number of trees. The validation curve often plateaus at a suboptimal level. There’s a floor beyond which adding more trees doesn’t help — the optimization gets stuck in a regime where each step is too small to escape local plateaus in function space. The diminishing returns are real.

Does early stopping make the eta choice less important?

Early stopping mitigates the risk of overfitting for any given eta, but it doesn’t eliminate the quality difference between eta values. A model with eta = 0.05 that stops at iteration 600 will typically outperform a model with eta = 0.3 that stops at iteration 80, even though both used early stopping. Early stopping prevents the larger-eta model from overfitting further, but it doesn’t recover the generalization benefit of the finer-grained optimization path.

How does eta interact with GPU training?

GPU training dramatically reduces the per-tree training cost, making large n_estimators more practical. With GPU acceleration, eta = 0.01 with 2000 trees might train in the same wall-clock time as eta = 0.1 with 200 trees on CPU. The tradeoff shifts: computational cost is less binding, so you can afford smaller eta values more often.

What about the learning_rate parameter in the native XGBoost API?

learning_rate is an alias for eta in the native XGBoost Python API and in the scikit-learn wrapper. They are identical. The parameter was renamed to learning_rate in newer versions for clarity, but eta remains accepted for backward compatibility.