XGBoost Hyperparameter Tuning · A Systematic Guide

Tuning XGBoost is sensitive to parameter ordering. Tuning parameters in the wrong order — or tuning everything at once — leads to suboptimal results and wasted compute. The established methodology tunes parameters in tiers, from most to least impactful.

The Tier System

Tier 1: Tree Structure (highest impact)

n_estimators and learning_rate are tuned together. Higher n_estimators requires lower learning_rate. The classic approach:

model = xgb.XGBRegressor(n_estimators=1000, learning_rate=0.01, early_stopping_rounds=50)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])
optimal_n = model.best_iteration  # e.g., 347

Then set n_estimators=optimal_n with a higher learning_rate and re-tune.

Tier 2: Regularization

max_depth (3–10), min_child_weight (1–10), gamma (0–5), lambda (0.1–10), alpha (0–1). These control overfitting. Tune together because they interact — deep trees need more regularization.

Tier 3: Subsampling

subsample (0.5–1.0), colsample_bytree (0.5–1.0), colsample_bylevel, colsample_bynode. These add randomness and reduce overfitting. Subsample of 0.8 is a good default.

Tier 4: Learning rate final

Reduce learning_rate and increase n_estimators proportionally for final model. Halving the learning rate typically requires doubling n_estimators.

Practical Search Strategies

Grid search: Exhaustive but exponential. Use only for Tier 1 (2 parameters max).

Random search: More efficient than grid search for 3+ parameters. Sample 50–200 combinations from reasonable ranges.

Bayesian optimization (Optuna, Hyperopt): Most efficient for 5+ parameters. Optuna’s TPESampler works well with XGBoost:

import optuna
def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 12),
        'learning_rate': trial.suggest_float('lr', 0.01, 0.3, log=True),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'lambda': trial.suggest_float('lambda', 0.1, 10, log=True),
    }
    model = xgb.XGBRegressor(**params, n_estimators=100, early_stopping_rounds=20)
    model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
    return model.best_score
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=50)

Common ranges:

• max_depth: 3–10. Deeper = more complex. Start at 6.
• learning_rate: 0.01–0.3. Lower = better generalization, more trees needed.
• subsample: 0.5–1.0. 0.8 is the safe default.
• colsample_bytree: 0.3–1.0. 0.5–0.8 often improves generalization.
• min_child_weight: 1–10. Higher = more conservative. Important for imbalanced classification.
• gamma: 0–5. Increase if tree depth is limited by regularization, not data.