XGBoost Cross-Validation · cv() vs Manual K-Fold

XGBoost provides a built-in cv() function for cross-validation that’s more efficient than manual scikit-learn loops — it reuses DMatrix across folds and returns per-iteration metrics. This article covers usage patterns and common pitfalls.

Basic Usage

import xgboost as xgb

dtrain = xgb.DMatrix(X, label=y)
params = {'max_depth': 6, 'eta': 0.1, 'objective': 'binary:logistic'}

cv_results = xgb.cv(
    params, dtrain,
    num_boost_round=1000,
    nfold=5,
    early_stopping_rounds=50,
    metrics='logloss',
    seed=42
)
# Output: DataFrame with train-logloss-mean, test-logloss-mean, etc.

Key Parameters

Parameter	Purpose
nfold	Number of folds (default 5)
stratified	Stratify folds for classification (default True)
metrics	List of evaluation metrics
early_stopping_rounds	Stop when test metric doesn’t improve
as_pandas	Return DataFrame (default True)
seed	Random seed for fold splitting
shuffle	Shuffle before splitting (default True)

Interpreting Results

print(f"Best iteration: {cv_results['test-logloss-mean'].idxmin()}")
# Or with early stopping: use the final row (early stopping stopped there)
best_round = len(cv_results) - 1
best_score = cv_results['test-logloss-mean'].iloc[best_round]
# Standard deviation across folds:
score_std = cv_results['test-logloss-std'].iloc[best_round]

Stratified Folds

For classification with imbalanced classes, stratified=True (default) ensures each fold maintains the same class distribution as the full dataset. For regression, stratification is not applicable.

Custom stratification (e.g., by group, by time period):

folds = [(train_idx, val_idx), ...]  # Pre-specified indices
cv_results = xgb.cv(params, dtrain, folds=folds, num_boost_round=1000)

CV for Hyperparameter Tuning

best_params = None
best_score = float('inf')

for max_depth in [3, 5, 7, 9]:
    for subsample in [0.6, 0.8, 1.0]:
        params = {'max_depth': max_depth, 'subsample': subsample, ...}
        cv = xgb.cv(params, dtrain, nfold=5, num_boost_round=500, 
                    early_stopping_rounds=30, metrics='rmse')
        score = cv['test-rmse-mean'].min()
        if score < best_score:
            best_score = score
            best_params = params.copy()

CV vs Manual K-Fold

xgb.cv() is faster than manual sklearn.model_selection.KFold because:

1. DMatrix is constructed once, not $k$ times
2. Internal slicing of the DMatrix is optimized (no data copies)
3. Training state is partially reusable across folds

Use manual K-Fold when you need custom preprocessing per fold (which should be rare with tree-based models).

Common Pitfall: Overfitting During Hyperparameter Tuning

If you use the same CV splits for hyperparameter selection AND final evaluation, you’ll overfit to the validation folds. Solution: nested cross-validation — an outer CV for evaluation, an inner CV for hyperparameter tuning. This is computationally expensive (25–100× training cost) but provides unbiased estimates.