XGBoost GPU Acceleration · Training Speed and Memory — XGBoost

XGBoost supports GPU-accelerated training through the tree_method parameter. On datasets with more than 100K rows, GPU training can be 5–15× faster than CPU. This article covers setup, benchmarks, and practical considerations.

Enabling GPU Training

params = {
    'tree_method': 'gpu_hist',  # GPU histogram-based algorithm
    'predictor': 'gpu_predictor',  # GPU-based prediction  
    'gpu_id': 0,  # Which GPU (default 0)
}

# Or via scikit-learn API
model = xgb.XGBRegressor(tree_method='gpu_hist', gpu_id=0)

The hist algorithm (histogram-based) works on both CPU and GPU. gpu_hist is the GPU implementation of the same algorithm — not a different algorithm, just an accelerated one.

Benchmarks

On the Higgs dataset (10M rows, 28 features):

Configuration	Training Time	Speedup vs CPU
CPU `hist` (16 threads)	320s	1×
GPU `gpu_hist` (V100)	28s	11.4×
GPU `gpu_hist` (A100)	18s	17.8×

On the airline dataset (115M rows, 14 features):

Configuration	Time	Memory
CPU `hist`	4,800s	32 GB
GPU `gpu_hist` (V100)	340s	14 GB GPU

GPU training shows its greatest advantage on large datasets. The overhead of CPU→GPU data transfer is amortized over many boosting rounds.

Specifying GPU

xgboost-org 配图

Single GPU: gpu_id=0 (default). Multi-GPU: Not natively supported in Python package. Use Dask-XGBoost for distributed multi-GPU training.

For single-node multi-GPU:

params = {'tree_method': 'gpu_hist', 'n_gpus': 2}
# Distributes data across GPUs

When GPU Is Worth It

✅ Use GPU: When datasets exceed 100K rows, or when training time is measured in minutes on CPU. GPU gives 5–20× speedup on large data.

❌ Don’t use GPU: For small datasets (<10K rows) — CPU overhead of GPU transfer dominates. For very deep trees (depth >15) — GPU implementation is optimized for the common case (depth ≤ 12).

⚠️ Watch memory: GPU (gpu_hist) stores the entire dataset in GPU memory. For very wide datasets (thousands of features), GPU memory may be insufficient even if CPU memory is plentiful. Use max_bin to reduce memory usage (default 256, reduce to 128 or 64).

GPU vs CPU Differences

xgboost-org 配图

The gpu_hist algorithm is deterministic — unlike CPU hist which may have non-deterministic behavior with multithreading. This means GPU training is reproducible with identical results on the same hardware.

However, gpu_hist uses floating-point operations on the GPU, which may produce slightly different splits compared to CPU due to floating-point arithmetic differences. These differences are typically negligible (<0.0001 difference in predictions) but may matter in sensitive applications.