GPU Acceleration in XGBoost

Updated 2026-09-21 for XGBoost 3.4: setup, parameters and code now use the current device API; the pre-2.0 gpu_hist, gpu_id and predictor settings are deprecated, and two parameters that XGBoost ignores (n_gpus, gpu_page_size) were removed from the examples. Speed-up figures come from the original version of this article and have not been re-verified. See the official GPU support page for current details.

If you have an NVIDIA GPU and datasets that aren’t trivially small, setting device='cuda' (with tree_method='hist') will typically accelerate your XGBoost training by 3-20× with a single parameter change. The GPU implementation is not a thin wrapper—it is a complete re-engineering of the histogram-based tree builder to run natively on CUDA cores, keeping all critical computations (histogram construction, split finding, and prediction) on-device and avoiding expensive CPU-GPU transfers during training iterations.

This guide covers the mechanics, setup, performance characteristics, and practical constraints of GPU-accelerated XGBoost.

How GPU Acceleration Works in XGBoost

XGBoost’s GPU backend runs the hist tree method on CUDA devices, an implementation of the histogram-based gradient boosting algorithm designed from the ground up for massively parallel architectures. You select it with device='cuda'; older code spells the same thing tree_method='gpu_hist', which is deprecated since XGBoost 2.0.

The Histogram Algorithm on GPU

In histogram-based tree building, the algorithm bins continuous features into discrete buckets (default 256) and constructs histograms of gradient statistics for each feature. For a given tree node, finding the best split requires scanning these histograms to evaluate candidate split points. On a CPU, this is inherently sequential or limited to thread-level parallelism within a single node. On a GPU, the same operation is decomposed into thousands of parallel threads, each responsible for a subset of the data or a subset of histogram bins.

The core operations that run entirely on GPU are:

• Histogram construction: Each row’s feature values are mapped to bin indices, and gradient/hessian contributions are atomically accumulated into per-feature histogram arrays. This is the most memory-intensive step and benefits enormously from the GPU’s high memory bandwidth.
• Split evaluation: Once histograms are built, scanning for the optimal split point across all features and all bins is embarrassingly parallel—the GPU evaluates thousands of candidate splits simultaneously.
• Node expansion: After the best split is found, rows are partitioned into left and right child nodes. This scatter/gather operation is performed by GPU kernels using the split thresholds.
• Prediction inference: When making predictions, the trained tree ensemble traverses all trees in parallel across the batch dimension, with each GPU thread handling a subset of samples.

Memory Model and Constraints

For full acceleration, the training data must fit in GPU VRAM. XGBoost copies the input matrix to the GPU at the start of training and maintains all intermediate structures—gradient vectors, histogram buffers, and tree metadata—on-device. If the dataset exceeds available VRAM, training fails with an out-of-memory error rather than silently degrading to slower CPU-GPU transfers, unless you deliberately use XGBoost’s external-memory training.

The GPU implementation uses CUDA exclusively. There is currently no support for AMD ROCm or Apple Metal backends. This means you need an NVIDIA GPU with a supported compute capability (generally Maxwell architecture or newer) and appropriate CUDA drivers installed.

Installation and Verification

The default xgboost package on PyPI includes GPU support on Linux (x86_64 and aarch64) and Windows x86_64; macOS builds have none. According to the official installation guide:

# PyPI: full package with GPU support (Linux wheels are built with CUDA 13.x)
pip install xgboost

# PyPI: for drivers that support CUDA 12 but not CUDA 13
pip install xgboost-cu12

# PyPI: smaller CPU-only package
pip install xgboost-cpu

# Conda: detects the GPU automatically; force the GPU variant if needed
conda install -c conda-forge py-xgboost
conda install -c conda-forge "py-xgboost=*=cuda*"

Multi-GPU training through Dask or Spark is supported on Linux only.

After installation, verify that CUDA support is enabled:

import xgboost as xgb
build_info = xgb.build_info()
print(build_info['USE_CUDA'])  # Must be True

If USE_CUDA is False, you have the CPU-only build. On Linux, this usually means the CUDA toolkit libraries (libcudart.so) are not on the library path. The XGBoost installation guide provides platform-specific troubleshooting.

The tree_method Parameter

XGBoost chooses the algorithm with tree_method and the hardware with device (added in XGBoost 2.0). The relevant tree_method options are:

• hist: The histogram-based builder and the default (auto now means hist). It runs on the CPU by default and on an NVIDIA GPU when you set device='cuda'. On CPU it is also the right choice for small datasets where GPU overhead dominates.
• gpu_hist (deprecated): The pre-2.0 way of asking for GPU training. Replace it with tree_method='hist' plus device='cuda'.
• approx: The CPU approximate greedy algorithm. Largely superseded by hist for CPU training.
• exact: The exact greedy algorithm. CPU-only, computationally expensive, and rarely necessary.

To select a specific GPU in a multi-GPU machine, give its ordinal in device; device='cpu' trains on the CPU:

params = {
    'tree_method': 'hist',
    'device': 'cuda:0',  # first GPU; 'cuda:1' for the second, 'cpu' for CPU
    'max_depth': 6,
    'learning_rate': 0.1
}

Performance Characteristics

GPU acceleration does not provide a uniform speedup across all workloads. The benefit depends on dataset dimensions, hyperparameters, and hardware.

Where GPU Excels

GPU acceleration shows its greatest advantage when the computational workload is large enough to saturate the GPU’s parallel cores and memory bandwidth. This typically means:

• Large row counts: Datasets with more than 10,000–50,000 rows. Below this threshold, CPU-GPU data transfer time and kernel launch overhead can negate the parallel speedup.
• Many features: Wide datasets (hundreds or thousands of features) benefit because histogram construction and split evaluation scale with feature count, and the GPU parallelizes across features naturally.
• Many trees and deep trees: Boosting rounds with 500+ trees and deeper trees (max_depth ≥ 6) create more histogram-building operations, amplifying the GPU’s throughput advantage.
• Dense numeric data: The GPU backend is optimized for dense float32/float64 matrices.

Typical speedups observed in practice:

• Mid-range consumer GPUs (RTX 3060, RTX 4060): 3–8× over an 8-core CPU
• High-end GPUs (RTX 4090, A100): 10–25× for large-scale training jobs

The NVIDIA developer blog on XGBoost GPU acceleration provides benchmark examples across different hardware configurations.

Where GPU Doesn’t Help

• Small datasets (<10K rows, <50 features): The fixed cost of GPU memory allocation, data transfer, and kernel launch overhead often makes GPU training slower than CPU hist. For such datasets, use tree_method='hist' on CPU.
• Extremely sparse data: While XGBoost’s GPU backend supports sparse input formats (CSR/CSC matrices), the irregular memory access patterns of sparse operations are less GPU-friendly. The CPU hist method often handles high sparsity more efficiently.
• Single-tree or very shallow ensembles: If you’re training only a few shallow trees, there isn’t enough parallel work to justify GPU acceleration.
• Mac systems: Apple Silicon Macs and Intel Macs with AMD GPUs cannot use CUDA. Use tree_method='hist' on the CPU, which is well-optimized for Apple’s M-series chips.

Benchmarking Code Example

The following example compares GPU and CPU training with hist across different dataset sizes, using synthetic data to isolate the computational scaling:

import xgboost as xgb
import numpy as np
import time
from sklearn.datasets import make_classification

def benchmark(n_samples, n_features, device):
    X, y = make_classification(
        n_samples=n_samples,
        n_features=n_features,
        n_informative=n_features // 2,
        random_state=42
    )
    X = X.astype(np.float32)

    params = {
        'tree_method': 'hist',
        'device': device,
        'max_depth': 8,
        'learning_rate': 0.1,
        'objective': 'binary:logistic',
        'eval_metric': 'logloss',
        'verbosity': 0
    }

    dtrain = xgb.DMatrix(X, label=y)
    start = time.perf_counter()
    xgb.train(params, dtrain, num_boost_round=500)
    elapsed = time.perf_counter() - start

    print(f"  {device:>10s} | {n_samples:>8d} rows × {n_features:>4d} features | {elapsed:>6.2f}s")
    return elapsed

sizes = [(10000, 50), (100000, 100), (500000, 200)]

for n_samples, n_features in sizes:
    print(f"\nDataset: {n_samples} rows, {n_features} features")
    t_gpu = benchmark(n_samples, n_features, 'cuda')
    t_cpu = benchmark(n_samples, n_features, 'cpu')
    if t_gpu > 0:
        print(f"  Speedup: {t_cpu/t_gpu:.1f}×")

On a system with an RTX 3080 and Ryzen 5950X, this script typically shows marginal speedup for the 10K-row dataset, 4–6× speedup at 100K rows, and 8–12× speedup at 500K rows.

Multi-GPU Training

A single xgb.train() or fit() call uses one GPU. To use several GPUs, on one machine or across many, XGBoost relies on its distributed integrations: Dask, Spark and PySpark (Linux only). Each worker gets one GPU and a partition of the data, builds histograms locally, and the workers exchange gradient statistics for global split finding. There is no n_gpus parameter; if you pass it, XGBoost ignores it and prints a “Parameters … are not used” warning. See the official Distributed XGBoost with Dask tutorial for a starting point.

Scaling is sub-linear due to communication overhead. In practice, 2 GPUs might yield 1.6–1.8× speedup, and 4 GPUs might yield 2.5–3.2× speedup over a single GPU, depending on dataset size and network topology (NVLink vs. PCIe). The communication cost of all-reduce operations on gradient histograms grows with the number of features and bins, so wide datasets benefit proportionally less from multi-GPU scaling.

GPU Memory Management

GPU out-of-memory (OOM) errors are the most common failure mode in GPU-accelerated training. XGBoost provides several mechanisms to control memory usage:

Start with QuantileDMatrix

The official GPU guide’s first recommendation for out-of-memory errors is xgboost.QuantileDMatrix, which builds the quantised representation directly instead of keeping a full copy of the input. The scikit-learn interface uses it automatically. With the native interface:

dtrain = xgb.QuantileDMatrix(X_train, label=y_train)
booster = xgb.train({'tree_method': 'hist', 'device': 'cuda'}, dtrain, num_boost_round=500)

For data that still does not fit, XGBoost offers external-memory training with a data iterator (see the official GPU and external-memory documentation). There is no gpu_page_size parameter; XGBoost ignores it if passed.

Other Memory Reduction Strategies

• Reduce max_bin: The default 256 bins means each feature histogram requires 256 float elements. Reducing to 128 or 64 bins cuts memory proportionally and often has minimal impact on model quality.
• Use subsampling: subsample=0.8 reduces the effective dataset size per boosting round, lowering peak memory.
• Reduce max_depth: Shallower trees require fewer gradient histograms active simultaneously.
• Use float32: Ensure your input data is float32 rather than float64; this halves the memory taken by your own input array before XGBoost builds its compressed training matrix.

When to Stay on CPU

Despite the allure of GPU acceleration, several scenarios call for CPU training:

• Small data: If your dataset has fewer than 10K rows, the CPU hist method completes training in seconds anyway, and GPU overhead makes it slower.
• Sparse data with extreme sparsity: Datasets with >99% zeros (e.g., one-hot encoded text features) often train faster on CPU due to optimized sparse matrix routines that avoid the overhead of GPU sparse kernels.
• Mac or AMD systems: Without an NVIDIA GPU, GPU training is unavailable. The CPU hist method is highly optimized and provides excellent performance on modern multi-core CPUs like Apple’s M-series chips.
• Cost-sensitive cloud environments: If you’re paying for GPU instances by the hour and training is a small fraction of your workflow, the cost of keeping a GPU instance alive may outweigh the training speed benefit.

Cloud GPU Options

For users without local GPU hardware, all major cloud providers offer GPU instances suitable for XGBoost training:

• AWS: p3 (V100), p4d (A100), and g5 (A10G) instances
• Google Cloud: A2 instances (A100) and G2 instances (L4)
• Azure: NCv3 (V100) and NC A100 v4 series
• Google Colab: Free tier provides T4 GPUs; Colab Pro+ provides access to V100 and A100

For most XGBoost workloads, a single mid-range GPU (T4, A10G) provides excellent price-performance. Only very large datasets (millions of rows, thousands of features) benefit from high-end datacenter GPUs like the A100.

Energy Efficiency

GPU training draws significantly more instantaneous power than CPU training—a high-end GPU can consume 300–450W under load, compared to 65–150W for a consumer CPU. However, the dramatic reduction in wall-clock time often means lower total energy consumption. A job that runs for 10 minutes on a 350W GPU consumes approximately 58 watt-hours, while the same job running for 90 minutes on a 150W CPU consumes 225 watt-hours. For batch training workflows, the energy tradeoff typically favors GPU acceleration.

---

FAQ

Does GPU training change the model mathematically? The algorithm is identical in principle, but floating-point arithmetic on GPUs can produce slightly different split points due to differences in accumulation order and atomic operation semantics. These differences are typically negligible for prediction quality but can cause minor divergence in feature importance scores and tree structures.

Can I train on GPU and deploy on CPU? Yes. The official documentation states that a model can be used on any device regardless of the one used to train it. The serialized model format is the same either way.

Why does my GPU utilization fluctuate during training? XGBoost alternates between highly parallel phases (histogram construction, which saturates the GPU) and sequential or lightly parallel phases (node evaluation, tree metadata updates). This is normal and reflects the inherent serial dependencies in tree building.

Does XGBoost support half-precision (float16) training on GPU? Not as a training option: there is no parameter that switches XGBoost to half-precision arithmetic. Keeping the input in float32 and using QuantileDMatrix are the supported ways to save memory.

How do I debug GPU memory issues? Set verbosity=2 in parameters and monitor the logs for memory allocation sizes. Tools like nvidia-smi show real-time GPU memory usage. If you see memory growing rapidly during the first few boosting rounds and then crashing, reduce max_bin or max_depth first.