XGBoost DMatrix · The Internal Data Structure

xgb.DMatrix is XGBoost’s optimized internal data structure. While the scikit-learn API (XGBRegressor, XGBClassifier) abstracts it away, understanding DMatrix is essential for advanced usage: external memory training, custom evaluation, and native API access.

What DMatrix Does

DMatrix converts raw data (NumPy, pandas, scipy sparse) into a compressed binary format that XGBoost’s C++ backend can read efficiently. The conversion:

1. Sorts feature values for histogram computation
2. Pre-computes feature quantiles (for tree_method='approx')
3. Handles missing values (stored as a bitmask)
4. Optionally stores labels, weights, and feature names

Once constructed, the same DMatrix can be reused across multiple training runs — avoiding repeated preprocessing.

Construction

import xgboost as xgb

dtrain = xgb.DMatrix(X_train, label=y_train)
dtrain = xgb.DMatrix(X_train, label=y_train, weight=sample_weights)
dtrain = xgb.DMatrix(X_train, label=y_train, missing=-999)

# From file (LibSVM format)
dtrain = xgb.DMatrix('train.txt')

# From CSV
dtrain = xgb.DMatrix('train.csv?format=csv&label_column=0')

# With feature names
dtrain = xgb.DMatrix(X_train, label=y_train, feature_names=feature_names)

DMatrix Attributes

dtrain.num_row()       # Number of rows
dtrain.num_col()       # Number of columns
dtrain.feature_names   # Feature names
dtrain.get_label()     # Get labels
dtrain.get_weight()    # Get sample weights
dtrain.get_base_margin()  # Get base margins (initial predictions)
dtrain.set_info(label=..., weight=..., base_margin=...)  # Update after creation

Saving and Loading

DMatrix supports a binary format (.buffer) that preserves the compressed representation:

dtrain.save_binary('train.buffer')
dtrain = xgb.DMatrix('train.buffer')

Binary format is ~2–4× smaller than CSV and loads 5–10× faster. Use for large datasets that will be reused across experiments.

External Memory Training

When a dataset exceeds RAM, XGBoost can train from disk using an external memory mode:

dtrain = xgb.DMatrix('large_train.txt#cacheprefix=mycache')
model = xgb.train(params, dtrain)

XGBoost page-blocks the data to disk, loading one block at a time. Specify #cacheprefix to control where cached page files are written.

Limitations:

• Only works with tree_method='approx' (not exact or hist)
• Slower than in-memory training (disk I/O dominates)
• Requires careful cache management (page files can be large)

Data Iterator

For truly massive datasets (e.g., streaming from a database), use a Python iterator:

class DataIterator:
    def __init__(self, chunks):
        self.chunks = chunks
        self.it = 0
    
    def next(self, input_data):
        if self.it == len(self.chunks):
            return 0  # Stop iteration
        X, y = self.chunks[self.it]
        input_data(data=X, label=y)
        self.it += 1
        return 1  # Continue

it = DataIterator(data_chunks)
model = xgb.train(params, it)

The next method returns 0 when data is exhausted. This pattern works for any data source — streaming APIs, databases, data lakes.

When to Use DMatrix Directly

• External memory: Dataset > RAM
• Custom training loop: Using xgb.train() with callbacks
• Native API: Direct parameter control, multiple eval sets
• Binary serialization: Large datasets reused across experiments

For standard scikit-learn pipelines, XGBRegressor/XGBClassifier are sufficient — they handle DMatrix conversion internally.