Custom Objective Functions in XGBoost

Q: Do custom objectives support GPU training?

Yes, but only when written in a way the GPU backend can compile. Pure NumPy-based objectives run on CPU even when the rest of training is on GPU. For full GPU acceleration with custom objectives, use the native CUDA interface or the upcoming `DeviceQuantileDMatrix` support.

Q: Can I use custom objectives with Dask for distributed training?

Custom Python objectives work with Dask, but the function must be serializable (no closures over non-picklable state). Define it at module level and avoid capturing external mutable state.

Q: Why does my custom focal loss converge slower than binary:logistic?

Focal loss down-weights well-classified examples, which can reduce the effective learning rate on easy samples. You may need to increase `learning_rate` or adjust `gamma` and `alpha` based on your class distribution.

Q: What about multi-class objectives?

Multi-class custom objectives receive `y_pred` as a 2D array of shape `(n_samples, n_classes)` in row-major format. Your gradient and Hessian must return arrays of the same shape. Computing softmax gradients correctly requires summing contributions across classes.

Custom objective functions let you optimize XGBoost for loss functions that aren’t included in the library’s built-in set. While standard objectives like reg:squarederror and binary:logistic handle most applications, domain-specific problems often demand tailored loss formulations. XGBoost’s gradient boosting framework requires only that you supply the first and second derivatives of your loss with respect to the raw model predictions — the rest of the tree-building machinery handles itself.

Why Write a Custom Objective?

Built-in objectives cover regression, classification, ranking, and survival analysis. But real-world modeling frequently pushes beyond these boundaries:

Quantile regression estimates conditional percentiles rather than conditional means. This matters when you need prediction intervals, when errors are asymmetric, or when over-prediction and under-prediction carry different costs.

Focal loss addresses severe class imbalance by down-weighting the contribution of easily classified examples. Standard cross-entropy treats all misclassifications equally; focal loss forces the model to concentrate on hard cases.

Survival analysis with censored data requires likelihood formulations (like Cox proportional hazards or accelerated failure time models) that don’t map neatly onto built-in objectives.

Fairness-aware objectives incorporate demographic parity or equalized odds constraints directly into the optimization criterion, penalizing predictions that violate fairness criteria.

Business-specific cost functions might assign asymmetric penalties to errors (a false negative costs 10x more than a false positive) or weight observations by customer lifetime value.

The Interface

A custom objective in XGBoost is a Python function with a fixed signature:

def custom_obj(y_true, y_pred):
    grad = ...  # first derivative of loss w.r.t. y_pred
    hess = ...  # second derivative of loss w.r.t. y_pred
    return grad, hess

Two critical details: y_pred is the raw margin (before any link function is applied), and y_true is the original label array. For classification, the margin is the log-odds, not the probability. Your gradient and Hessian must be computed with respect to this raw margin, which means you’ll typically apply the link function inside the objective.

The Hessian must be strictly positive for the algorithm to work — XGBoost uses it as a weight in the leaf weight calculation $\left(w = -\frac{\sum g_i}{\sum h_i + \lambda}\right)$, and a zero or negative Hessian breaks this computation.

Example 1: Quantile Loss

The pinball loss for quantile $q \in (0,1)$ is:

$$L(y, \hat{y}) = \sum_i \max(q \cdot (y_i - \hat{y}_i), (q-1) \cdot (y_i - \hat{y}_i))$$

The gradient is piecewise constant:

When $y_i > \hat{y}_i$: $\text{grad} = q - 1$
When $y_i < \hat{y}_i$: $\text{grad} = q$
When $y_i = \hat{y}_i$: subgradient in $[q-1, q]$; conventionally set to 0

The Hessian is zero almost everywhere. For numerical stability, supply a small positive constant:

import numpy as np

def quantile_obj(y_true, y_pred, quantile=0.5, eps=1e-6):
    residual = y_true - y_pred
    grad = np.where(residual > 0, quantile - 1, quantile)
    grad = np.where(np.abs(residual) < eps, 0.0, grad)
    hess = np.where(np.abs(residual) < eps, eps, eps)
    return grad, hess

Usage with the scikit-learn API:

import xgboost as xgb

model = xgb.XGBRegressor(objective=quantile_obj, n_estimators=100)
model.fit(X_train, y_train)
predictions = model.predict(X_test)

For $q=0.5$, this recovers median regression. For $q=0.1$ or $q=0.9$, you get lower and upper decile estimates, which pair naturally into 80% prediction intervals.

Example 2: Focal Loss for Classification

Focal loss was introduced by Lin et al. to address class imbalance:

$$L_{\text{focal}} = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$

where $p_t = p$ if $y=1$ else $1-p$, and $\alpha_t$ is a class-weighting factor. For XGBoost, we work with the raw margin $z$ (log-odds), so $p = 1 / (1 + e^{-z})$.

Let $y \in {0, 1}$ be the label and define $p = \sigma(z)$. After derivation through the chain rule, the gradient and Hessian with respect to $z$ are:

For $y=1$: $$\text{grad} = -\alpha (1-p)^\gamma \left[\gamma p \log(p) + p - 1\right]$$ $$\text{hess} = \alpha (1-p)^\gamma p \left[\gamma(\gamma p \log(p) + p - 1) + (1-p)\right] \cdot (1-p)$$

For $y=0$ (substituting $p \to 1-p$ and flipping signs appropriately):

def focal_loss_obj(y_true, y_pred, alpha=0.25, gamma=2.0):
    p = 1.0 / (1.0 + np.exp(-y_pred))
    p = np.clip(p, 1e-8, 1 - 1e-8)
    
    pt = np.where(y_true == 1, p, 1 - p)
    alpha_t = np.where(y_true == 1, alpha, 1 - alpha)
    
    # Gradient w.r.t. raw margin
    grad = alpha_t * (1 - pt)**gamma * (
        pt - 1 + gamma * pt * np.log(pt) + gamma * (1 - pt)
    )
    grad = np.where(y_true == 1, grad, -grad)
    
    # Hessian: second derivative
    hess_term1 = gamma * pt * np.log(pt) + pt - 1
    hess = alpha_t * (1 - pt)**gamma * pt * (1 - pt) * (
        gamma * hess_term1 + 1
    )
    hess = np.clip(hess, 1e-8, None)
    
    return grad, hess

The gradient derivation follows from applying the chain rule to the focal loss expression. The clipping on p and hess prevents numerical instability from extreme probability values.

Example 3: Weighted MSE with Huber-style Robustness

When outliers contaminate your training data, you can down-weight samples with large residuals:

def weighted_mse_obj(y_true, y_pred, delta=1.0):
    residual = y_true - y_pred
    abs_res = np.abs(residual)
    
    # Huber-style weight: full weight for small residuals, 
    # inverse weight for large ones
    weight = np.where(abs_res <= delta, 1.0, delta / abs_res)
    
    grad = -2.0 * weight * residual
    hess = 2.0 * weight
    hess = np.clip(hess, 1e-8, None)
    
    return grad, hess

This smoothly transitions from quadratic to linear behavior for large residuals, reducing the influence of outliers without discarding them entirely.

Custom Evaluation Metrics

Custom objectives determine what the model optimizes. Custom evaluation metrics monitor performance during training and drive early stopping. The interface differs: a metric function receives y_pred (still margins for classification) and y_true, and returns a tuple of (name, value):

def pinball_loss_metric(y_pred, y_true, quantile=0.5):
    residual = y_true - y_pred
    loss = np.mean(np.maximum(quantile * residual, (quantile - 1) * residual))
    return "pinball_loss", loss

model = xgb.XGBRegressor(
    objective=quantile_obj,
    eval_metric=pinball_loss_metric,
    early_stopping_rounds=10
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)])

A metric does not need to match the objective. You might optimize quantile loss but monitor mean absolute error, or train with focal loss but track AUC.

Common Pitfalls and Guardrails

Zero or negative Hessians cause division-by-zero in the leaf weight formula. Always clip the Hessian to a small positive value: hess = np.maximum(hess, 1e-8). This is essential for quantile loss, whose true Hessian is identically zero.

Base score mismatch: XGBoost initializes predictions to base_score (default 0.5 for classification, 0.5 for regression with reg:squarederror). For custom objectives, verify that this starting point is reasonable. Focal loss with default base_score=0.5 starts at probability $\sigma(0.5) \approx 0.62$, which may be far from your class balance.

Output format consistency: For multi:softmax or multi:softprob, predictions are 2D arrays of shape (n_samples, n_classes). Custom multi-class objectives must handle this dimensionality correctly in both gradient and Hessian computations.

Numerical stability: Exponentiations in focal loss and sigmoid computations can overflow. Clip probabilities to [1e-8, 1-1e-8] and clip gradient/Hessian magnitudes to avoid exploding values.

Testing against known equivalents: Always verify your custom objective against a built-in one where they should match. A custom MSE objective should produce identical results to reg:squarederror on the same data and hyperparameters:

def custom_mse(y_true, y_pred):
    grad = 2 * (y_pred - y_true)
    hess = 2 * np.ones_like(y_pred)
    return grad, hess

# These should produce near-identical predictions
model_builtin = xgb.XGBRegressor(objective='reg:squarederror', seed=42)
model_custom  = xgb.XGBRegressor(objective=custom_mse, seed=42)

FAQ

Do custom objectives support GPU training? Yes, but only when written in a way the GPU backend can compile. Pure NumPy-based objectives run on CPU even when the rest of training is on GPU. For full GPU acceleration with custom objectives, use the native CUDA interface or the upcoming DeviceQuantileDMatrix support.

Can I use custom objectives with Dask for distributed training? Custom Python objectives work with Dask, but the function must be serializable (no closures over non-picklable state). Define it at module level and avoid capturing external mutable state.

Why does my custom focal loss converge slower than binary:logistic? Focal loss down-weights well-classified examples, which can reduce the effective learning rate on easy samples. You may need to increase learning_rate or adjust gamma and alpha based on your class distribution.

What about multi-class objectives? Multi-class custom objectives receive y_pred as a 2D array of shape (n_samples, n_classes) in row-major format. Your gradient and Hessian must return arrays of the same shape. Computing softmax gradients correctly requires summing contributions across classes.

Custom Objective Functions and Evaluation Metrics in XGBoost