A Comprehensive Guide to XGBoost Loss Functions

Q: When should I use Tweedie regression instead of Gamma regression?

Use Tweedie when your target has exact zeros mixed with continuous positive values (like insurance claims where most policies have zero claims). Use Gamma when all targets are strictly positive (like claim severity given that a claim occurred). The Tweedie variance power parameter $p$ controls the distribution: $p=1.5$ gives compound Poisson-Gamma, $p=2$ gives Gamma, $p=3$ gives inverse Gaussian.

Q: Why does binary:logistic outperform binary:hinge in most practical cases?

Binary:logistic provides well-calibrated probabilities and smooth gradients everywhere, enabling more efficient tree optimization. Hinge loss has zero Hessians in the flat region, which loses the benefits of second-order optimization. The probabilistic output of logistic loss is also more useful for threshold tuning and decision-making.

Q: How do I choose between rank:pairwise and rank:ndcg?

rank:pairwise is simpler and faster but treats all misordered pairs equally. rank:ndcg uses LambdaRank to focus on pairs where correct ordering matters most for the NDCG metric, typically yielding better ranking quality at the cost of more computation. If your evaluation metric is NDCG, use rank:ndcg. If you need faster training and the ranking metric isn't specifically NDCG, rank:pairwise may suffice.

Q: Can I combine multiple loss functions?

Not directly as a single objective, but you can write a custom objective that implements a weighted sum of losses. Ensure the gradient and Hessian reflect the combined loss, and be mindful that mixing objectives with different scales may require careful weighting.

The loss function is the mathematical engine that tells XGBoost what “wrong” means for your specific problem. It defines the objective minimized during training and directly shapes how trees are built. XGBoost uses second-order Taylor approximation to optimize these losses efficiently—each loss function provides both a gradient (first derivative) and a Hessian (second derivative) that guide split finding and leaf weight calculation. Choosing the right loss function aligns your model’s optimization target with the actual cost of errors in your domain.

This guide covers every major built-in loss function in XGBoost: their mathematical forms, gradients, Hessians, appropriate use cases, outlier sensitivity, and parameter configuration. We also cover how to write custom objectives when the built-in options don’t fit.

Regression Loss Functions

Regression losses define how XGBoost penalizes the difference between predicted values $\hat{y}$ and true continuous targets $y$. The choice among them reflects assumptions about the error distribution and the cost of large versus small mistakes.

reg:squarederror (MSE)

This is the default and most commonly used regression objective. It minimizes mean squared error under the assumption of Gaussian-distributed errors with constant variance.

Mathematical form: $$\ell(y, \hat{y}) = \frac{1}{2}(y - \hat{y})^2$$

Gradient and Hessian: $$g = \frac{\partial \ell}{\partial \hat{y}} = \hat{y} - y$$ $$h = \frac{\partial^2 \ell}{\partial \hat{y}^2} = 1$$

The constant Hessian makes this loss particularly efficient—every data point contributes equal weight to the optimization regardless of its residual magnitude. The gradient is linear in the error, so residuals scale proportionally with their distance from the prediction.

Use cases: Standard regression tasks with roughly symmetric, well-behaved errors. Financial forecasting with clean data, predicting physical measurements, any situation where large errors are genuinely and quadratically worse than small ones.

Outlier sensitivity: High. A single residual of 10 contributes 100× more to the objective than a residual of 1. Outliers can dominate gradient updates and pull the model toward extreme predictions. Not appropriate for data with fat-tailed error distributions.

Configuration:

params = {
    'objective': 'reg:squarederror',
    'eval_metric': 'rmse'
}

reg:squaredlogerror

This objective operates in log space, penalizing relative rather than absolute errors. It’s designed for targets that span orders of magnitude and where a $10 error on a value of 10 is far worse than a $10 error on a value of 10,000.

Mathematical form: $$\ell(y, \hat{y}) = \frac{1}{2}\left[\log(y + 1) - \log(\hat{y} + 1)\right]^2$$

The “+1” shift ensures the loss is well-defined when targets approach zero. The loss is equivalent to minimizing RMSLE (Root Mean Squared Logarithmic Error).

Gradient and Hessian: $$g = \frac{\log(\hat{y} + 1) - \log(y + 1)}{\hat{y} + 1}$$ $$h = \frac{1 - \log(\hat{y} + 1) + \log(y + 1)}{(\hat{y} + 1)^2}$$

Use cases: Targets with exponential growth patterns (sales forecasting across multiple product tiers), house prices spanning several orders of magnitude, any domain where relative error matters more than absolute error. The objective naturally handles heteroscedasticity since the log transform stabilizes variance.

Outlier sensitivity: Moderate to low for large values, higher for values near zero. The log transform compresses large targets, so extreme positive values have dampened influence. However, predictions near -1 (where the denominator approaches zero) can cause numerical instability—ensure targets are positive.

Configuration:

params = {
    'objective': 'reg:squaredlogerror',
    'eval_metric': 'rmsle'
}

reg:pseudohubererror

The pseudo-Huber loss provides smooth, differentiable robustness to outliers. It behaves like MSE for small residuals but transitions to linear (MAE-like) behavior for large residuals, controlled by a slope parameter $\delta$.

Mathematical form: $$\ell(y, \hat{y}) = \delta^2 \left(\sqrt{1 + \left(\frac{y - \hat{y}}{\delta}\right)^2} - 1\right)$$

When $|y - \hat{y}| \ll \delta$, this approximates $\frac{1}{2}(y - \hat{y})^2$. When $|y - \hat{y}| \gg \delta$, it approximates $\delta|y - \hat{y}|$.

Gradient and Hessian: $$g = \frac{\hat{y} - y}{\sqrt{1 + \left(\frac{y - \hat{y}}{\delta}\right)^2}}$$ $$h = \frac{1}{\left(1 + \left(\frac{y - \hat{y}}{\delta}\right)^2\right)^{3/2}}$$

Use cases: Regression with known outliers that shouldn’t be removed (sensor errors, data entry mistakes), financial data with occasional shocks, any situation where you want robust optimization without the non-differentiability of absolute loss. The $\delta$ parameter controls the threshold where robustness kicks in—smaller values make the loss more robust but less efficient for inliers.

Outlier sensitivity: Low. Residuals beyond $\delta$ contribute only linearly to the gradient, preventing outlier domination. The smooth transition avoids optimization issues that plague pure Huber loss implementations.

Configuration:

params = {
    'objective': 'reg:pseudohubererror',
    'eval_metric': 'mae',
    'huber_slope': 1.0  # delta parameter
}

reg:gamma and reg:tweedie

These objectives derive from generalized linear models and handle non-negative, right-skewed targets common in insurance, healthcare, and reliability engineering.

reg:gamma assumes targets follow a Gamma distribution with variance proportional to the square of the mean. The loss uses a log link function by default.

Mathematical form (with log link, $\mu = e^{\hat{y}}$): $$\ell(y, \hat{y}) = \frac{y}{\mu} + \log(\mu) = y e^{-\hat{y}} + \hat{y}$$

Gradient and Hessian: $$g = 1 - y e^{-\hat{y}}$$ $$h = y e^{-\hat{y}}$$

Use cases: Positive continuous targets with constant coefficient of variation. Insurance claim severity (given a claim occurred), survival times, rainfall amounts.

reg:tweedie generalizes to a family of distributions indexed by a variance power parameter $p$ ($1 < p < 2$). At $p=1.5$, it corresponds to a compound Poisson-Gamma distribution that naturally models data with a point mass at zero and continuous positive values—exactly matching insurance claims where most policies have zero claims but some have positive claim amounts.

The Tweedie log-likelihood with log link: $$\ell(y, \hat{y}) = -y e^{(1-p)\hat{y}}/(1-p) + e^{(2-p)\hat{y}}/(2-p)$$

Use cases: Insurance claims frequency-severity modeling, zero-inflated continuous data, any domain with exact zeros mixed with positive continuous values.

Configuration:

# Gamma regression
params = {
    'objective': 'reg:gamma',
    'eval_metric': 'gamma-nloglik'
}

# Tweedie regression (compound Poisson-Gamma)
params = {
    'objective': 'reg:tweedie',
    'eval_metric': 'tweedie-nloglik@1.5',
    'tweedie_variance_power': 1.5
}

Outlier sensitivity: Both objectives handle right-skewed data naturally through their distributional assumptions. The variance structure (variance proportional to mean-squared for Gamma, mean-to-the-p for Tweedie) means large predictions naturally have larger expected variance, reducing the disproportionate influence of large values.

Binary Classification Loss Functions

binary:logistic

The standard loss for binary classification, minimizing logistic loss (cross-entropy) and outputting probability scores after sigmoid transformation.

Mathematical form: $$\ell(y, \hat{y}) = y \log(1 + e^{-\hat{y}}) + (1 - y) \log(1 + e^{\hat{y}})$$

Where $y \in {0, 1}$ and $\hat{y}$ is the raw margin before sigmoid transformation. The predicted probability is $\sigma(\hat{y}) = 1/(1 + e^{-\hat{y}})$.

Gradient and Hessian: $$g = \sigma(\hat{y}) - y$$ $$h = \sigma(\hat{y})(1 - \sigma(\hat{y}))$$

Use cases: Standard binary classification (churn prediction, fraud detection, medical diagnosis). The probabilistic output is well-calibrated when the model is properly regularized, making it suitable when decision thresholds need to be tuned post-training.

Outlier sensitivity: Moderate. The sigmoid function naturally bounds predictions to (0,1), and the Hessian approaches zero when predictions are confident, automatically down-weighting well-classified examples. However, mislabeled examples far from the decision boundary can still exert influence.

Configuration:

params = {
    'objective': 'binary:logistic',
    'eval_metric': 'logloss'
}

binary:hinge

Implements SVM-style hinge loss, outputting class predictions rather than probabilities. The loss penalizes only when the margin is less than 1, producing sparse updates.

Mathematical form: $$\ell(y, \hat{y}) = \max(0, 1 - y^* \hat{y})$$

Where $y^* \in {-1, 1}$ (internally converted from ${0, 1}$ labels).

Gradient and Hessian: $$g = \begin{cases} -y^* & \text{if } y^* \hat{y} < 1 \ 0 & \text{otherwise} \end{cases}$$ $$h = 0$$

Use cases: Maximum-margin classification, problems where only class assignments matter rather than calibrated probabilities. The zero Hessian means XGBoost uses a constant second-order approximation for optimization. Less common in modern XGBoost applications but useful when SVM-like behavior is desired in a tree ensemble framework.

Configuration:

params = {
    'objective': 'binary:hinge',
    'eval_metric': 'error'
}

binary:logitraw

Identical to binary:logistic in terms of the loss function, but outputs the raw margin $\hat{y}$ rather than the sigmoid-transformed probability. This is useful when you need access to the raw scores for custom thresholding or when combining with external calibration procedures.

Configuration:

params = {
    'objective': 'binary:logitraw',
    'eval_metric': 'logloss'
}

The mathematical form, gradient, and Hessian are identical to binary:logistic—only the prediction output format differs.

Multi-class Classification Loss Functions

XGBoost uses the softmax function for multi-class problems, training one tree per class per boosting round (or one vector-output tree, depending on configuration).

multi:softmax and multi:softprob

Both use the same underlying loss (categorical cross-entropy with softmax activation) but differ in output format. multi:softmax outputs the predicted class index, while multi:softprob outputs a probability vector over all classes.

Mathematical form (for $K$ classes): $$\ell(\mathbf{y}, \hat{\mathbf{y}}) = -\sum_{k=1}^K y_k \log\left(\frac{e^{\hat{y}k}}{\sum{j=1}^K e^{\hat{y}_j}}\right)$$

Where $\mathbf{y}$ is a one-hot encoded vector and $\hat{\mathbf{y}}$ contains raw margins for each class.

Gradient and Hessian (per class $k$): $$g_k = \frac{e^{\hat{y}k}}{\sum{j=1}^K e^{\hat{y}_j}} - y_k = p_k - y_k$$ $$h_k = p_k(1 - p_k)$$

Where $p_k$ is the softmax probability for class $k$. The Hessian matrix has off-diagonal terms $h_{kj} = -p_k p_j$, but XGBoost approximates with only the diagonal for efficiency.

Use cases: Multi-class classification with mutually exclusive classes (image categorization, document topic classification, sensor type identification). Use multi:softprob when you need per-class probabilities (for threshold tuning, ensemble stacking, or uncertainty estimation). Use multi:softmax when only the final class assignment matters.

Configuration:

params = {
    'objective': 'multi:softprob',
    'num_class': 10,
    'eval_metric': 'mlogloss'
}

# For class output only:
params = {
    'objective': 'multi:softmax',
    'num_class': 10,
    'eval_metric': 'merror'
}

Ranking Loss Functions

Ranking objectives optimize the relative ordering of items rather than their absolute scores. They’re designed for learning-to-rank tasks where the goal is to produce a relevance-ordered list.

rank:pairwise

Implements pairwise ranking loss inspired by RankNet (Burges et al., 2005). For each pair of documents with different relevance labels, the loss penalizes ordering inversions.

Mathematical form: $$\ell(y_i, y_j, \hat{y}_i, \hat{y}_j) = \log(1 + e^{-\sigma(\hat{y}_i - \hat{y}_j)})$$

Applied only when $y_i > y_j$ (document $i$ is more relevant than document $j$). The loss encourages the model score for the more relevant document to exceed the score for the less relevant one.

Configuration:

params = {
    'objective': 'rank:pairwise',
    'eval_metric': 'ndcg@10'
}

rank:ndcg

Optimizes the Normalized Discounted Cumulative Gain directly using the LambdaRank framework (Burges, 2010). Rather than treating all pairs equally, LambdaRank weights the gradient from each pair by the change in NDCG that would result from swapping their positions. This focuses optimization on pairs where correct ordering matters most for the ranking metric.

The gradient magnitude for a pair is scaled by $|\Delta\text{NDCG}_{ij}|$, the absolute change in NDCG if documents $i$ and $j$ were swapped.

Configuration:

params = {
    'objective': 'rank:ndcg',
    'eval_metric': 'ndcg@20',
    'lambdarank_num_pair_per_sample': 10,
    'lambdarank_pair_method': 'topk'
}

rank:map

Optimizes Mean Average Precision using a similar LambdaRank approach but with pair weights derived from MAP changes rather than NDCG changes. Useful when the evaluation metric is MAP and you want the optimization objective to align directly.

Configuration:

params = {
    'objective': 'rank:map',
    'eval_metric': 'map@100'
}

Important: All ranking objectives require the data to be grouped by query using set_group() on the DMatrix before training. XGBoost uses these group boundaries to know which documents belong to the same query and should be compared only within queries.

import xgboost as xgb

dtrain = xgb.DMatrix(X_train, label=y_train)
dtrain.set_group(query_groups)  # array of query sizes

The XGBoost documentation provides additional details on ranking parameter tuning at the Learning to Rank tutorial.

Writing Custom Loss Functions

When built-in objectives don’t match your problem, XGBoost allows you to define custom objectives by providing a function that returns gradient and Hessian arrays. This is one of XGBoost’s most powerful features.

Function Signature

A custom objective function must accept two arguments—predicted values and a DMatrix object containing the training data—and return a tuple of (gradient, hessian):

def custom_loss(y_pred, dtrain):
    """
    Custom objective function for XGBoost.

    Parameters
    ----------
    y_pred : numpy.ndarray
        Current predictions (raw margins before any transformation)
    dtrain : xgboost.DMatrix
        Training data with labels accessible via dtrain.get_label()

    Returns
    -------
    grad : numpy.ndarray
        First derivative of loss with respect to y_pred
    hess : numpy.ndarray
        Second derivative of loss with respect to y_pred
    """
    y_true = dtrain.get_label()
    # Compute gradient and hessian
    grad = ...  # shape must match y_pred
    hess = ...  # shape must match y_pred
    return grad, hess

Example: Quantile Regression

Quantile regression minimizes pinball loss to predict conditional quantiles rather than the conditional mean. For a target quantile $\alpha \in (0, 1)$:

$$\ell(y, \hat{y}) = \begin{cases} \alpha(y - \hat{y}) & \text{if } y \geq \hat{y} \ (\alpha - 1)(y - \hat{y}) & \text{if } y < \hat{y} \end{cases}$$

Gradient and Hessian: $$g = \begin{cases} -\alpha & \text{if } y \geq \hat{y} \ 1 - \alpha & \text{if } y < \hat{y} \end{cases}$$ $$h = 0$$

The Hessian is zero, but XGBoost handles this by using a small positive constant for numerical stability. In practice, we provide a small constant Hessian:

import numpy as np
import xgboost as xgb

def quantile_loss(alpha):
    def objective(y_pred, dtrain):
        y_true = dtrain.get_label()
        residuals = y_true - y_pred
        grad = np.where(residuals >= 0, -alpha, 1 - alpha)
        hess = np.ones_like(y_pred) * 1e-6  # small constant for stability
        return grad, hess
    return objective

# Predict the 90th percentile
model = xgb.train(
    {'objective': quantile_loss(0.9)},
    dtrain,
    num_boost_round=100
)

Example: Focal Loss for Imbalanced Classification

Focal loss (Lin et al., 2017) down-weights well-classified examples to focus learning on hard cases. For binary classification with sigmoid output:

def focal_loss(gamma=2.0, alpha=0.25):
    def objective(y_pred, dtrain):
        y_true = dtrain.get_label()
        p = 1.0 / (1.0 + np.exp(-y_pred))  # sigmoid
        # Clip probabilities for numerical stability
        p = np.clip(p, 1e-8, 1 - 1e-8)

        # Focal loss gradient
        alpha_t = alpha * y_true + (1 - alpha) * (1 - y_true)
        pt = p * y_true + (1 - p) * (1 - y_true)
        grad = alpha_t * (pt ** gamma) * (p - y_true)

        # Approximate Hessian
        hess = alpha_t * (pt ** gamma) * p * (1 - p)
        hess = np.maximum(hess, 1e-8)

        return grad, hess
    return objective

params = {
    'objective': focal_loss(gamma=2.0, alpha=0.25),
    'eval_metric': 'logloss'
}

For more details on custom objectives and the mathematical framework behind XGBoost’s second-order optimization, consult the official XGBoost tutorial on custom objectives and the foundational paper XGBoost: A Scalable Tree Boosting System by Chen and Guestrin (2016).

Frequently Asked Questions

When should I use Tweedie regression instead of Gamma regression?

Use Tweedie when your target has exact zeros mixed with continuous positive values (like insurance claims where most policies have zero claims). Use Gamma when all targets are strictly positive (like claim severity given that a claim occurred). The Tweedie variance power parameter $p$ controls the distribution: $p=1.5$ gives compound Poisson-Gamma, $p=2$ gives Gamma, $p=3$ gives inverse Gaussian.

Why does binary:logistic outperform binary:hinge in most practical cases?

Binary:logistic provides well-calibrated probabilities and smooth gradients everywhere, enabling more efficient tree optimization. Hinge loss has zero Hessians in the flat region, which loses the benefits of second-order optimization. The probabilistic output of logistic loss is also more useful for threshold tuning and decision-making.

How do I choose between rank:pairwise and rank:ndcg?

rank:pairwise is simpler and faster but treats all misordered pairs equally. rank:ndcg uses LambdaRank to focus on pairs where correct ordering matters most for the NDCG metric, typically yielding better ranking quality at the cost of more computation. If your evaluation metric is NDCG, use rank:ndcg. If you need faster training and the ranking metric isn’t specifically NDCG, rank:pairwise may suffice.

Can I combine multiple loss functions?

Not directly as a single objective, but you can write a custom objective that implements a weighted sum of losses. Ensure the gradient and Hessian reflect the combined loss, and be mindful that mixing objectives with different scales may require careful weighting.