XGBoost Custom Objective Functions · Beyond Built-in Losses

XGBoost’s built-in objective functions cover the most common cases (regression with squared error, classification with log loss, ranking with pairwise loss), but custom loss functions unlock the framework for domain-specific objectives. The key requirement: provide gradient and hessian (second derivative) of the loss.

The Interface

XGBoost needs two functions:

def custom_obj(y_true, y_pred):
    """Return gradient and hessian."""
    grad = ...  # First derivative of loss w.r.t. y_pred
    hess = ...  # Second derivative of loss w.r.t. y_pred
    return grad, hess

This is a functional requirement — any differentiable loss can be used, provided you can compute its first and second derivatives.

Example 1: Quantile Regression

Quantile loss (pinball loss) predicts the $\tau$-th quantile rather than the mean. Useful for prediction intervals and asymmetric cost functions.

$$ L(y, \hat{y}) = \sum_{i} [\tau \cdot \max(0, y_i - \hat{y}_i) + (1-\tau) \cdot \max(0, \hat{y}_i - y_i)] $$

def quantile_obj(y_true, y_pred, tau=0.5):
    residuals = y_true - y_pred
    grad = np.where(residuals > 0, -tau, 1 - tau)
    hess = np.ones_like(y_true)  # Constant hessian for quantile loss
    return grad, hess

model = xgb.XGBRegressor(objective=quantile_obj)

Interpretation: $\tau = 0.5$ → median regression. $\tau = 0.9$ → predicts the 90th percentile (overestimate is penalized 9× more than underestimate).

Example 2: Huber Loss

Huber loss combines squared error for small residuals with absolute error for large residuals — robust to outliers without sacrificing sensitivity to small errors.

def huber_obj(y_true, y_pred, delta=1.0):
    residuals = y_true - y_pred
    abs_res = np.abs(residuals)
    
    # Gradient
    grad = np.where(abs_res <= delta, -residuals, -delta * np.sign(residuals))
    # Hessian: 1 for squared region, 0 for linear region
    hess = np.where(abs_res <= delta, 1.0, 0.0)
    return grad, hess

Note: Zero hessian for the linear region — XGBoost handles this by adding a small constant ($\text{max}(\text{hess}, \epsilon)$).

Custom Evaluation Metric

Custom evaluation functions follow a similar pattern but return metric name and value:

def rmsle(y_true, y_pred):
    """Root mean squared logarithmic error."""
    y_pred = np.maximum(y_pred, 0)  # Avoid log(negative)
    err = np.sqrt(np.mean((np.log1p(y_true) - np.log1p(y_pred)) ** 2))
    return 'rmsle', err

model.fit(X_train, y_train, eval_set=[(X_val, y_val)], eval_metric=rmsle)

Important: Use numpy Not Other Libraries

XGBoost’s custom objective functions receive NumPy arrays — not DataFrames, tensors, or Python lists. Using non-NumPy operations can cause silent failures or performance degradation.

Weighted Objectives

To implement sample weighting in a custom objective:

def weighted_mse(y_true, y_pred):
    """MSE with per-sample weights passed through the model."""
    # Access weights from DMatrix if available
    grad = y_pred - y_true
    hess = np.ones_like(y_true)  # Will be multiplied by weight automatically
    return grad, hess

# Use sample_weight parameter
model.fit(X_train, y_train, sample_weight=sample_weights)

XGBoost automatically scales gradient and hessian by sample weights — you don’t need to handle this manually.