XGBoost for Classification · Binary, Multi-Class, and Ranking

XGBoost supports three classification modes: binary logistic, multi-class softmax, and multi-class softprob. Each requires a different objective function and output interpretation.

Binary Classification

model = xgb.XGBClassifier(
    objective='binary:logistic',
    eval_metric='logloss',
    max_depth=6, learning_rate=0.1, n_estimators=100
)
model.fit(X_train, y_train)
probs = model.predict_proba(X_test)[:, 1]

Objective binary:logistic: Outputs probability via logistic transform. Regression trees predict log-odds; final output passes through sigmoid.

Evaluation: logloss (logarithmic loss) is preferred. error (misclassification rate) is insensitive to probability quality. auc (area under ROC) is scale-invariant.

Class Imbalance

Scale Pos Weight

n_neg, n_pos = 10000, 100  # 100:1 imbalance
model = xgb.XGBClassifier(scale_pos_weight=n_neg/n_pos)  # = 100

Scales the gradient of positive instances. Over-weighs minority class in the loss function. Equivalent to SMOTE-like effect without generating synthetic samples.

Focal Loss (XGBoost ≥ 1.6)

# Custom objective for focal loss
def focal_loss(y_true, y_pred, gamma=2.0, alpha=0.25):
    p = 1 / (1 + np.exp(-y_pred))
    pt = y_true * p + (1 - y_true) * (1 - p)
    grad = alpha * (1 - pt)**gamma * (p - y_true)
    hess = alpha * (1 - pt)**gamma * pt * (1 - pt)
    return grad, hess

Focal loss down-weights easy examples and focuses on hard ones. Effective for extreme imbalance (1000:1+).

AUC vs Accuracy

With imbalance >10:1, accuracy is misleading — 99% accuracy by predicting all negatives. AUC-based evaluation is scale-invariant and preferred. Set eval_metric='auc'.

Multi-Class Classification

# softprob: probability for each class
model = xgb.XGBClassifier(objective='multi:softprob', num_class=10)
probs = model.predict_proba(X_test)  # shape: (n, 10)

# softmax: predicted class only
model = xgb.XGBClassifier(objective='multi:softmax', num_class=10)
preds = model.predict(X_test)  # shape: (n,)

Internally: XGBoost trains num_class regression trees per boosting round — one for each class. Softmax converts logits to probabilities.

Threshold Tuning

Default threshold is 0.5. For asymmetric costs, tune:

from sklearn.metrics import f1_score
best_thresh = 0.5
best_f1 = 0
for t in np.linspace(0.1, 0.9, 100):
    preds = (probs >= t).astype(int)
    f1 = f1_score(y_val, preds)
    if f1 > best_f1:
        best_f1, best_thresh = f1, t