Feature Importance in XGBoost: Gain, Weight, Cover, and SHAP

Feature Importance Methods in XGBoost: A Complete Guide

XGBoost offers multiple distinct ways to measure feature importance, each capturing a different aspect of how features contribute to a model’s predictions. The five main types are weight (split frequency), gain (average loss improvement), cover (sample coverage), their cumulative counterparts (total gain/total cover), and SHAP (Shapley additive explanations). Gain serves as XGBoost’s default and is sufficient for most exploratory work, but SHAP provides the most theoretically rigorous and consistent importance measures when interpretability matters for decision-making. Understanding the mechanics and biases of each method is essential for drawing valid conclusions from importance rankings.

Built-in Importance Types

XGBoost computes importance scores directly from tree structures during or after training. These are accessible via the get_score() method or the plot_importance() function, with the importance_type parameter controlling which metric is returned.

Weight: Split Frequency

Weight counts the total number of times a feature is chosen for a split across all trees in the ensemble. Mathematically, for feature $j$ across $K$ trees:

$$\text{Weight}j = \sum{k=1}^{K} \sum_{n \in \text{nodes}_k} \mathbb{1}[\text{split_feature}(n) = j]$$

This metric is straightforward to compute and interpret, but it carries a well-known bias: features with many possible split points—particularly high-cardinality categorical variables or continuous features with many unique values—receive inflated weight scores simply because the tree-growing algorithm encounters more opportunities to select them. This bias persists even when those features contribute little to predictive performance.

Weight is most useful as a rough diagnostic for understanding which features the model considers frequently, but should not be used in isolation to assess feature importance for model interpretation. You can access it with:

xgb.plot_importance(model, importance_type='weight')

Gain: Average Loss Improvement

Gain measures the average improvement in the objective function (loss reduction) attributable to splits made on a feature. For each node where feature $j$ is used, the gain contribution is the reduction in loss before and after the split, weighted by the number of instances in that node. For regression with squared error, this is:

$$\text{Gain}j = \frac{1}{N_j} \sum{s \in \text{splits}j} \left( \text{loss}{\text{before}} - \text{loss}_{\text{after}} \right)$$

where $N_j$ is the number of splits using feature $j$.

Gain is the default importance type in plot_importance() and represents a substantial improvement over weight because it accounts for the quality of splits rather than just their quantity. A feature that produces large reductions in loss on a few splits will rank higher than one producing negligible improvements on many splits.

However, gain exhibits a structural bias toward features used early in trees. Early splits inherently have more samples to partition and therefore can produce larger absolute loss reductions. This does not necessarily mean early-split features are more important in a causal sense—it reflects the greedy nature of tree construction.

# Gain is the default
xgb.plot_importance(model, importance_type='gain')

Cover: Sample Coverage

Cover measures the number of observations that pass through nodes where a feature is used for splitting. In XGBoost’s implementation, this is computed using the sum of hessian values (second-order gradient statistics) for the instances in those nodes, rather than raw sample counts. For the common squared error loss, the hessian is constant (equal to 1), and cover reduces to the raw count of samples routed through splits on that feature.

$$\text{Cover}j = \frac{1}{N_j} \sum{s \in \text{splits}j} \sum{i \in \text{node}(s)} h_i$$

where $h_i$ is the hessian value for instance $i$ and $\text{node}(s)$ is the set of instances in the node where split $s$ occurs.

Cover captures a feature’s breadth of influence across the dataset rather than its predictive power. A feature with high average cover touches many training examples through its splits, making it structurally important to the model’s decision pathways, even if those splits produce modest loss reductions.

xgb.plot_importance(model, importance_type='cover')

Total Gain and Total Cover: Cumulative Measures

Total gain and total cover are the summed (rather than averaged) versions of their respective metrics:

$$\text{TotalGain}j = \sum{s \in \text{splits}j} \left( \text{loss}{\text{before}} - \text{loss}_{\text{after}} \right)$$

$$\text{TotalCover}j = \sum{s \in \text{splits}j} \sum{i \in \text{node}(s)} h_i$$

These variants favor features that appear many times in the ensemble, making them sensitive to model complexity (number of trees, depth). Total gain can be useful when you want to account for both per-split quality and frequency of use without normalizing, but comparisons across models with different hyperparameters become difficult. These are less commonly used in practice than their averaged counterparts but are available in XGBoost’s native API.

SHAP: Game-Theoretic Feature Attribution

SHAP (SHapley Additive exPlanations), introduced by Lundberg and Lee (2017), provides a unified framework for feature attribution based on Shapley values from cooperative game theory. For gradient-boosted trees, XGBoost integrates with the shap package through a specialized TreeExplainer that computes exact SHAP values efficiently.

SHAP values decompose each prediction into additive contributions from each feature:

$$\hat{y}i = \phi_0 + \sum{j=1}^{M} \phi_{ij}$$

where $\phi_0$ is the expected model output (base value) and $\phi_{ij}$ is the SHAP value for feature $j$ on instance $i$. The magnitude of $\phi_{ij}$ indicates how much feature $j$ pushed the prediction away from the base value for that specific instance.

Key advantages of SHAP over built-in importance types:

  • Consistency: If a feature’s contribution increases or stays the same (regardless of other features present), its SHAP importance never decreases—a property that gain, weight, and cover do not satisfy.
  • Feature interaction handling: SHAP values properly account for interactions, unlike permutation-based methods that can produce misleading results with correlated features.
  • Local and global interpretability: Individual SHAP values enable per-prediction explanations, while aggregating $|\phi_{ij}|$ across all instances yields global feature importance.
  • Directionality: SHAP preserves the sign of contributions, revealing whether a feature pushes predictions higher or lower.

Computational cost is the main trade-off. Exact SHAP computation for tree ensembles is $O(TLD^2)$ where $T$ is the number of trees, $L$ is the maximum number of leaves, and $D$ is the maximum tree depth—substantially more expensive than extracting built-in importance scores, which are computed from already-available tree statistics.

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)

# Global importance: mean absolute SHAP value per feature
shap.summary_plot(shap_values, X, plot_type="bar")

# Detailed summary with direction and feature value coloring
shap.summary_plot(shap_values, X)

# Dependence plot for a specific feature
shap.dependence_plot("feature_name", shap_values, X)

The XGBoost documentation on feature importance provides technical details on the built-in metrics, while the SHAP documentation covers TreeExplainer usage and interpretation.

Permutation Importance: Model-Agnostic Validation

Permutation importance operates on a fundamentally different principle: it measures the drop in a model’s performance metric when a feature’s values are randomly shuffled, breaking the association between that feature and the target. It can be computed on training data, validation data, or a held-out test set.

$$\text{PermImp}j = \text{score}{\text{original}} - \text{score}_{\text{shuffled}(j)}$$

Using a held-out set is strongly preferred, as training-set permutation importance can be misleading—a feature that the model overfits to will appear artificially important when evaluated on the training data.

Permutation importance has the advantage of being model-agnostic and conceptually simple. However, it suffers from a critical limitation with correlated features: if two features are highly correlated, shuffling one individually may produce only a small drop in performance because the model can fall back on the correlated partner. This can cause both features to appear less important than they truly are, an issue that does not affect SHAP values computed from the model structure.

from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_test, y_test,
    n_repeats=10,
    random_state=42,
    scoring='neg_mean_squared_error'
)

# result.importances_mean contains the importance scores

Comparison and Practical Guidance

For quick exploratory analysis, gain is the pragmatic default. It is available immediately after training with no additional computation and captures split quality, making it substantially more informative than weight. Most XGBoost visualizations default to gain for good reason.

For high-stakes decision-making or regulatory contexts, SHAP is the recommended approach. Its theoretical guarantees around consistency and its ability to handle feature interactions make it the most defensible choice when importance rankings drive business decisions. The additional computational cost is typically negligible compared to the cost of incorrect interpretation.

For model-agnostic validation, permutation importance on a held-out test set provides an independent check that does not rely on tree structure. If permutation importance and SHAP importance agree, confidence in the ranking increases. If they diverge significantly—particularly with correlated features—the SHAP ranking is generally more reliable.

Avoid weight alone for any serious interpretation. Its bias toward high-cardinality features is well-documented and can produce importance rankings that bear little relationship to actual predictive contribution.

Common Pitfalls

Correlated features split importance scores. When two features are strongly correlated, both built-in importance and permutation importance will divide the contribution between them. Neither receives full credit for the predictive signal they jointly provide. This is not a bug but a property of additive importance decompositions. SHAP values are affected similarly unless interaction values are explicitly computed.

One-hot encoding fragments importance. Converting a categorical variable with $k$ levels into $k-1$ binary features (or $k$ with some encodings) means the importance of the original categorical concept is distributed across multiple features. Summing importance across all one-hot columns for the same original feature provides a more accurate picture of that variable’s total contribution.

High-cardinality features inflate weight. As noted, weight-based importance systematically overstates the importance of features with many unique values. This is a structural artifact of decision tree construction and has no relationship to predictive utility.

Early-split bias in gain. Features used at the root or near the top of trees affect more samples and therefore accumulate larger gain values. This reflects the greedy tree-building process and means gain importance should be interpreted as “how useful was this feature in the constructed trees” rather than “how intrinsically important is this feature to the prediction problem.”

FAQ

Why do importance rankings change when I retrain with different random seeds? Tree-based models have inherent stochasticity from subsampling (both row and column) and the greedy nature of split selection. With correlated features, minor perturbations can cause the algorithm to select one feature over its correlated partner at similar split points, leading to rank swaps. Averaging importance across multiple training runs or using SHAP values (which are deterministic given a trained model) can help stabilize rankings.

Should I use training or test data for computing SHAP values? Use the data you want to explain. For global feature importance (understanding the model’s behavior on the data distribution it was trained on), training data is appropriate. For understanding model behavior on unseen data or detecting distribution shifts, use test data. SHAP values computed on training data reflect the model’s learned relationships; SHAP values on test data reflect how those relationships manifest on new observations.

How do I handle importance for multi-class classification? XGBoost trains one tree per class per boosting round for multi-class problems. Built-in importance scores are aggregated across all classes. For SHAP, you can request SHAP values for a specific class or examine class-specific importance patterns, which can reveal whether features contribute differently to different class predictions.

Can I compare importance scores across different XGBoost models? Generally no, unless the models share identical hyperparameters, training data, and the importance metric is appropriately normalized. Total gain and total cover scale with the number of trees, making cross-model comparisons meaningless. Even averaged metrics like gain and cover depend on tree depth, regularization, and subsampling rates. Importance scores are best used for ranking features within a single model.