XGBoost Feature Importance · Gain, Weight, Cover, and SHAP

XGBoost provides five distinct measures of feature importance. They answer different questions, and naively using plot_importance() without understanding which metric is being shown can lead to misleading conclusions.

The Five Importance Metrics

1. Weight (default)

The number of times a feature is used to split a node across all trees. Simple, but biased toward continuous features (which can be split in more ways) and high-cardinality categoricals.

Answers: “How often was this feature used?”

Weakness: A feature used in many shallow splits may have high weight but low impact.

2. Gain

The average loss reduction achieved when using a feature for splitting. More informative than weight — a feature that produces a large improvement in a few splits may have low weight but high gain.

Answers: “How much did this feature improve the model?”

Weakness: Can overstate the importance of features used early in trees (early splits tend to have larger gains).

3. Cover

The average number of training instances affected by splits on this feature. Measures “reach” rather than impact. A feature that splits a node with 10,000 instances has high cover.

Answers: “How many observations were affected by this feature?”

Weakness: Early splits naturally have higher cover — this is a structural property, not necessarily an importance signal.

4. Total Gain / Total Cover

Sum over all splits (not averaged). When the number of splits varies widely across features, total gain/cover can be more informative than the average.

5. SHAP Values

SHAP (SHapley Additive exPlanations) provides per-prediction feature contributions grounded in cooperative game theory. Unlike the four built-in metrics (which are global, averaged over the dataset), SHAP decomposes every single prediction into feature contributions.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)
shap.summary_plot(shap_values, X)

SHAP values satisfy three desirable properties:

  • Local accuracy: The sum of SHAP values equals the difference between the prediction and the average prediction
  • Missingness: Features not used in a prediction receive zero SHAP value
  • Consistency: If a model changes so a feature becomes more important, its SHAP value magnitude increases

Winner: SHAP for interpretability and per-prediction explanations. Gain for a quick global view.

Common Pitfalls

xgboost-org 配图

  1. Using weight as the sole metric: Weight is the default in plot_importance() but is the least informative. Always specify importance_type='gain' or use SHAP.

  2. Comparing importance across different models: Importance is model-specific and scale-dependent. A feature with gain=100 in model A and gain=10 in model B may be more important in B if B’s total gain is much lower.

  3. Ignoring feature correlation: XGBoost importance measures can be unstable when features are correlated. If $X_1$ and $X_2$ are highly correlated, XGBoost may split on either, and their individual importances will be lower than the true importance of the underlying signal. Use permutation importance or SHAP for correlated features.

  4. Taking importance as causal: Feature importance tells you which features the model used — not which features cause the outcome. A feature may be important because it proxies for an unobserved confound, not because it has a causal effect.

Practical Recommendations

xgboost-org 配图

  • Quick inspection: xgb.plot_importance(model, importance_type='gain')
  • Correlated features: SHAP + dependence plots
  • Production monitoring: Track top-10 feature importance over time to detect data drift
  • Feature selection: Use SHAP values or permutation importance; remove features with near-zero importance