Feature Interaction Constraints in XGBoost

Q: Are interaction constraints supported for all tree methods?

Yes — `approx`, `hist`, and `exact` tree methods all support interaction constraints. GPU implementations (`gpu_hist`) also support them in recent versions, though you should verify compatibility with your specific XGBoost version.

Feature interaction constraints let you control which features can jointly influence predictions inside a single tree. When you specify interaction_constraints, you partition your features into disjoint groups. XGBoost then enforces that no root-to-leaf path contains splits from more than one group. The result: features in separate groups can only combine additively across the ensemble — they never interact nonlinearly within a tree. This is a structural constraint, not a statistical penalty, so the restriction is absolute for every tree in the booster.

# Each list is a group of features that can interact
constraints = [[0, 1], [2, 3, 4]]
# Feature 0 and 1 can appear together in a path
# Feature 2, 3, 4 can appear together in a path
# But no path mixes features from [0,1] and [2,3,4]
model = xgb.XGBRegressor(interaction_constraints=constraints)

How Interaction Constraints Work Internally

During tree construction, XGBoost evaluates candidate splits greedily. With interaction constraints active, the algorithm tracks which feature group (if any) has already been used on the current path from the root. When choosing the next split for a node, any feature belonging to a different group is excluded from consideration. Features that haven’t appeared yet in any group (unconstrained features) remain available everywhere. This partition logic is applied per path, not per tree — so a single tree can contain splits from multiple groups, just never interleaved along the same branch.

The constraint specification must form a partition: each feature index can appear in at most one group. If you omit features, they remain unconstrained and can interact freely with any other feature. If you want a fully additive model where no features interact at all, place every feature in its own singleton group.

Why Use Interaction Constraints

Interpretability Through Additivity

The most common motivation is interpretability. When each feature occupies its own group, the model becomes an additive function of individual features — a Generalized Additive Model (GAM) implemented via gradient boosting. You can then meaningfully inspect partial dependence plots and say “this feature contributes $f_j(x_j)$ to the prediction” without worrying about hidden interactions muddying the interpretation. This matters enormously in regulated domains like credit underwriting, insurance pricing, and medical risk scoring, where you may need to explain every factor’s marginal effect to auditors or regulators.

Preventing Undesirable Interactions

Sometimes you know certain features should not interact, either because of fairness constraints or domain theory. In credit scoring, you might want income and employment length to interact (someone with high income but very short employment may be riskier), but you want to prevent gender or ethnicity from interacting with financial variables — the model should treat demographic attributes as additive offsets only. Interaction constraints enforce this directly, unlike post-hoc fairness corrections that retrofit constraints onto an already-trained model.

Encoding Domain Knowledge

Domain experts often have strong priors about which variable relationships are physically or causally plausible. For example, in a model predicting building energy consumption, outdoor temperature and building insulation level might interact (poor insulation amplifies temperature effects), while occupancy patterns might interact with appliance usage but not with structural characteristics. Encoding this knowledge through constraints reduces the hypothesis space to plausible models, which can improve generalization when training data is limited.

Regularization by Reducing Capacity

Limiting interactions reduces the model’s representational capacity. A fully unconstrained tree can learn arbitrarily complex feature crosses; a constrained tree cannot. This acts as a form of structural regularization that’s complementary to L1/L2 penalties, subsampling, and early stopping. In practice, this often reduces overfitting on small-to-medium datasets where spurious interactions might otherwise be memorized.

Specifying Constraints

The interaction_constraints parameter accepts a list of lists, where each inner list contains zero-indexed feature indices:

# Additive model: every feature in its own group
additive_constraints = [[0], [1], [2], [3]]

# Two interaction groups, rest unconstrained
partial_constraints = [[0, 1, 2], [3, 4]]

# Single group — equivalent to no constraints
no_constraints = [[0, 1, 2, 3, 4]]

You can also pass the string "[[0, 1], [2, 3]]" directly as a hyperparameter. The native XGBoost API and the scikit-learn wrapper both support this parameter. Feature indices correspond to the column positions in your training data matrix, so if you’re using pandas DataFrames, you’ll need to map column names to integer positions.

Features not appearing in any constraint group are unconstrained: they can interact with any other feature, constrained or not. If this isn’t what you want, explicitly include every feature in some group.

Extreme Cases and Their Properties

Fully Additive Model (GAM): Place each feature in its own singleton group. The resulting model approximates:

$$f(\mathbf{x}) = \sum_{j=1}^{p} f_j(x_j)$$

where each $f_j$ is learned by the ensemble of trees, but no tree path ever combines multiple features. This is a powerful baseline for interpretability. You can examine the partial dependence of each feature in isolation and trust that those curves represent the model’s complete behavior. Prediction explanations become trivial: just look up each feature’s contribution.

Single Group (No Constraints): This is standard XGBoost. All features can interact arbitrarily within trees. Maximum flexibility, minimum interpretability.

Hybrid Models: The real power comes from mixing approaches. Group related features together for within-group interactions, but keep groups separate for additive across-group behavior. A credit model might group [income, debt_to_income, employment_length] in one basket, [credit_history_length, num_inquiries] in another, and [age] in its own group. The model can learn complex interactions among financial variables and among credit history variables separately, but treats the combination of financial and credit-history signals as additive.

Interaction with Other Parameters

Monotonic Constraints: You can combine interaction_constraints with monotonic_constraints to build models that are both structurally interpretable and directionally aligned with domain expectations. For instance, you might constrain income to have a monotonically decreasing effect on default probability while also preventing it from interacting with demographic features. The two constraint types operate independently — monotonicity constrains the sign of feature effects, while interaction constraints limit which features can combine.

max_depth: The max_depth parameter still applies globally. Even within an allowed interaction group, tree depth limits the order of interactions. A max_depth of 2 in a group of 5 features means at most pairwise interactions can be learned, regardless of group membership.

GPU vs CPU Backends: Implementation details differ between backends. As of recent XGBoost versions, both support interaction constraints, but the pruning logic and performance characteristics may vary. The official XGBoost documentation recommends testing both backends if performance is critical.

Practical Guidance

Start without constraints. Interaction constraints reduce model capacity. If you don’t need interpretability and have sufficient data, unconstrained XGBoost will typically perform better. Add constraints only when you have a specific reason: regulatory requirements, fairness concerns, domain knowledge, or a need for additive interpretability.

For regulatory applications, use an additive model as a baseline. Train a fully additive model alongside your more complex models. If the additive model performs nearly as well, prefer it — the interpretability dividend is substantial. If the unconstrained model substantially outperforms, you have a concrete measure of what interactions are buying you, which can inform discussions with compliance teams.

Validate constraints with partial dependence. After training a constrained model, use partial dependence plots (via sklearn.inspection.PartialDependenceDisplay) to verify that the model behaves as expected. For an additive model, the partial dependence of each feature should be independent of other features’ values. You can check this by computing partial dependence at different fixed values of other features — the curves should be identical up to an additive shift.

Consider feature importance patterns. In constrained models, feature importance metrics (gain, cover, weight) will reflect the restricted interaction structure. Features in larger groups may show higher importance simply because they can participate in more complex splits. This isn’t a bug — it reflects the model’s actual reliance — but it’s worth noting when comparing importance across groups.

Performance Considerations

Interaction constraints reduce the number of valid splits considered at each node. This can modestly speed up training, particularly with many features and deep trees, since fewer candidates need evaluation. The effect is typically small unless constraints are very restrictive.

More importantly, constraints often improve test-set performance on small datasets by preventing overfitting to spurious interactions. This is not guaranteed — if real interactions exist in the data-generating process, constraining them will hurt performance. The only reliable approach is cross-validation with and without constraints.

Code Example: Constrained vs Unconstrained

import xgboost as xgb
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score
from sklearn.metrics import make_scorer, mean_squared_error
import numpy as np

# Load data
data = fetch_california_housing()
X, y = data.data, data.target
feature_names = data.feature_names
n_features = X.shape[1]

# Fully additive model: each feature in its own group
additive_constraints = [[i] for i in range(n_features)]

# Unconstrained model
unconstrained = xgb.XGBRegressor(
    n_estimators=200, max_depth=5, learning_rate=0.05, random_state=42
)

# Additive model
additive = xgb.XGBRegressor(
    n_estimators=200, max_depth=5, learning_rate=0.05,
    interaction_constraints=additive_constraints, random_state=42
)

# Hybrid: group related features by domain knowledge
# MedInc, AveRooms, AveBedrms, AveOccup (housing characteristics)
# HouseAge, Population, Latitude, Longitude (location/temporal)
hybrid_constraints = [[0, 3, 4, 5], [1, 2, 6, 7]]
hybrid = xgb.XGBRegressor(
    n_estimators=200, max_depth=5, learning_rate=0.05,
    interaction_constraints=hybrid_constraints, random_state=42
)

# Evaluate
rmse_scorer = make_scorer(
    lambda y, y_pred: np.sqrt(mean_squared_error(y, y_pred)),
    greater_is_better=False
)

for name, model in [("Unconstrained", unconstrained),
                     ("Additive", additive),
                     ("Hybrid", hybrid)]:
    scores = cross_val_score(model, X, y, cv=5,
                            scoring=rmse_scorer, n_jobs=-1)
    print(f"{name}: RMSE = {-scores.mean():.4f} (+/- {scores.std():.4f})")

In practice on the California housing dataset, the unconstrained model typically achieves slightly lower RMSE than the additive model, with the hybrid model falling in between. The gap between additive and unconstrained performance tells you how much predictive power comes from feature interactions — a useful diagnostic in itself.

For further details, consult the XGBoost feature interaction constraint tutorial and the scikit-learn inspection module documentation for partial dependence validation.

FAQ

Can I change constraints per tree or per boosting round?

No. Constraints are specified once at training time and apply uniformly to all trees. If you need different constraints at different stages, you’d need to train separate models and ensemble them manually.

What happens if a feature appears in multiple constraint groups?

XGBoost will raise an error. Each feature can belong to at most one group. This ensures the partition is well-defined and the path-restriction logic is unambiguous.

Do interaction constraints affect feature importance calculations?

They affect what the model learns, which in turn affects measured importance. Features in larger groups may show higher gain importance because they can participate in more complex splits, but this reflects genuine model reliance under the constraints, not an artifact of the importance metric itself.

Are interaction constraints supported for all tree methods?

Yes — approx, hist, and exact tree methods all support interaction constraints. GPU implementations (gpu_hist) also support them in recent versions, though you should verify compatibility with your specific XGBoost version.

Can I use interaction constraints with categorical features?

Yes. Categorical splits are treated like any other split for constraint purposes. The feature index for a categorical column is subject to the same group membership rules as numeric features.