GradientBoost Notes

XGBoost
A Reader's Encyclopedia

An independent knowledge resource for the XGBoost algorithm. We don't replace the official docs — we complement them with concept explanations, mathematical derivations, annotated papers, and unbiased benchmarks. Maintained by open-source contributors. Not affiliated with the XGBoost project.

XGBoost (eXtreme Gradient Boosting) is one of the most widely used machine learning algorithms in production today. It powers winning solutions on Kaggle, drives recommendation systems at scale, and handles tabular data with unmatched speed and accuracy. This encyclopedia covers everything from foundational concepts to advanced optimization techniques, with fully worked mathematical derivations and reproducible Python code examples. Whether you're preparing for a data science interview, optimizing a production pipeline, or simply curious about how gradient boosting works under the hood — you'll find clear, rigorous, and practical explanations here.

Concept

From decision trees to gradient boosting — build intuition step by step.

Math

Loss functions, Newton approximation, regularization. With full derivations.

Code

Practical guides with reproducible Python snippets. Everything tested.

Compare

XGBoost vs LightGBM vs CatBoost. Real benchmarks, not marketing.

Papers

Annotated readings of Chen & Guestrin 2016 and related work.

Latest entries

concept

What Is Gradient Boosting? The Core Idea Behind XGBoost

A conceptual introduction to gradient boosting: how additive models, weak learners, and gradient descent combine to create one of machine learning's most powerful algorithms.

 concept

Additive Training in XGBoost: How Boosting Rounds Build Trees Iteratively

Understanding the additive training process in XGBoost: how each boosting round fits a tree to pseudo-residuals, the role of shrinkage, and how to choose the right number of estimators.

 concept

How XGBoost Works: The Algorithm Step by Step

A walkthrough of the XGBoost algorithm from initialization to final prediction, covering additive training, second-order approximation, tree construction, and regularization.

 concept

XGBoost's Approximate Greedy Algorithm and Weighted Quantile Sketch

How XGBoost scales to large datasets using the approximate split-finding algorithm and weighted quantile sketch — the engineering behind fast training on millions of samples.

 concept

Handling Categorical Features in XGBoost: Encoding Strategies and Best Practices

A practical guide to encoding categorical variables for XGBoost — label encoding, one-hot encoding, target encoding, and the enable_categorical parameter — with performance comparisons.

 concept

Custom Objective Functions and Evaluation Metrics in XGBoost

How to write custom objective functions and evaluation metrics for XGBoost — from simple weighted regression to complex domain-specific loss functions with gradient and hessian implementations.