Adaptive Forests For Classification

D. Bertsimas and Y. Cui, Adaptive Forests For Classification, arXiv:2510.22991, 2025. (code)

Random Forests (RF) and Extreme Gradient Boosting (XGBoost) are two of the most widely used and highly performing classification and regression models. They aggregate equally weighted CART trees, generated randomly in RF or sequentially in XGBoost. In this paper, we propose Adaptive Forests (AF), a novel approach that adaptively selects the weights of the underlying CART models. AF combines (a) the Optimal PredictivePolicy Trees (OP2T) framework to prescribe tailored, input-dependent unequal weights to trees and (b) Mixed Integer Optimization (MIO) to refine weight candidates dynamically, enhancing overall performance. We demonstrate that AF consistently outperforms RF, XGBoost, and other weighted RF in binary and multi-class classification problems over 20+ real-world datasets.

It is amazing every time to see that Prof. Bertsimas's group continues to improve the state of the art.

在這篇 MIT 的論文，作者使用加權法和整數規劃，在20多個資料集上面，平均預測性能硬是比隨機森林和 XGBoost 好，的確很驚人。後面這兩篇論文，引用次數已經超過 23 萬。 

第一作者Bertsimas 已經60多歲了，依然創意無限，崇拜。 

上作業研究的時候，總是和同學們強調，工業系的這門課程真的很重要，要對自己的專業有信心。 MIT的商業分析 (business analytics)碩士，必修有最佳化、最佳化和機器學習；所以工業系是非常適合從事資料科學的研究。