Machine Learning Under a Modern Optimization Lens

Dimitris Bertsimas and Jack Dunn, Machine Learning Under a Modern Optimization Lens, Dynamic Ideas LLC, 2019.

The book provides an original treatment of machine learning (ML) using convex, robust and mixed integer optimization that leads to solutions to central ML problems at large scale that can be found in seconds/minutes, can be certified to be optimal in minutes/hours, and outperform classical heuristic approaches in out-of-sample experiments.

Structure of the book: 

Part I covers robust, sparse, nonlinear, holistic regression and extensions. 

Part II contains optimal classification and regression trees. 

Part III outlines prescriptive ML methods. 

Part IV shows the power of optimization over randomization in design of experiments, exceptional responders, stable regression and the bootstrap.

Part V describes unsupervised methods in ML: optimal missing data imputation and interpretable clustering. 

Part VI develops matrix ML methods: sparse PCA, sparse inverse covariance estimation, factor analysis, matrix and tensor completion. 

Part VII demonstrates how ML leads to interpretable optimization.  

Philosophical principles of the book: 

Interpretability is materially important in the real world. 

Practical tractability not polynomial solvability leads to real world impact. 

NP-hardness is an opportunity not an obstacle. 

ML is inherently linked to optimization not probability theory.  

Data represent an objective reality; models only exist in our imagination. 

Optimization has a significant edge over randomization. 

The ultimate objective in the real world is prescription, not prediction.

This amazing and groundbreaking book is a collection of research papers done by Prof. Bertsimas, his colleagues, and Ph.D. students.

When I presented a paper by Bertsimas and Tsitsiklis during my Ph.D. study, my advisor Jeff S. Shamma told me that "read everything written by these 2 persons whenever possible."(*) The clearness of their writing style and thinking methodology will definitely open your eyes and mind.

The academics could access the free version of the software at Interpretable AI.

Some typos to note in case you need to implement the algorithms:

• (5.3) on page 57: The vector a is p dimensional and the number of variables z is m, what does it mean | a_j | <= M z_j ? The objective is to find the minimum support of a, so the index number of z is p instead of m. 
• The 5th line from the bottom of page 72: The summation is over j in S.
• the first equation on page 119: epsilon_j is positive, so it is x_j^(i) - x_j^(i + 1) > 0.
• Page 161: V_i should be V_i^hat in the bottom equation.
• pages 182 - 183, 205 - 207: z_{it} instead of z_{ik}

(*) Prof.  Bertsimas is supervising 20 to 30 Ph.D. students, so it is impossible for me to achieve this goal.