Nonlinear Optimization (非線性最佳化)

• Course objective: Nonlinear optimization is widely used in engineering, business, data science, and machine learning. We will introduce fundamental algorithms through examples in this course, enabling students to read research articles, formulate their own problems, and solve them efficiently using the appropriate algorithms.
• Prerequisite subjects (先修科目): 
• (required) Calculus, linear algebra, (Python) programming, any undergraduate optimization course
• (helpful) Machine learning
• Please ensure that you are comfortable with the (introductory) material in math and Python, which are free once you log in.
• Dimitris Bertsimas et al., The Analytics Edge, edX. 
• A beautiful entry course for machine learning and optimization
• Require 10-15 hours per week (I keep saying that working hard is only the entry threshold for outstanding performance!)
• Required course for MIT Sloan Master of Business Analytics (It will cost you 10k US dollars if you take it at MIT. It is free on edX once you register!)
• Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning, Cambridge University Press, 2020.
• Tentative evaluation method: 
• Biweekly homework: You are welcome to discuss with your classmates, but please write your own solutions.
• Midterm (or paper reading)
• Final project (1-2 persons per group)
• Main references: 
• Ryan Tibshirani, 10-725 Convex Optimization, CMU, Fall 2018. (videos and lecture notes)
• Stephen J. Wright and Benjamin Recht, Optimization for Data Analysis, Cambridge University Press, 2022.
• Flipped classroom: Please listen to the videos in Ryan Tibshirani before the class and look at my notes for further explanation. In the class, we will discuss the subtle points and work on the homework problems. 
• (Tentative) Schedule: 
1. Holiday
2. Introduction
3. Convexity
4. Canonical problem forms
• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press
• M.X. Goemans and D.P. Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, Journal of the ACM (JACM), 1995 (Influential, over 5k citations)
5. Gradient Descent 
6. Holiday
7. (Online) Subgradient
8. Proximal and stochastic gradient descent  
9. Duality
10. Holiday
11. KKT conditions and Duality uses
12.  Newton's and Barrier method
13. (Online) Quasi-Newton methods and Proximal Newton method
14. (Online) Numerical linear algebra and Coordinate descent
15. Dual ascent and ADMM
16. Frank-Wolfe method and Modern stochastic methods
17. Holiday 
18. Final presentation
• Further references:
• Survey 
• Léon Bottou, Frank E. Curtis, and Jorge Nocedal, Optimization methods for large-scale machine learning, SIAM Review, 2018, 60 (2), 223-311.
• Claudio Gambella, Bissan Ghaddar, and Joe Naoum-Sawaya, Optimization problems for machine learning: A survey, European Journal of Operational Research, 2022, Volume 290; Pages 807-828 (Best Paper Awards 2024, available using CYCU VPN)
• Suvrit Sra, Sebastian Nowozin and Stephen J. Wright, Optimization for Machine Learning, MIT Press, 2011.
• Shiliang Sun, Zehui Cao, Han Zhu, Jing Zhao, A Survey of Optimization Methods from a Machine Learning Perspective, arXiv:1906.06821. 
• Undergraduate textbooks
• Francisco J. Aragón, Miguel A. Goberna, Marco A. López, and Margarita M.L. Rodríguez, Nonlinear Optimization, Springer, 2019. 
• Oliver Stein, Basic Concepts of Nonlinear Optimization, Springer, 2024. 
• Courses:
• Constantine Caramanis, Combinatorial Optimization, UT Austin, Fall 2020. (YouTube)
• Yuxin Chen, STAT9910-303: Large-Scale Optimization for Data Science (2023), STAT 991-302: Mathematics of High-Dimensional Data (2022), University of Pennsylvania.
• Nicolas Flammarion, CS-439 Optimization for Machine Learning, EPFL. (videos, github)
• Benyamin Ghojogh, Optimization Techniques (ENGG-6140), Section-2, Winter 2023, University of Guelph.
• Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark Crowley, KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey, arXiv preprint arXiv:2110.01858
• Moritz Hardt, EE 227C Convex Optimization and Approximation, UC Berkeley, Spring 2018. (Python codes)
• Sven Leyffer, Pietro Belotti, and Ashutosh Mahajan, GIAN course on Advances in Mixed-Integer Nonlinear Optimization, IIT Bombay, IEOR, in January 2024. (YouTube, Lectures Slides and Tutorials)
• Chih-Jen Lin, Optimization Methods for Deep Learning (2023), Deep Learning Algorithms and Implementations (2025), NTU.
• MIT, 6.7220 / 15.084 Nonlinear Optimization (6: EECS, 15: Management)
• Suvrit Sra, EECS 6.881 Optimization for Machine Learning, MIT. (A few textbooks)
• Madeleine Udell, ORIE 7191: Topics in Optimization for Machine Learning, Cornell.
• Bao Wang, Math 5750/6880 Mathematics of Data Science, Utah. (Self-attention Mechanism and Transformers, Diffusion Models)
• Yinyu Ye, MS&E314 Optimization Algorithms in Market Design, Data Science and Machine Learning, Winter 2023-2024, Stanford.
• Graduate textbooks (with videos and more): You could learn a lot about writing succinctly and clearly from the following terrific and amazing authors.
• Francis Bach, Learning Theory from First Principles, MIT Press, 2025 (blog)
• Aharon Ben-Tal, Aakadi Nemirovski, Laurent El Ghaoui, Robust Optimization, Princeton University Press, 2009.
• Dimitri P. Bertsekas, Nonlinear Programming, Athena Scientific. (book, lecture, and more)
• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press (videos in 2023, cover the whole book in 9 weeks)
• David G. Luenberger and Yinyu Ye, Linear and Nonlinear Programming, Springer, 2021.
• Jorge Nocedal and Stephen J. Wright, Numerical Optimization, Springer, 2006.
• John Wright and Yi Ma, High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications, Cambridge University Press, 2022. (videos)
• Applications:
• R. Venkata Rao and Vimal J. Savsani, Mechanical Design Optimization using Advanced Optimization Techniques, Springer, 2012. 
• More