Nonlinear Optimization (非線性最佳化)

• Course objective: Nonlinear optimization is widely used in engineering, business, data science, and machine learning. We will introduce fundamental algorithms through examples in this course, enabling students to read research articles, formulate their own problems, and solve them efficiently using the appropriate algorithms.
• Prerequisite subjects (先修科目): 
• (required) Calculus, linear algebra, (Python) programming, any undergraduate optimization course
• (helpful) Machine learning
• Please ensure that you are comfortable with the (introductory) material in math and Python, which are free once you log in.
• Dimitris Bertsimas et al., The Analytics Edge, edX. 
• A beautiful entry course for machine learning and optimization
• Require 10-15 hours per week (I keep saying that working hard is only the entry threshold for outstanding performance!)
• Required course for MIT Sloan Master of Business Analytics (It will cost you 10k US dollars if you take it at MIT. It is free on edX once you register!)
• Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, Mathematics for Machine Learning, Cambridge University Press, 2020.
• Tentative evaluation method: 
• Homework 25%: You are welcome to discuss with your classmates, but please write your own solutions without using LLM tools.
• Midterm 30%
• Final project (1-2 persons per group) 35% (Form for you to fill in)
• Attendance and class participation 10%
• David P. Woodruff, 15-451/651: Algorithm Design and Analysis, CMU, Spring 2025.

For all problems, whether SOLO or GROUP, you must not attempt to find the solution online, in a book, in a journal, by asking an AI tool, or searching anywhere else not explicitly permitted.

• Final project timeline:
• May 8 (4 points): Submit the question after your group meet with the instructor to discuss project ideas in class. Novelty: Your project should propose something new (either a new application, method, or perspective).
• May 15 (3 points): Submit the list of papers you are reading (which submit a one-page summary as a conference paper format to explain your choice to school ilearning)
• May 29 (5 points): Each group will give a short presentation in class about a paper related to their project.  
• June 12 (3 points): PowerPoint review during the class meeting time (Around 1 minute per page). Please upload your files before the class (ppt and pdf, group name-topic, e.g. 1-Federated-learning). 
• June 26 (10 points): Formal presentations (15-20 min). Please upload your files before the class (ppt and pdf). 
• July 3 (10 points): Reports to school ilearning (inclduing modified presentation file, written report as a conference paper format, code) (3-10 pages)
• Main references: 
• Ryan Tibshirani, 10-725 Convex Optimization, CMU, Fall 2018. (videos and lecture notes)
• Stephen J. Wright and Benjamin Recht, Optimization for Data Analysis, Cambridge University Press, 2022.
• Flipped classroom: Please listen to the videos by Prof. Ryan Tibshirani before the class and look at my notes for further explanation. In the class, we will discuss the subtle points and work on the homework problems. 
• Each assignment will include selected exercises from the document "NLO-Homework.pdf" in my Google Drive folder. Please be sure to download the most recent version before starting your homework, as some questions may be revised.
• (Tentative) Schedule: 
1. Holiday
2. Introduction (Homework 1, due Thursday, 3/18, 11:59pm: Questions under Lecture 1 to 3) 
3. Convexity
4. Canonical problem forms (Homework 2, due Thursday, 4/7, 11:59pm: Questions under Lecture 4 to 5) 
• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press (Chapters 4, 6, 7, 8)
• M.X. Goemans and D.P. Williamson, Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming, Journal of the ACM, 1995 (Influential, over 5k citations)
5. Gradient Descent 
6. Holiday
7. (Online) Subgradient (Homework 3, due Thursday, /18, 11:59pm: Questions under Lecture  to ) 
8. Proximal and stochastic gradient descent  
9. Duality
10. Holiday
11. KKT conditions and Duality uses (Homework 4, due Thursday, /18, 11:59pm: Questions under Lecture  to ) 
12. Newton's and Barrier method
13. Midterm  
14. Numerical linear algebra and Coordinate descent (Homework 5, due Thursday, /18, 11:59pm: Questions under Lecture  to ) 
15. Dual ascent and ADMM
16. Frank-Wolfe method and Modern stochastic methods
17. Holiday 
18. Final presentation
• Further references:
• Survey 
• Léon Bottou, Frank E. Curtis, and Jorge Nocedal, Optimization methods for large-scale machine learning, SIAM Review, 2018, 60 (2), 223-311.
• Claudio Gambella, Bissan Ghaddar, and Joe Naoum-Sawaya, Optimization problems for machine learning: A survey, European Journal of Operational Research, 2022, Volume 290; Pages 807-828 (Best Paper Awards 2024, available using CYCU VPN)
• Suvrit Sra, Sebastian Nowozin and Stephen J. Wright, Optimization for Machine Learning, MIT Press, 2011.
• Shiliang Sun, Zehui Cao, Han Zhu, Jing Zhao, A Survey of Optimization Methods from a Machine Learning Perspective, arXiv:1906.06821. 
• Undergraduate textbooks
• Francisco J. Aragón, Miguel A. Goberna, Marco A. López, and Margarita M.L. Rodríguez, Nonlinear Optimization, Springer, 2019. 
• Oliver Stein, Basic Concepts of Nonlinear Optimization, Springer, 2024. 
• Courses:
• Constantine Caramanis, Combinatorial Optimization, Optimization Algorithms, UT Austin, Fall 2020. (YouTube) 
• Yuxin Chen, STAT9910-303: Large-Scale Optimization for Data Science (2023), STAT 991-302: Mathematics of High-Dimensional Data (2022), University of Pennsylvania.
• Nicolas Flammarion, CS-439 Optimization for Machine Learning, EPFL. (videos, github)
• Benyamin Ghojogh, Optimization Techniques (ENGG-6140), Section-2, Winter 2023, University of Guelph.
• Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark Crowley, KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey, arXiv preprint arXiv:2110.01858
• Moritz Hardt, EE 227C Convex Optimization and Approximation, UC Berkeley, Spring 2018. (Python codes)
• Sven Leyffer, Pietro Belotti, and Ashutosh Mahajan, GIAN course on Advances in Mixed-Integer Nonlinear Optimization, IIT Bombay, IEOR, in January 2024. (YouTube, Lectures Slides and Tutorials)
• Chih-Jen Lin, Optimization Methods for Deep Learning (2023), Deep Learning Algorithms and Implementations (2025), NTU.
• MIT, 6.7220 / 15.084 Nonlinear Optimization (6: EECS, 15: Management)
• Mert Pilanci, EE270 Large scale matrix computation, optimization and learning, Stanford University (quarter system)
• Mert Pilanci, EE364b - Convex Optimization II, Stanford University.
• Suvrit Sra, EECS 6.881 Optimization for Machine Learning, MIT. (A few textbooks)
• Madeleine Udell, ORIE 7191: Topics in Optimization for Machine Learning, Cornell.
• Bao Wang, Math 5750/6880 Mathematics of Data Science, Utah. (Self-attention Mechanism and Transformers, Diffusion Models)
• Yinyu Ye, MS&E314 Optimization Algorithms in Market Design, Data Science and Machine Learning, Winter 2023-2024, Stanford.
• Graduate textbooks (with videos and more): You could learn a lot about writing succinctly and clearly from the following terrific and amazing authors.
• Francis Bach, Learning Theory from First Principles, MIT Press, 2025 (blog)
• Aharon Ben-Tal, Aakadi Nemirovski, Laurent El Ghaoui, Robust Optimization, Princeton University Press, 2009.
• Dimitri P. Bertsekas, Nonlinear Programming, Athena Scientific. (book, lecture, and more)
• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press (videos in 2023, cover the whole book in 9 weeks)
• David G. Luenberger and Yinyu Ye, Linear and Nonlinear Programming, Springer, 2021.
• Jorge Nocedal and Stephen J. Wright, Numerical Optimization, Springer, 2006.
• John Wright and Yi Ma, High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications, Cambridge University Press, 2022. (videos)
• Applications:
• R. Venkata Rao and Vimal J. Savsani, Mechanical Design Optimization using Advanced Optimization Techniques, Springer, 2012. 
• More