Optimization Methods

Siliang Zhang (slzhang at ecnu dot edu dot cn)

Fall 2023, 2024, 2025

Course Information:

• Course Code: STAT0031132114.01

• Prerequisites: Calculus and Linear Algebra

• Language: English textbooks and materials, Chinese instruction

• Course Website: Optimization Methods

教材与参考资料 Textbooks & References

主教材 Primary Textbook:

• Numerical Optimization by Jorge Nocedal and Stephen J. Wright, Springer, 2006

辅助教材 Supplementary Textbook:

• 最优化：建模、算法与理论（第二版） by 刘浩洋, 户将, 李勇锋，文再文, 高教出版社，2022.

参考资料 References:

• Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge University Press, 2004

• 数值最优化方法 by 高立, 北京大学出版社, 2014

• Probabilistic Machine Learning - An Introduction by Kevin P. Murphy, MIT Press, 2023.

• Python Data Science Handbook by Jake VanderPlas, O'Reilly Media, 2016

• Matrix Algebra Useful for Statistics by Searle and Khuri.

• Proximal Algorithms by Parikh and Boyd, Foundations and Trends in optimization, 2014.

• Optimization Theory and Methods: Nonlinear Programming by Sun and Yuan, Springer Science & Business Media, 2006.

• Optimization Methods for Large-Scale Machine Learning by Bottou at el.

课程简介 Course Description

本课程是为数学、统计学和工程等相关专业的学生设计的一门非常重要的专业选修课程。优化方法的核心内容是研究如何从一组可能的选择中找出最佳或者“最优”的方法。最优化方法不仅是许多数学和工程学科的基础，同时也在大数据、机器学习和人工智能等现代技术领域中发挥着至关重要的作用。

本课程主要讲授：(1) 最优化的基本理论：包括函数、导数和梯度，凸性和凹性的概念，线性规划的基本理论和方法，以及非线性优化的基本理论和方法；(2) 典型优化问题和方法：包括线性规划，非线性优化，无约束优化和约束优化，拉格朗日乘数法和KKT条件；(3) 优化算法和实践：包括梯度下降法、随机梯度下降法、牛顿法、内点法等的原理和Python实现；(4) 通过案例学习的方式，了解最优化在统计学、机器学习、运筹学、经济学等领域的应用。

This course is a highly important professional elective designed for undergraduate students majoring in mathematics, statistics, engineering, and related fields. Optimization methods study how to find the best or “optimal” solution from a set of possible choices, forming the foundation of many mathematical and engineering disciplines while playing a vital role in modern technological areas such as big data, machine learning, and artificial intelligence.

The course primarily focuses on: (1) Basic theories of optimization: encompassing functions, derivatives and gradients, concepts of convexity and concavity, fundamental theories and methods of linear programming, and basic theories and methods of nonlinear optimization; (2) Typical optimization problems and methods: including linear programming, nonlinear optimization, both unconstrained and constrained optimization, the Lagrange multiplier method, and KKT conditions; (3) Optimization algorithms and practice: covering the principles and Python implementations of Gradient Descent (GD), Stochastic Gradient Descent (SGD), Newton-Raphson Method, Interior Point Method, etc., and how to address actual optimization problems using Python; (4) Through case studies, understanding the applications of optimization in various fields such as statistics, machine learning, operations research, economics, etc.

课程目标 Course Objectives

• 掌握最优化基本概念、函数性质、凸性理论和线性/非线性优化方法

• 理解典型优化算法的原理和适用范围，能够针对具体问题选择合适的算法

• 熟练使用Python实现优化算法，解决实际最优化问题

• 培养批判性思维和问题分析能力，具备独立解决复杂优化问题的能力

教学内容 Course Content

章节	主题	周
第一章 最优化方法导论和Python基础	导论	1-2
第一章 最优化方法导论和Python基础	Python基础与最优化	2
第二章 最优化基本理论与方法	函数、导数和梯度	3
第二章 最优化基本理论与方法	凸性和凹性	4
第三章 典型优化问题与最优性理论	线性规划与最小二乘问题	5
第三章 典型优化问题与最优性理论	❖复杂优化问题❖	6
第三章 典型优化问题与最优性理论	❖高级优化问题❖	7
第三章 典型优化问题与最优性理论	❖最优性理论（上）❖	8
第三章 典型优化问题与最优性理论	❖最优性理论（下）❖	9
第四章 优化算法及其实现	无约束优化算法	10
第四章 优化算法及其实现	约束优化算法	11
高级优化主题选讲	Proximal算法、流形约束优化、非凸优化等	12

Assessment Focus: Theoretical understanding, practical programming skills, problem-solving ability, and professional presentation skills