Siliang Zhang (slzhang at ecnu dot edu dot cn)
Spring 2024 (Graduate)
Spring 2023 (Graduate)
Spring 2022 (Graduate)
Convex Optimization by S. Boyd and L. Vandenberghe, Cambridge University Press, 2004
Introductory Lectures on Convex Optimization: a basic course by Yurii Nesterov, Springer Science & Business Media, 2003.
Numerical Optimization by Jorge Nocedal and Stephen J. Wright, Springer, 2006.
Computational Statistics by Givens and Hoeting, John Wiley & Sons, 2012.
Matrix Algebra Useful for Statistics by Searle and Khuri.
Disciplined Convex Programming and CVX by S. Boyd.
CVXPY Online Document for package installation, tutorial, and examples.
Proximal Algorithms by Parikh and Boyd, Foundations and Trends in optimization, 2014.
Optimization Theory and Methods: Nonlinear Programming by Sun and Yuan, Springer Science & Business Media, 2006.
Optimization Methods for Large-Scale Machine Learning by Bottou at el.
近些年来，机器学习与计算统计学发展迅速，几乎其中的每个问题都可被转化成特定约束下某些函数的优化问题。但不是所有的机器学习问题都可被很好的解决，这意味着没有一般性的方法解决优化问题。然而，很多我们感兴趣的问题可被看作具有独特性质的优化问题，这些特性包含凸性、光滑、稀疏、可分性等，这些特性使得标准化的高效求解成为可能。
本课程旨在提供研究生阶段优化问题的背景知识，包括基本特性和这些特性在优化中所起作用。更广泛地，学生将学习利用这些特性的经典优化算法。课程将重点讨论凸优化问题（根据课程进度也会涉及少部分非凸优化内容）。本课程的教学中将重新审视机器学习和统计学中很多重要的应用问题。基于本课程的学习，学生将能够从整体的理解优化问题，并将其思想与技巧应用到机器学习或统计为背景的具体问题中。本课程面向统计、金融、工程、心理测量等研究生。
In recent years, machine learning and computational statistics have developed rapidly, and almost every problem in it can be formulated as an optimization problem of some functions under certain constraints. Obviously, not all machine learning problems can be solved well, which means that we cannot solve the corresponding optimization problems in general. However, many interesting problems in statistics and machine learning can be viewed as optimization problems with unique properties, such as convexity, smoothness, sparsity, separability, etc. It is these properties that permitting standardized and efficient solutions.
This course is designed to provide students with a background in optimization problems at the graduate-level, including the fundamental properties and the role these properties play in optimization, and more generally, students will learn classical algorithms that exploit these properties. The course will focus on convex optimization problems (additionally, a small amount of non-convex optimization will be covered depending on the course progresses). Many important application problems in machine learning and statistics will be revisited during the teaching. Upon completing of this course, students will be able to understand optimization problems as a whole and apply the key ideas and techniques to specific problems with machine learning or statistics background. This course is aimed at graduated students in statistics, finance, management, psychometrics etc.
介绍数值代数背景知识
介绍凸优化中的主要方法及理论，特别是关键特性，例如凸性、光滑、稀疏行等，让参与学生能够理解数学优化这一主题中的核心思想，训练学生能够将一般优化问题改写为具有这些特性的问题
学习凸优化在解决几类问题中的经典算法，理解不同算法的优缺点及适用范围，能够根据特定问题，选择合适的算法
通过高级编程语言（Python）的实践课程，加深学生对理论方法的理解，训练相应的编程能力，能够实现优化算法解决问题
学习凸优化在机器学习、统计学中的应用，并为将来工作和进一步学习研究更广法的内容打下基础
| 章节 | 主题 | 周 |
| 第一章 凸优化基本概念与理论 | Intro, 凸集与凸函数 | 1-2 |
| 第一章 凸优化基本概念与理论 | 优化基本概念 | 2 |
| 第一章 凸优化基本概念与理论 | ❖典型问题形式❖ | 3 |
| 第二章 算法1:一阶方法 | 梯度下降（GD） | 4 |
| 第二章 算法1:一阶方法 | 次梯度与次梯度方法 | 5 |
| 第二章 算法1:一阶方法 | ❖近段梯度下降及加速❖ | 6 |
| 第二章 算法1:一阶方法 | 随机梯度下降（SGD） | 6 |
| 第三章 对偶与最优性 | 线性规划中的对偶性 | 7 |
| 第三章 对偶与最优性 | ❖一般规划问题中的对偶性❖ | 8 |
| 第三章 对偶与最优性 | ❖KKT条件❖ | 9 |
| 第三章 对偶与最优性 | 对偶性应用 | 9 |
| 第四章 算法2:二阶方法 | 牛顿法 | 10 |
| 第四章 算法2:二阶方法 | 内点方法 | 11 |
| 高级优化主题选讲 | 拟牛顿法、对偶分解、ADMM、非凸优化等 | 12 |