生活中到處充滿著不確定性，例如吃飯排隊的等候時間、機台生產的良率、民調統計數字的分析等等。工業系也開設許多相關的課程，例如品質管制、資料分析、智慧製造、實驗設計、機器學習、人工智慧等等，以解決工商業的問題。不確定性和雜訊，也是 (電機、軟工、機械、化工、土木、生醫、材料) 工程系統中常見的問題。
許多人修這門課的時候，很痛苦 (註 1)。除了了解其應用外， 有許多的網路資源：
Steven L. Brunton and J. Nathan Kutz, Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control, 2nd edition, Cambridge University Press (pdf, amazing videos, connection with dynamical systems in engineering)
R. Sambharya and B. Stellato, Data-Driven Performance Guarantees for Classical and Learned Optimizers, arXiv e-prints:2404.13831,2024. (Python code)
Kaufmann, E., Bauersfeld, L., Loquercio, A. et al. Champion-level drone racing using deep reinforcement learning. Nature 620, 982–987 (2023). https://doi.org/10.1038/s41586-023-06419-4
First-person view (FPV) drone racing is a televised sport in which professional competitors pilot high-speed aircraft through a 3D circuit. Each pilot sees the environment from the perspective of their drone by means of video streamed from an onboard camera. Reaching the level of professional pilots with an autonomous drone is challenging because the robot needs to fly at its physical limits while estimating its speed and location in the circuit exclusively from onboard sensors. Here we introduce Swift, an autonomous system that can race physical vehicles at the level of the human world champions. The system combines deep reinforcement learning (RL) in simulation with data collected in the physical world. Swift competed against three human champions, including the world champions of two international leagues, in real-world head-to-head races. Swift won several races against each of the human champions and demonstrated the fastest recorded race time. This work represents a milestone for mobile robotics and machine intelligence, which may inspire the deployment of hybrid learning-based solutions in other physical systems.
N. Schlezinger, Y. Eldar, and S. Boyd, Model-Based Deep Learning: On the Intersection of Deep Learning and Optimization, IEEE Access, vol. 10, 2022.
Decision making algorithms are used in a multitude of different applications. Conventional approaches for designing decision algorithms employ principled and simplified modelling, based on which one can determine decisions via tractable optimization. More recently, deep learning approaches that use highly parametric architectures tuned from data without relying on mathematical models, are becoming increasingly popular. Model-based optimization and data-centric deep learning are often considered to be distinct disciplines. Here, we characterize them as edges of a continuous spectrum varying in specificity and parameterization, and provide a tutorial-style presentation to the methodologies lying in the middle ground of this spectrum, referred to as model-based deep learning. We accompany our presentation with running examples in super-resolution and stochastic control, and show how they are expressed using the provided characterization and specialized in each of the detailed methodologies. The gains of combining model-based optimization and deep learning are demonstrated using experimental results in various applications, ranging from biomedical imaging to digital communications.
Dimitris Bertsimas and Dick Den Hertog, Robust and Adaptive Optimization, Dynamic Ideas, 2022.
The purpose of this book is to provide a unified, insightful, and original treatment of robust and adaptive optimization.
Autopilot Safety
In 2021, we recorded 0.22 crashes for every million miles driven in which drivers were using Autopilot technology (Autosteer and active safety features). For drivers who were not using Autopilot technology (no Autosteer and active safety features), we recorded 0.77 crashes for every million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there are 1.81 automobile crashes for every million miles driven.
Wurman, P.R., Barrett, S., Kawamoto, K. et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature 602, 223–228 (2022). https://doi.org/10.1038/s41586-021-04357-7.
Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical manoeuvres to pass or block opponents while operating their vehicles at their traction limits. Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the non-linear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world’s best e-sports drivers. We combine state-of-the-art, model-free, deep reinforcement learning algorithms with mixed-scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing’s important, but under-specified, sportsmanship rules. We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world’s best Gran Turismo drivers. By describing how we trained championship-level racers, we demonstrate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.
Joy Zhang, 7 real-world applications of reinforcement learning, gocoder, February 17, 2022
1. Autonomous driving with Wayve
2. Personalizing your Netflix recommendations
3. Optimizing inventory levels for Walmart
4. Improving search engine results with search.io
5. Improving language models with OpenAI's WebGPT
6. Trading on the financial markets with IBM's DSX platform
7. Robotics with the University of California, Berkeley
W.B. Powell, A unified framework for stochastic optimization, European Journal of Operational Research, 2019, Volume 275, Issue 3, 16 June 2019, Pages 795-821. (pdf)
Stochastic optimization is an umbrella term that includes over a dozen fragmented communities, using a patchwork of sometimes overlapping notational systems with algorithmic strategies that are suited to specific classes of problems. This paper reviews the canonical models of these communities, and proposes a universal modeling framework that encompasses all of these competing approaches. At the heart is an objective function that optimizes over policies that is standard in some approaches, but foreign to others. We then identify four meta-classes of policies that encompasses all of the approaches that we have identified in the research literature or industry practice. In the process, we observe that any adaptive learning algorithm, whether it is derivative-based or derivative-free, is a form of policy that can be tuned to optimize either the cumulative reward (similar to multi-armed bandit problems) or final reward (as is used in ranking and selection or stochastic search). We argue that the principles of bandit problems, long a niche community, should become a core dimension of mainstream stochastic optimization.
Paul Vicol, Luke Metz, and Jascha Sohl-Dickstein, Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies, ICML 2021. (paper, Outstanding Paper Awards)
Unrolled computation graphs arise in many scenarios, including training RNNs, tuning hyperparameters through unrolled optimization, and training learned optimizers. Current approaches to optimizing parameters in such computation graphs suffer from high variance gradients, bias, slow updates, or large memory usage. We introduce a method called Persistent Evolution Strategies (PES), which divides the computation graph into a series of truncated unrolls, and performs an evolution strategies-based update step after each unroll. PES eliminates bias from these truncations by accumulating correction terms over the entire sequence of unrolls. PES allows for rapid parameter updates, has low memory usage, is unbiased, and has reasonable variance characteristics. We experimentally demonstrate the advantages of PES compared to several other methods for gradient estimation on synthetic tasks, and show its applicability to training learned optimizers and tuning hyperparameters.
S. Meyn, Control Systems and Reinforcement Learning, Cambridge University Press, 2021. (pdf)