Efficient method for training deep networks with unitary matrices

Bobak Kiani, Randall Balestriero, Yann Lecun, and Seth Lloyd,  projUNN: efficient method for training deep networks with unitary matrices, arXiv:2203.05483v2.

In learning with recurrent or very deep feed-forward networks, employing unitary matrices in each layer can be very effective at maintaining long-range stability. However, restricting network parameters to be unitary typically comes at the cost of expensive parameterizations or increased training runtime. We propose instead an efficient method based on rank-k updates -- or their rank-k approximation -- that maintains performance at a nearly optimal training runtime. We introduce two variants of this method, named Direct (projUNN-D) and Tangent (projUNN-T) projected Unitary Neural Networks, that can parameterize full N-dimensional unitary or orthogonal matrices with a training runtime scaling as O(kN^2). Our method either projects low-rank gradients onto the closest unitary matrix (projUNN-T) or transports unitary matrices in the direction of the low-rank gradient (projUNN-D). Even in the fastest setting (k=1), projUNN is able to train a model's unitary parameters to reach comparable performances against baseline implementations. By integrating our projUNN algorithm into both recurrent and convolutional neural networks, our models can closely match or exceed benchmarked results from state-of-the-art algorithms.

Strong mixed-integer programming formulations for trained neural networks

R. Anderson, J. Huchette, W. Ma, C. Tjandraatmadja, and J.P. Vielma, Strong mixed-integer programming formulations for trained neural networks, Mathematical Programming, 2020, 183(1-2):3-39, ISSN 14364646, URL http://dx.doi.org/10.1007/s10107-020-01474-5.

We present strong mixed-integer programming (MIP) formulations for high-dimensional piecewise linear functions that correspond to trained neural networks. These formulations can be used for a number of important tasks, such as verifying that an image classification network is robust to adversarial inputs, or solving decision problems where the objective function is a machine learning model. We present a generic framework, which may be of independent interest, that provides a way to construct sharp or ideal formulations for the maximum of d affine functions over arbitrary polyhedral input domains. We apply this result to derive MIP formulations for a number of the most popular nonlinear operations (e.g. ReLU and max pooling) that are strictly stronger than other approaches from the literature. We corroborate this computationally, showing that our formulations are able to offer substantial improvements in solve time on verification tasks for image classification networks.

Machine Learning into Metaheuristics: A Survey and Taxonomy

El-Ghazali Talbi, Machine Learning into Metaheuristics: A Survey and Taxonomy, ACM Computing Surveys, Volume 54, Issue 6, July 2022, Article No.: 129, pp 1–32, https://doi.org/10.1145/3459664.

 During the past few years, research in applying machine learning (ML) to design efficient, effective, and robust metaheuristics has become increasingly popular. Many of those machine learning-supported metaheuristics have generated high-quality results and represent state-of-the-art optimization algorithms. Although various appproaches have been proposed, there is a lack of a comprehensive survey and taxonomy on this research topic. In this article, we will investigate different opportunities for using ML into metaheuristics. We define uniformly the various ways synergies that might be achieved. A detailed taxonomy is proposed according to the concerned search component: target optimization problem and low-level and high-level components of metaheuristics. Our goal is also to motivate researchers in optimization to include ideas from ML into metaheuristics. We identify some open research issues in this topic that need further in-depth investigations.

Global and Robust Optimization for Engineering Design

 Berk Öztürk, Global and Robust Optimization for Engineering Design, Ph.D. Thesis, MIT, 2022. (thesis, code, talk)

There is a need to adapt and improve conceptual design methods through better optimization, in order to address the challenge of designing future engineered systems. Aerospace design problems are tightly-coupled optimization problems, and require all-at-once solution methods for design consensus and global optimality. Although the literature on design optimization has been growing, it has generally focused on the use of gradient-based and heuristic methods, which are limited to local and low-dimensional optimization respectively. There are significant benefits to leveraging structured mathematical optimization instead. Mathematical optimization provides guarantees of solution quality, and is fast, scalable, and compatible with using physics-based models in design. More importantly perhaps, there has been a wave of research in optimization and machine learning that provides new opportunities to improve the engineering design process. This thesis capitalizes on two such opportunities.

Tackling Climate Change with Machine Learning

David Rolnic et al., Tackling Climate Change with Machine Learning, ACM Computing Surveys, Volume 55, Issue 2, March 2023, Article No.: 42, pp 1–96, https://doi.org/10.1145/3485128

Climate change is one of the greatest challenges facing humanity, and we, as machine learning (ML) experts, may wonder how we can help. Here we describe how ML can be a powerful tool in reducing greenhouse gas emissions and helping society adapt to a changing climate. From smart grids to disaster management, we identify high impact problems where existing gaps can be filled by ML, in collaboration with other fields. Our recommendations encompass exciting research questions as well as promising business opportunities. We call on the ML community to join the global effort against climate change.

from lab bench to public office and back

Chen Chien-jen, Taiwan’s pandemic vice-president — from lab bench to public office and back, Nature 603, 203 (2022)

Two years on from the World Health Organization’s official declaration of the pandemic, I’ve been thinking about lessons I’ve learnt toggling between science and public service. I think all researchers — from bench scientists to physicists to computational social scientists — might find this exercise useful. Government advisers, too.

Principles for Dealing with the Changing World Order by Ray Dalio

 Principles for Dealing with the Changing World Order by Ray Dalio

To make money, study maths or economics at a top university

 Economist, To make money, study maths or economics at a top university, Nov 26th, 2019.

Datasets for machine learning and more

Easy to download:

• Jason Brownlee, Machine learning datasets used in tutorials on MachineLearningMastery.com

• M. Fernández-Delgado, E. Cernadas, S. Barro, & D. Amorim, (2014). Do we need hundreds of classifiers to solve real world classification problems?. The Journal of Machine Learning Research, 15(1), 3133-3181. (121 datasets in a click!)

• https://datasetsearch.research.google.com/

• Prognostics Data Repository

• scikit-learn

讓 AI 幫你最佳化太陽能電池材料的製程參數

採訪撰文 簡克志，美術設計 林洵安，機器學習 x 鈣鈦礦材料：讓 AI 幫你最佳化太陽能電池材料的製程參數！，研之有物，2022-02-21

機器學習輔助材料設計

為了 2050 淨零排放的目標，太陽能發電為不可或缺的再生能源之一，其中「鈣鈦礦太陽能電池」是近年最熱門的研究領域，不僅成本低廉、光電轉換效率也可達到 25%。然而，鈣鈦礦材料在環境中容易降解，影響使用壽命。材料科學家為了做出效能好又穩定的鈣鈦礦「料理」，無不卯足了勁，替這道菜加上各種「食材」，但是越複雜的菜，調出好味道就越困難。人腦畢竟有限，如果交給機器呢？中央研究院「研之有物」專訪院內應用科學研究中心包淳偉研究員，他與團隊訓練了一套機器學習模型，可以又快又準的找出複雜鈣鈦礦材料的最佳化條件！

Interpretable machine learning by Molnar

Molnar, Christoph. “Interpretable machine learning. A Guide for Making Black Box Models Explainable”, 2019. https://christophm.github.io/interpretable-ml-book/.

This book started as a side project when I was working as a statistician in clinical research. I worked four days a week, and on my “day off” I worked on side projects. Eventually, interpretable machine learning became one of my side projects. At first I had no intention of writing a book. Instead, I was simply interested in finding out more about interpretable machine learning and was looking for good resources to learn from. Given the success of machine learning and the importance of interpretability, I expected that there would be tons of books and tutorials on this topic. But I only found the relevant research papers and a few blog posts scattered around the internet, but nothing with a good overview. No books, no tutorials, no overview papers, nothing. This gap inspired me to start writing this book. I ended up writing the book I wished was available when I began my study of interpretable machine learning. My intention with this book was twofold: to learn for myself and to share this new knowledge with others.

分工和效率

備課的時候，除了論文和雜誌以外，我習慣看很多的書，以便整理出適合學生的教材。

系上配置4位助理，所以請助理幫我輸入圖書推薦清單。沒想到，才兩個禮拜，圖書館就已經通知我，可以借其中的十本。更開心的是，學校允許老師較長的借期，算一算，1年後才需回館。

亞當斯密早在十八世紀，就已經強調分工的重要性，以提升所有人的效率。

從排隊理論的觀點來看，如果一個隨機的服務系統，當使用率接近1的時候 ，就會產生事情延誤的狀況。更不要說，需要規劃與思考的工作，太過忙碌，很容易掛一漏萬。

對管理者而言，當然很難決定資源的配置。高強教授當成大校長的時候，使用資料包絡法，分析和比較系所的生產力。某董事長使用類比法，她說：等到你們研究像在坐的某教授一樣好 (剛從中正退休)，再來爭取。

無心插柳

系助理幫我發郵件給研究生，沒想到，有一位菲律賓籍的博一學生，之前在菲律賓當講師，說她努力工作，要來當我的助教。她說不懂中文，很擔心。想想以前，在國外的情景；就安慰她，沒有關係，一切都可以學習。

系上的研究所課程，只要有外籍學生，就 (希望) 用英文上課。沒想到，這位學生已經找了兩位外籍 (和外所) 生要來修機器學習。

我的錄影課程，用中文教學，採翻轉式教學。真的希望，有像台大學生程度的助教，幫我的影片轉成英文。

當然，可遇不可求。只好趕緊準備，每個禮拜的重點提示。