讀書寫作

12/24/2022

Robust Stochastic Optimization Made Easy

Chen, Zhi, and Peng Xiong. 2021. RSOME in Python: an open-source package for robust stochastic optimization made easy. Optimization Online.

Chen, Zhi, Melvyn Sim, Peng Xiong. 2020. Robust stochastic optimization made easy with RSOME. Management Science 66(8) 3329–3339.

RSOME in Python (Also solver interfaces)

RSOME (Robust Stochastic Optimization Made Easy) is an open-source Python package for modeling generic optimization problems. Models in RSOME are constructed by variables, constraints, and expressions that are formatted as N-dimensional arrays. These arrays are consistent with the NumPy library in terms of syntax and operations, including broadcasting, indexing, slicing, element-wise operations, and matrix calculation rules, among others. In short, RSOME provides a convenient platform to facilitate developments of optimization models and their applications.

12/22/2022

Training One Million Machine Learning Models in Record Time with Ray

Eric Liang and Robert Nishihara, Training One Million Machine Learning Models in Record Time with Ray, Anyscale, December 17, 2022.

Ray and Anyscale are used by companies like Instacart to speed up machine learning training workloads (often demand forecasting) by 10x compared with tools like Celery, AWS Batch, SageMaker, Vertex AI, Dask, and more.

In this blog, we’ll cover:

Why companies are doing many model training
How to use Ray to train multiple models
The properties of Ray that enable efficient many model training

12/18/2022

The Forward-Forward Algorithm

Geoffrey Hinton, The Forward-Forward Algorithm: Some Preliminary Investigations, 2022. (code)

The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth serious investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes can be separated in time, the negative passes can be done offline, which makes the learning much simpler in the positive pass and allows video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.