Advanced Optimization and Automated Machine Learning

School year 2020/2021

To take this class for credit, you must register to eCampus. This is where you can turn in your assignments and be graded.

Anyone who is respectful of the class is welcome to audit using this VIDEO-CONFERENCE link. Classes will be held by tele-presence.

Schedule

Classes begin Jan 07 2021.

• Classes are held every Thursday afternoon @ 3pm CET (15h): quiz correction + lecture + TP. Quizzes are due before class starts.

• Office hour @ 6 pm CET (18h) on Wednesday, starting Jan 13.

Lecture notes

Lecture notes are accessible on-line.

Since this is the first year we are teaching this class, this is work-in-progress. If you report mistakes, you effort will be acknowledged in the notes!!

Slides

Following the students' request, we will try to have a class as interactive as possible, use few pre-made slides, and make hand-write during the lecture. The slides will be made available by clicking on the link of the class.

1. Jan 7 - Introduction, taxonomy of optimization problems, examples of optimization problems, basic methods (stochastic gradient, grid search, random search).

2. Jan 14 - Unconstrained continuous optimization, gradient methods, acceleration and reduction of variance of the stochastic gradient (momentum, Nesterov, Adam), true gradient methods (LBFGS, conjugate gradients), notions of speed of convergence for the problems convex.

3. Jan 21- Constraint optimization (linear programming, quadratic programming, convex programming)

4. Jan 28 -Black box optimization (optimization without gradient, simulated annealing and MCMC, evolutionary methods, CMA-ES, Bayesian methods)

5. Feb 4 - Hyper-parameter optimization (application of black box methods (wrapper) e.g. SMAC, `` embedded '' methods, regularization paths)

6. Feb 11 - Neural Architecture Search (search space, constructive methods, elimination methods)

7. Feb 18 - Meta-learning (taxonomy of meta-learning, methods of recommendation, few-shot learning, transfer learning, learning to optimize)

Homework and grades

The students can earn a total of 100 points in the class:

1) Continuous control (CC) will be performed via quizzes DUE BEFORE CLASS STARTS and available on eCampus.

There are 7 quizzes. Each quiz is worth 10 points. Total CC = 70 points.

2) Project: the students must read and present a paper of their choice, see INSTRUCTIONS.

The presentations take place in class between Jan 21 and Feb 18. A temporary grade will be give. The final report is due March 1, 2021. Worth 30 points.