All courses › Machine Learning and Data Analysis › Learning rate

Learning rate

The learning rate is the most important choice in gradient descent. If it is too large, the weights jump past the minimum and the loss can blow up. If it is too small, learning is very slow. Often several values are tried while watching the loss curve.

η∈[10−4, 10−1]\eta \in [10^{-4},\ 10^{-1}]common range to try
Lt+1>LtL_{t+1} > L_ta sign of too large a learning rate

Symbols

η\etalearning rate
LtL_tloss after step t

Example

The loss goes 5, 12, 40, 150 … : the learning rate is too large.

The loss goes 5, 4.99, 4.98 … : it is probably too small.

Plot the loss for every step. The curve almost always tells you what is wrong.
Practise linear regression and gradient descent for free →

← Gradient descent · The confusion matrix →

Part of Machine Learning and Data Analysis: Linear regression and gradient descent.