All courses › Machine Learning and Data Analysis › Gradient descent

Gradient descent

Gradient descent finds the weights that give the smallest loss by taking small steps downhill. The gradient points uphill towards increasing loss, so we go the opposite way. The learning rate decides how long the steps are.

w \leftarrow w - \eta\,\frac{\partial L}{\partial w}

one step

\frac{\partial L}{\partial w} = -\frac{2}{n}\sum x_i\,(y_i - \hat y_i)

gradient for MSE

Symbols

$\eta$	learning rate
$L$	loss function
$\frac{\partial L}{\partial w}$	gradient

Example

$w = 3$ , gradient 4 and $\eta = 0.1$ :

$w \leftarrow 3 - 0.4 = 2.6$ .

Do not forget the minus sign. We want to go downhill, towards lower loss.

Practise linear regression and gradient descent for free →

← Linear model · Learning rate →

Part of Machine Learning and Data Analysis: Linear regression and gradient descent.