Regression
Regression predicts a number. How does a computer find the best line?
Linear regression with gradient descent
Gradient descent takes steps downhill on the error surface.
The error surface (click to start elsewhere)
- the steps
- the exact answer
- lower error higher error
- Step
- 0
- w0
- –
- w1
- –
- MSE
- –
- RMSE
- –
- MAE
- –
The error after each step (log scale)
MSE, RMSE and MAE: three ways to sum up the errors.
Show the steps
Bias and variance
A simple model misses the same way every time; a complex one differs every time.
12 of the 30 models, their average and the truth
- one model
- the average model
- the true curve
The error on new data, split up, per degree (log scale)
- bias²
- variance
- total
- noise
Regularisation: Ridge, Lasso and Elastic net
A penalty on large weights keeps a complex model from following the noise.
The model on training (circles) and validation (squares) points
The nine weights now
Every weight as λ grows
Decision trees
A tree cuts x into pieces and guesses the mean in each.
The tree's guess: a staircase
The tree
A split reads "x < 0.42": smaller x go left.
Random forest
Many deep trees on bootstrap samples, averaged: the variance drops.
Ten of the trees and the forest's average
Validation error as more trees vote
- the forest
- one tree on all the data
Ensembles: voting and boosting
Bagging, voting and boosting.
Voting: three different models and their average
- polynomial
- k-NN
- tree
- the vote
Validation error for each, and for the vote
Gradient boosting
Each tree learns the residuals of the ones before.
The model after the rounds
What the next tree learns: the residuals
Error per round
- training error
- validation error