Machine Learning and Data Analysis (ML): free practice, theory and problems

1. Data and core concepts

What is it about?

Machine learning means letting the computer find patterns in data instead of us writing the rules. You give the model many examples, and it learns a relationship it can apply to new cases: spotting faults on a production line, estimating tomorrow's power use or sorting email.

Concepts and formulas

• Features $x$ are what the model receives. The label $y$ is the answer we want it to give.
• Supervised learning: the examples have a known answer $y$. Regression gives a number, classification gives a category.
• Unsupervised learning: no answers. For example clustering, which finds groups.
• Training and test sets: the model learns on training data and is measured on test data it has never seen, often split 80/20.
• Overfitting: the model memorizes the training data, noise included, and does poorly on new data.
• Min–max normalization: $x' = \dfrac{x - x_{min}}{x_{max} - x_{min}}$ gives values between 0 and 1.
• Standardization: $z = \dfrac{x - \mu}{\sigma}$ gives mean 0 and standard deviation 1.
• Mean squared error: $\text{MSE} = \dfrac{1}{n}\sum (y_i - \hat y_i)^2$.

How to solve the problems

1. Work out what $x$ is and what $y$ is, and whether the answer is a number or a category.
2. For normalization: find min and max (or $\mu$ and $\sigma$) and insert.
3. For error measures: compute the error for each point, square it, and take the mean.

Example

The temperatures 10, 15 and 30 °C are to be min–max normalized. What does 15 become?

1. $x_{min} = 10$, $x_{max} = 30$.
2. $x' = (15 - 10)/(30 - 10) = 5/20 = 0.25$.

Common mistakes

• Testing the model on the same data it was trained on. It then looks much better than it is.
• Normalizing with min and max from the whole dataset, test data included. That leaks information.
• Believing a more complex model is always better. It can easily overfit.

Concepts in this part

Practise data and core concepts in the app →

2. Linear regression and gradient descent

What is it about?

The simplest model that learns from data is a straight line: $\hat y = wx + b$. The model adjusts the weight $w$ and the intercept $b$ so the line fits the points as well as possible. Exactly the same idea, just with many more weights, is behind large neural networks.

Concepts and formulas

• Model: $\hat y = wx + b$. $w$ is the slope, $b$ is the intercept with the $y$-axis.
• Loss function: $L = \text{MSE} = \dfrac1n\sum (y_i - \hat y_i)^2$. The goal is to make $L$ as small as possible.
• Least squares gives the answer directly:

• Gradient descent: take a small step against the gradient, over and over:

where $\eta$ is the learning rate.

• For MSE, $\dfrac{\partial L}{\partial w} = -\dfrac2n\sum x_i\,(y_i - \hat y_i)$.
• Coefficient of determination $R^2$ between 0 and 1: how much of the variation in $y$ the model explains.

How to solve the problems

1. Compute the means $\bar x$ and $\bar y$.
2. Compute the deviations from the mean and insert into the formula for $w$, then $b$.
3. For one gradient step: compute the gradient, multiply by $\eta$ and subtract.

Example

The points $(1, 2)$, $(2, 4)$ and $(3, 5)$. Find $w$.

1. $\bar x = 2$, $\bar y = 11/3$.
2. $\sum (x-\bar x)(y-\bar y) = (-1)(-5/3) + 0 + (1)(4/3) = 3$ and $\sum (x-\bar x)^2 = 2$.
3. $w = 3/2 = 1.5$, and $b = 11/3 - 1.5\cdot 2 \approx 0.667$.

Common mistakes

• Too large a learning rate: the loss jumps back and forth and can blow up. Too small: it takes forever.
• Forgetting the minus sign in the update. We go downhill, towards lower loss.
• Reading a good fit as cause and effect.

Concepts in this part

Practise linear regression and gradient descent in the app →

3. Classification and evaluation

What is it about?

A classification model says "yes" or "no": is the part defective, is the tumour malignant, is the transaction fraud? To know whether the model is good, we must count where it hits and where it misses. And we must choose how strict it should be.

Concepts and formulas

• The confusion matrix counts four outcomes:

true positive (TP), false positive (FP), true negative (TN) and false negative (FN).

• Accuracy: $\dfrac{TP + TN}{N}$.
• Precision: $\dfrac{TP}{TP + FP}$. Of those the model said yes to, how many were right?
• Recall: $\dfrac{TP}{TP + FN}$. Of all the real positives, how many did the model find?
• F1 score: $F_1 = \dfrac{2PR}{P + R}$, a compromise between precision $P$ and recall $R$.
• Logistic regression gives a probability with the sigmoid function $\sigma(z) = \dfrac{1}{1 + e^{-z}}$. Above a threshold (often 0.5) the model says yes.
• k nearest neighbours (k-NN): a new point gets the class of the majority of the $k$ nearest points.

How to solve the problems

1. Set up the four numbers TP, FP, TN and FN.
2. Pick the right formula: precision looks at the positive answers, recall at the positive cases.
3. Think about what is worse: a false alarm or a missed case?

Example

A model finds 40 of 50 defective parts and raises a false alarm on 10 good parts. Precision and recall?

1. $TP = 40$, $FN = 10$, $FP = 10$.
2. Precision $= 40/50 = 0.8$. Recall $= 40/50 = 0.8$.

Common mistakes

• Trusting accuracy when the classes are imbalanced. If only 1 % is fraud, a model that always says "no" gets 99 %.
• Swapping precision and recall.
• Believing the threshold must be 0.5. A lower threshold gives higher recall but lower precision.

Concepts in this part

Practise classification and evaluation in the app →