LaTeX for Machine Learning

✅ Copy-ready LaTeX (MathJax) code for Machine Learning, organised from beginner to advanced, with where each formula is used in Machine Learning, AI and Data Science. 😻

New to LaTeX? Start with the LaTeX (MathJax) Basics for Greek letters, dots and spaces. For the maths behind these formulas, see Calculus for ML, Linear Algebra for ML and Probability for ML.

Just click the "copy code" button below 'markdown'!

Summary Overview

Level	Section	What you'll learn	Key commands
Beginner	1. Notation and Setup 1.1 Dataset Notation 1.2 Model and Parameters 1.3 Predictions with a Hat 1.4 Train, Validation and Test Sets 1.5 Feature Scaling 1.6 Arg Min: the Best Parameters	how ML papers write data and models	\mathbf{x}^{(i)}, \hat{y}, \arg\min
Beginner	2. Regression 2.1 Linear Regression Model 2.2 Regression Loss Functions 2.3 Cost Function and Gradient Descent 2.4 Polynomial Regression 2.5 Regularisation	predicting numbers	\sum, \lVert \rVert, \lambda
Intermediate	3. Classification 3.1 Sigmoid and Logistic Regression 3.2 Decision Boundary 3.3 Binary Cross-Entropy 3.4 Softmax Function 3.5 Categorical Cross-Entropy 3.6 k-Nearest Neighbours 3.7 Support Vector Machine 3.8 Decision Trees	predicting categories	\sigma, \log, \begin{cases}
Intermediate	4. Model Evaluation 4.1 Confusion Matrix 4.2 Accuracy, Precision and Recall 4.3 F1 Score 4.4 ROC Curve and AUC 4.5 Bias-Variance Decomposition 4.6 k-Fold Cross-Validation	measuring how good a model is	array, \hline, \frac{TP}{TP + FP}
Intermediate	5. Unsupervised Learning 5.1 k-Means Clustering 5.2 Principal Component Analysis 5.3 Gaussian Mixture Model	finding patterns without labels	\mu_k, C_k, \pi_k
Advanced	6. Neural Networks and Deep Learning 6.1 A Single Neuron 6.2 Layer Notation 6.3 Activation Functions 6.4 Backpropagation 6.5 Mini-Batch Gradient Descent 6.6 Adam Optimiser 6.7 Dropout 6.8 Convolution 6.9 Attention in Transformers	how deep learning models work	^{[l]}, \odot, \tanh
Advanced	7. Reinforcement Learning 7.1 Markov Decision Process 7.2 Return 7.3 Value Functions 7.4 Bellman Equation 7.5 Q-Learning 7.6 Policy Gradient	learning from rewards	\gamma, \pi(a \mid s), \max_{a'}

1. Notation and Setup (Beginner)

1.1 Dataset Notation

Training Data

Example:

$$\mathcal{D}=\{({\mathbf{x}}^{(i)},y^{(i)}){\}}_{i=1}^n,{\mathbf{x}}^{(i)}\in {\mathbb{R}}^d$$

There are $n$ training examples, each with $d$ features. The brackets in ${}^{(i)}$ show it is an example number, not a power.

Where it's used: describing the training data at the start of almost every ML paper and assignment.

$$
\mathcal{D} = \big\{ (\mathbf{x}^{(i)}, y^{(i)}) \big\}_{i=1}^{n}, \qquad \mathbf{x}^{(i)} \in \mathbb{R}^{d}
$$

1.2 Model and Parameters

Model: "f_{\theta}"

Example:

$$\hat{y}=f_{\mathit{\theta }}(\mathbf{x})\mathit{\theta }=(\mathbf{w},b)$$

$\mathit{\theta }$ collects all the parameters the model learns, such as the weights $\mathbf{w}$ and bias $b$.

Where it's used: writing any model in a general way.

$$
\hat{y} = f_{\boldsymbol{\theta}}(\mathbf{x}) \qquad \boldsymbol{\theta} = (\mathbf{w}, b)
$$

1.3 Predictions with a Hat

Prediction: "\hat{y}"

Example:

$$\hat{y}={\mathbf{w}}^{\mathrm{\top }}\mathbf{x}+b$$

$y$ is the true value, and $\hat{y}$ is the model's prediction.

Where it's used: marking a predicted or estimated value.

$$
\hat{y} = \mathbf{w}^{\top} \mathbf{x} + b
$$

1.4 Train, Validation and Test Sets

Data Splits

Example:

$$\mathcal{D}={\mathcal{D}}_{\text{train}}\cup {\mathcal{D}}_{\text{val}}\cup {\mathcal{D}}_{\text{test}}$$

The three sets do not overlap. The model learns from the training set, is tuned on the validation set, and is judged on the test set.

Where it's used: setting up any experiment fairly.

$$
\mathcal{D} = \mathcal{D}_{\text{train}} \cup \mathcal{D}_{\text{val}} \cup \mathcal{D}_{\text{test}}
$$

1.5 Feature Scaling

Min-Max Scaling and Standardisation

Example:

$$x^{\prime }=\frac{x-min(x)}{max(x)-min(x)}z=\frac{x-\mu }{\sigma }$$

Min-max scaling puts values between 0 and 1. Standardisation gives mean 0 and standard deviation 1.

Where it's used: preparing features so that gradient descent and distance-based models work well.

$$
x' = \frac{x - \min(x)}{\max(x) - \min(x)} \qquad z = \frac{x - \mu}{\sigma}
$$

1.6 Arg Min: the Best Parameters

Arg Min: "\underset{\theta}{\arg\min}"

Example:

$${\mathit{\theta }}^{\ast }=\underset{\mathit{\theta }}{\arg min}J(\mathit{\theta })$$

Where it's used: describing training as finding the parameters ${\mathit{\theta }}^{\ast }$ that give the smallest cost.

$$
\boldsymbol{\theta}^{*} = \underset{\boldsymbol{\theta}}{\arg\min} \; J(\boldsymbol{\theta})
$$

2. Regression (Beginner)

2.1 Linear Regression Model

Linear Regression

Example:

$$\hat{y}=b+w_1{x}_1+w_2{x}_2+\cdots +w_d{x}_d={\mathbf{w}}^{\mathrm{\top }}\mathbf{x}+b$$

Where it's used: predicting a number, such as a house price, from several features.

$$
\hat{y} = b + w_1 x_1 + w_2 x_2 + \cdots + w_d x_d = \mathbf{w}^{\top} \mathbf{x} + b
$$

2.2 Regression Loss Functions

Loss Functions: "\text{MSE}"

Example:

$$\text{MSE}=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2\text{RMSE}=\sqrt{\text{MSE}}\text{MAE}=\frac{1}{n}\sum _{i=1}^n|y_i-{\hat{y}}_i|$$

\text{} writes normal words inside maths. MAE is less affected by outliers than MSE.

Where it's used: training and evaluating regression models.

$$
\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \qquad \text{RMSE} = \sqrt{\text{MSE}} \qquad \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} \lvert y_i - \hat{y}_i \rvert
$$

2.3 Cost Function and Gradient Descent

Gradient Descent for Linear Regression

Example:

$$J(\mathbf{w},b)=\frac{1}{2n}\sum _{i=1}^n({\hat{y}}_i-y_i{)}^2\frac{\partial J}{\partial w_j}=\frac{1}{n}\sum _{i=1}^n({\hat{y}}_i-y_i)x_{ij}$$$$w_j:=w_j-\alpha \frac{\partial J}{\partial w_j}$$

The $\frac{1}{2}$ is there so it cancels when you differentiate. $\alpha$ is the learning rate.

Where it's used: training linear regression step by step.

$$
J(\mathbf{w}, b) = \frac{1}{2n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2 \qquad \frac{\partial J}{\partial w_j} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i) \, x_{ij}
$$

$$
w_j := w_j - \alpha \frac{\partial J}{\partial w_j}
$$

2.4 Polynomial Regression

Polynomial Regression

Example:

$$\hat{y}=w_0+w_{1}x+w_2{x}^2+\cdots +w_p{x}^p$$

Where it's used: fitting curves instead of straight lines.

$$
\hat{y} = w_0 + w_1 x + w_2 x^2 + \cdots + w_p x^p
$$

2.5 Regularisation

Ridge (L2) and Lasso (L1)

Example:

$$J_{\text{Ridge}}=\text{MSE}+\lambda \Vert \mathbf{w}{\Vert }_2^2{J}_{\text{Lasso}}=\text{MSE}+\lambda \Vert \mathbf{w}{\Vert }_1$$

$\lambda$ controls how strongly large weights are penalised. Lasso can set some weights to exactly 0.

Where it's used: preventing overfitting and selecting features.

$$
J_{\text{Ridge}} = \text{MSE} + \lambda \lVert \mathbf{w} \rVert_2^2 \qquad J_{\text{Lasso}} = \text{MSE} + \lambda \lVert \mathbf{w} \rVert_1
$$

3. Classification (Intermediate)

3.1 Sigmoid and Logistic Regression

Sigmoid: "\sigma(z)"

Example:

$$\sigma (z)=\frac{1}{1+e^{-z}}P(y=1\mid \mathbf{x})=\sigma ({\mathbf{w}}^{\mathrm{\top }}\mathbf{x}+b)$$

The sigmoid turns any number into a probability between 0 and 1.

Where it's used: binary classification, such as spam or not spam.

$$
\sigma(z) = \frac{1}{1 + e^{-z}} \qquad P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w}^{\top} \mathbf{x} + b)
$$

3.2 Decision Boundary

Decision Rule: "cases"

Example:

$$\hat{y}=\{\begin{array}{ll}1& \text{if }\sigma (z)\ge 0.5\\ 0& \text{otherwise}\end{array}\sigma (z)\ge 0.5⟺z\ge 0$$

Where it's used: turning predicted probabilities into class labels.

$$
\hat{y} = \begin{cases} 1 & \text{if } \sigma(z) \ge 0.5 \\ 0 & \text{otherwise} \end{cases} \qquad \sigma(z) \ge 0.5 \iff z \ge 0
$$

3.3 Binary Cross-Entropy

Log Loss

Example:

$$L=-\frac{1}{n}\sum _{i=1}^n[y_i\log {\hat{y}}_i+(1-y_i)\log (1-{\hat{y}}_i)]$$

Where it's used: the loss function for logistic regression and binary classifiers.

$$
L = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i) \Big]
$$

3.4 Softmax Function

Softmax: "\frac{e^{z_i}}{\sum e^{z_j}}"

Example:

$$\mathrm{softmax}(\mathbf{z}{)}_i=\frac{e^{z_i}}{\sum _{j=1}^K{e}^{z_j}}$$

The $K$ outputs are all positive and add up to 1.

Where it's used: multi-class classification, such as recognising digits 0 to 9.

$$
\operatorname{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}
$$

3.5 Categorical Cross-Entropy

Cross-Entropy: "-\sum y \log \hat{y}"

Example:

$$L=-\sum _{k=1}^K{y}_k\log ({\hat{y}}_k)$$

$y_k$ is 1 for the correct class and 0 for the others.

Where it's used: the loss function for multi-class classification and language models.

$$
L = -\sum_{k=1}^{K} y_k \log(\hat{y}_k)
$$

3.6 k-Nearest Neighbours

k-NN and Euclidean Distance

Example:

$$d(\mathbf{x},{\mathbf{x}}^{\prime })=\Vert \mathbf{x}-{\mathbf{x}}^{\prime }{\Vert }_2=\sqrt{\sum _{j=1}^d(x_j-x_j^{\prime }{)}^2}$$

The prediction is the most common label among the $k$ closest training examples.

Where it's used: simple classifiers and recommender systems.

$$
d(\mathbf{x}, \mathbf{x}') = \lVert \mathbf{x} - \mathbf{x}' \rVert_2 = \sqrt{\sum_{j=1}^{d} (x_j - x'_j)^2}
$$

3.7 Support Vector Machine

Hard-Margin SVM

Example:

$$\begin{array}{rl}& \underset{\mathbf{w},b}{min}\frac{1}{2}\Vert \mathbf{w}{\Vert }^2\\ & \text{subject to }{y}^{(i)}({\mathbf{w}}^{\mathrm{\top }}{\mathbf{x}}^{(i)}+b)\ge 1\text{for all }i\end{array}$$

Here the labels are $y^{(i)}\in \{-1,1\}$, and the margin between the classes is $\frac{2}{\Vert \mathbf{w}\Vert }$.

Where it's used: classifiers that separate two classes with the widest possible gap.

$$
\begin{aligned}
& \underset{\mathbf{w}, b}{\min} \; \frac{1}{2} \lVert \mathbf{w} \rVert^2 \\
& \text{subject to } \; y^{(i)} \big( \mathbf{w}^{\top} \mathbf{x}^{(i)} + b \big) \ge 1 \quad \text{for all } i
\end{aligned}
$$

3.8 Decision Trees

Gini Impurity and Information Gain

Example:

$$G=1-\sum _{k=1}^K{p}_k^{2}H=-\sum _{k=1}^K{p}_k{\log }_2{p}_k\text{IG}=H(\text{parent})-\sum _c\frac{n_c}{n}H(c)$$

$p_k$ is the share of examples in class $k$. The tree chooses the split with the highest information gain.

Where it's used: decision trees, random forests and gradient boosting.

$$
G = 1 - \sum_{k=1}^{K} p_k^2 \qquad H = -\sum_{k=1}^{K} p_k \log_2 p_k \qquad \text{IG} = H(\text{parent}) - \sum_{c} \frac{n_c}{n} \, H(c)
$$

4. Model Evaluation (Intermediate)

4.1 Confusion Matrix

Confusion Matrix: "array" with "\hline"

Example:

$$\begin{array}{ccc}& \text{Predicted }+& \text{Predicted }-\\ \text{Actual }+& TP& FN\\ \text{Actual }-& FP& TN\end{array}$$

{c|cc} adds a vertical line after the first column, and \hline adds a horizontal line.

Where it's used: seeing exactly which mistakes a classifier makes.

$$
\begin{array}{c|cc} & \text{Predicted } + & \text{Predicted } - \\ \hline \text{Actual } + & TP & FN \\ \text{Actual } - & FP & TN \end{array}
$$

4.2 Accuracy, Precision and Recall

Accuracy, Precision and Recall

Example:

$$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\text{Precision}=\frac{TP}{TP+FP}\text{Recall}=\frac{TP}{TP+FN}$$$$TP=40,FP=10,FN=20,TN=30\Rightarrow \text{Accuracy}=0.70,\text{Precision}=0.80,\text{Recall}\approx 0.67$$

Precision asks how many predicted positives were right. Recall asks how many real positives were found.

Where it's used: evaluating classifiers, especially when one class is rare.

$$
\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} \qquad \text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}
$$

$$
TP = 40, \; FP = 10, \; FN = 20, \; TN = 30 \;\Rightarrow\; \text{Accuracy} = 0.70, \; \text{Precision} = 0.80, \; \text{Recall} \approx 0.67
$$

4.3 F1 Score

F1 Score

Example:

$$F_1=2\cdot \frac{\text{Precision}\cdot \text{Recall}}{\text{Precision}+\text{Recall}}{F}_1=2\cdot \frac{0.80\cdot \frac{2}{3}}{0.80+\frac{2}{3}}\approx 0.73$$

The F1 score balances precision and recall in one number.

Where it's used: comparing classifiers on imbalanced data, such as fraud detection.

$$
F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \qquad F_1 = 2 \cdot \frac{0.80 \cdot \frac{2}{3}}{0.80 + \frac{2}{3}} \approx 0.73
$$

4.4 ROC Curve and AUC

ROC and AUC

Example:

$$\text{TPR}=\frac{TP}{TP+FN}\text{FPR}=\frac{FP}{FP+TN}\text{AUC}={\int }_0^1\text{TPR}d(\text{FPR})$$

The ROC curve plots TPR against FPR for every threshold. An AUC of 1 is perfect, and 0.5 is no better than guessing.

Where it's used: comparing classifiers without choosing a single threshold.

$$
\text{TPR} = \frac{TP}{TP + FN} \qquad \text{FPR} = \frac{FP}{FP + TN} \qquad \text{AUC} = \int_{0}^{1} \text{TPR} \; d(\text{FPR})
$$

4.5 Bias-Variance Decomposition

Bias and Variance

Example:

$$\mathbb{E}[(y-\hat{f}(\mathbf{x}){)}^2]=\mathrm{Bias}[\hat{f}(\mathbf{x}){]}^2+\mathrm{Var}[\hat{f}(\mathbf{x})]+{\sigma }^2$$

${\sigma }^2$ is the noise in the data, which no model can remove.

Where it's used: understanding underfitting (high bias) and overfitting (high variance).

$$
\mathbb{E}\Big[ \big( y - \hat{f}(\mathbf{x}) \big)^2 \Big] = \operatorname{Bias}\big[ \hat{f}(\mathbf{x}) \big]^2 + \operatorname{Var}\big[ \hat{f}(\mathbf{x}) \big] + \sigma^2
$$

4.6 k-Fold Cross-Validation

Cross-Validation Score

Example:

$$\text{CV}=\frac{1}{k}\sum _{i=1}^k{E}_i$$

Split the data into $k$ parts, test on each part once, and average the $k$ errors $E_i$.

Where it's used: choosing models and hyperparameters fairly with limited data.

$$
\text{CV} = \frac{1}{k} \sum_{i=1}^{k} E_i
$$

5. Unsupervised Learning (Intermediate)

5.1 k-Means Clustering

k-Means Objective

Example:

$$J=\sum _{k=1}^K\sum _{\mathbf{x}\in C_k}\Vert \mathbf{x}-{\mathit{\mu }}_k{\Vert }^2{\mathit{\mu }}_k=\frac{1}{|C_k|}\sum _{\mathbf{x}\in C_k}\mathbf{x}$$

Each point joins the closest centre ${\mathit{\mu }}_k$, then each centre moves to the mean of its cluster $C_k$.

Where it's used: customer segmentation and grouping similar data.

$$
J = \sum_{k=1}^{K} \sum_{\mathbf{x} \in C_k} \lVert \mathbf{x} - \boldsymbol{\mu}_k \rVert^2 \qquad \boldsymbol{\mu}_k = \frac{1}{\lvert C_k \rvert} \sum_{\mathbf{x} \in C_k} \mathbf{x}
$$

5.2 Principal Component Analysis

PCA Objective

Example:

$$\underset{\mathbf{w}}{max}{\mathbf{w}}^{\mathrm{\top }}\mathrm{\Sigma }\mathbf{w}\text{subject to}\Vert \mathbf{w}\Vert =1$$

The best direction $\mathbf{w}$ is the eigenvector of the covariance matrix $\mathrm{\Sigma }$ with the largest eigenvalue.

Where it's used: reducing many features to a few, and visualising high-dimensional data.

$$
\underset{\mathbf{w}}{\max} \; \mathbf{w}^{\top} \Sigma \, \mathbf{w} \quad \text{subject to} \quad \lVert \mathbf{w} \rVert = 1
$$

5.3 Gaussian Mixture Model

Gaussian Mixture

Example:

$$p(\mathbf{x})=\sum _{k=1}^K{\pi }_k\mathcal{N}(\mathbf{x}\mid {\mathit{\mu }}_k,{\mathrm{\Sigma }}_k),\sum _{k=1}^K{\pi }_k=1$$

Here ${\pi }_k$ are the mixing weights, not the number pi.

Where it's used: soft clustering and density estimation.

$$
p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \, \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \Sigma_k), \qquad \sum_{k=1}^{K} \pi_k = 1
$$

6. Neural Networks and Deep Learning (Advanced)

6.1 A Single Neuron

Neuron

Example:

$$a=g({\mathbf{w}}^{\mathrm{\top }}\mathbf{x}+b)$$

$g$ is the activation function and $a$ is the neuron's output, called its activation.

Where it's used: the building block of every neural network.

$$
a = g(\mathbf{w}^{\top} \mathbf{x} + b)
$$

6.2 Layer Notation

Layer Notation: "^{[l]}"

Example:

$${\mathbf{z}}^{[l]}=W^{[l]}{\mathbf{a}}^{[l-1]}+{\mathbf{b}}^{[l]}{\mathbf{a}}^{[l]}=g^{[l]}({\mathbf{z}}^{[l]})$$

Square brackets ${}^{[l]}$ show the layer number, and round brackets ${}^{(i)}$ show the example number.

Where it's used: describing a network layer by layer.

$$
\mathbf{z}^{[l]} = W^{[l]} \mathbf{a}^{[l-1]} + \mathbf{b}^{[l]} \qquad \mathbf{a}^{[l]} = g^{[l]}\big( \mathbf{z}^{[l]} \big)
$$

6.3 Activation Functions

Common Activation Functions

Example:

$$\begin{array}{rl}\mathrm{ReLU}(z)& =max(0,z)=\{\begin{array}{ll}z& \text{if }z>0\\ 0& \text{otherwise}\end{array}\\ \mathrm{LeakyReLU}(z)& =max(0.01z,z)\\ \tanh (z)& =\frac{e^z-e^{-z}}{e^z+e^{-z}}\end{array}$$

The cases environment works for any piecewise function.

Where it's used: adding non-linearity, so networks can learn complex patterns.

$$
\begin{aligned}
\operatorname{ReLU}(z) &= \max(0, z) = \begin{cases} z & \text{if } z > 0 \\ 0 & \text{otherwise} \end{cases} \\
\operatorname{LeakyReLU}(z) &= \max(0.01 z, \, z) \\
\tanh(z) &= \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}
\end{aligned}
$$

6.4 Backpropagation

Backpropagation Equations

Example:

$${\mathit{\delta }}^{[l]}=(W^{[l+1]}{)}^{\mathrm{\top }}{\mathit{\delta }}^{[l+1]}\odot g^{\prime }({\mathbf{z}}^{[l]})\frac{\partial L}{\partial W^{[l]}}={\mathit{\delta }}^{[l]}({\mathbf{a}}^{[l-1]}{)}^{\mathrm{\top }}$$

${\mathit{\delta }}^{[l]}$ is the error at layer $l$. It is passed backwards one layer at a time using the chain rule.

Where it's used: computing the gradients that train a neural network.

$$
\boldsymbol{\delta}^{[l]} = \big( W^{[l+1]} \big)^{\top} \boldsymbol{\delta}^{[l+1]} \odot g'\big( \mathbf{z}^{[l]} \big) \qquad \frac{\partial L}{\partial W^{[l]}} = \boldsymbol{\delta}^{[l]} \big( \mathbf{a}^{[l-1]} \big)^{\top}
$$

6.5 Mini-Batch Gradient Descent

Mini-Batch Update

Example:

$$\mathit{\theta }:=\mathit{\theta }-\alpha \frac{1}{|B|}\sum _{i\in B}{\mathrm{\nabla }}_{\mathit{\theta }}{L}_i(\mathit{\theta })$$

$B$ is a small random batch of examples. With one example per batch, this is stochastic gradient descent (SGD).

Where it's used: training large models efficiently.

$$
\boldsymbol{\theta} := \boldsymbol{\theta} - \alpha \, \frac{1}{\lvert B \rvert} \sum_{i \in B} \nabla_{\boldsymbol{\theta}} L_i(\boldsymbol{\theta})
$$

6.6 Adam Optimiser

Adam

Example:

$$\begin{array}{rl}{\mathbf{m}}_t& ={\beta }_1{\mathbf{m}}_{t-1}+(1-{\beta }_1){\mathbf{g}}_t\\ {\mathbf{v}}_t& ={\beta }_2{\mathbf{v}}_{t-1}+(1-{\beta }_2){\mathbf{g}}_t^2\\ {\hat{\mathbf{m}}}_t& =\frac{{\mathbf{m}}_t}{1-{\beta }_1^t},{\hat{\mathbf{v}}}_t=\frac{{\mathbf{v}}_t}{1-{\beta }_2^t}\\ {\mathit{\theta }}_t& ={\mathit{\theta }}_{t-1}-\alpha \frac{{\hat{\mathbf{m}}}_t}{\sqrt{{\hat{\mathbf{v}}}_t}+ϵ}\end{array}$$

${\mathbf{g}}_t$ is the gradient at step $t$. Common defaults are ${\beta }_1=0.9$ and ${\beta }_2=0.999$.

Where it's used: the default optimiser for training most deep learning models.

$$
\begin{aligned}
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1 - \beta_1) \, \mathbf{g}_t \\
\mathbf{v}_t &= \beta_2 \mathbf{v}_{t-1} + (1 - \beta_2) \, \mathbf{g}_t^2 \\
\hat{\mathbf{m}}_t &= \frac{\mathbf{m}_t}{1 - \beta_1^t}, \quad \hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1 - \beta_2^t} \\
\boldsymbol{\theta}_t &= \boldsymbol{\theta}_{t-1} - \alpha \, \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t} + \epsilon}
\end{aligned}
$$

6.7 Dropout

Inverted Dropout

Example:

$$\tilde{\mathbf{a}}=\frac{\mathbf{m}\odot \mathbf{a}}{1-p},m_j\sim \mathrm{Bernoulli}(1-p)$$

Each activation is switched off with probability $p$ during training, and the rest are scaled up to keep the same average.

Where it's used: preventing overfitting in neural networks.

$$
\tilde{\mathbf{a}} = \frac{\mathbf{m} \odot \mathbf{a}}{1 - p}, \qquad m_j \sim \operatorname{Bernoulli}(1 - p)
$$

6.8 Convolution

Convolution and Output Size

Example:

$$(I\ast K)(i,j)=\sum _m\sum _{n}I(i+m,j+n)K(m,n)n_{\text{out}}=\lfloor \frac{n+2p-f}{s}\rfloor +1$$

$I$ is the image, $K$ is the filter of size $f$, $p$ is the padding and $s$ is the stride. For example, $n=32$, $f=3$, $p=1$, $s=1$ gives an output of 32.

Where it's used: convolutional neural networks for images.

$$
(I * K)(i, j) = \sum_{m} \sum_{n} I(i + m, \, j + n) \, K(m, n) \qquad n_{\text{out}} = \left\lfloor \frac{n + 2p - f}{s} \right\rfloor + 1
$$

6.9 Attention in Transformers

Scaled Dot-Product Attention

Example:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}\right)V$$

$Q$, $K$ and $V$ are the query, key and value matrices, and $d_k$ is the size of each key.

Where it's used: transformer models such as GPT and Claude.

$$
\operatorname{Attention}(Q, K, V) = \operatorname{softmax} \left( \frac{Q K^{\top}}{\sqrt{d_k}} \right) V
$$

7. Reinforcement Learning (Advanced)

7.1 Markov Decision Process

MDP

Example:

$$(\mathcal{S},\mathcal{A},P,R,\gamma )P(s^{\prime }\mid s,a)$$

$\mathcal{S}$ is the set of states, $\mathcal{A}$ the actions, $P$ the transition probabilities, $R$ the rewards and $\gamma$ the discount factor. It builds on Markov chains.

Where it's used: describing any reinforcement learning problem, from games to robotics.

$$
(\mathcal{S}, \mathcal{A}, P, R, \gamma) \qquad P(s' \mid s, a)
$$

7.2 Return

Discounted Return

Example:

$$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+\cdots =\sum _{k=0}^{\mathrm{\infty }}{\gamma }^k{R}_{t+k+1}$$

With $0\le \gamma <1$, rewards further in the future count for less. This is a geometric series.

Where it's used: measuring the total reward an agent collects.

$$
G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots = \sum_{k=0}^{\infty} \gamma^{k} R_{t+k+1}
$$

7.3 Value Functions

State and Action Values

Example:

$$V^{\pi }(s)={\mathbb{E}}_{\pi }[G_t\mid S_t=s]Q^{\pi }(s,a)={\mathbb{E}}_{\pi }[G_t\mid S_t=s,A_t=a]$$

$\pi$ here is the agent's policy, not the number pi.

Where it's used: judging how good a state or an action is.

$$
V^{\pi}(s) = \mathbb{E}_{\pi}[G_t \mid S_t = s] \qquad Q^{\pi}(s, a) = \mathbb{E}_{\pi}[G_t \mid S_t = s, A_t = a]
$$

7.4 Bellman Equation

Bellman Equations

Example:

$$V^{\pi }(s)=\sum _a\pi (a\mid s)\sum _{s^{\prime }}P(s^{\prime }\mid s,a)[R(s,a,s^{\prime })+\gamma V^{\pi }(s^{\prime })]$$$$V^{\ast }(s)=\underset{a}{max}\sum _{s^{\prime }}P(s^{\prime }\mid s,a)[R(s,a,s^{\prime })+\gamma V^{\ast }(s^{\prime })]$$

The first equation evaluates a policy. The second, the Bellman optimality equation, describes the best possible policy.

Where it's used: value iteration, policy iteration and Q-learning.

$$
V^{\pi}(s) = \sum_{a} \pi(a \mid s) \sum_{s'} P(s' \mid s, a) \big[ R(s, a, s') + \gamma V^{\pi}(s') \big]
$$

$$
V^{*}(s) = \max_{a} \sum_{s'} P(s' \mid s, a) \big[ R(s, a, s') + \gamma V^{*}(s') \big]
$$

7.5 Q-Learning

Q-Learning Update

Example:

$$Q(s,a)\leftarrow Q(s,a)+\alpha [r+\gamma \underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime })-Q(s,a)]$$

\leftarrow is the arrow for an update step.

Where it's used: learning to act without knowing the transition probabilities, such as game-playing agents.

$$
Q(s, a) \leftarrow Q(s, a) + \alpha \Big[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \Big]
$$

7.6 Policy Gradient

REINFORCE

Example:

$${\mathrm{\nabla }}_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}[{\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(a\mid s)G_t]$$

Where it's used: training policies directly, and fine-tuning large language models with reinforcement learning from human feedback (RLHF).

$$
\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi_{\theta}} \big[ \nabla_{\theta} \log \pi_{\theta}(a \mid s) \, G_t \big]
$$