LaTeX for Linear Algebra and Matrices

✅ Copy-ready LaTeX (MathJax) code for Linear Algebra and Matrices, organised from beginner to advanced, with where each formula is used in maths and in Machine Learning, AI and Data Science. 😻

New to LaTeX? Start with the LaTeX (MathJax) Basics for Greek letters, dots and spaces.

Just click the "copy code" button below 'markdown'!

Summary Overview

Level	Section	What you'll learn	Key commands
Beginner	1. Vectors 1.1 Vector Notation 1.2 Addition and Scalar Multiplication 1.3 Dot Product 1.4 Norms 1.5 Unit Vectors 1.6 Angle and Cosine Similarity 1.7 Cross Product	lists of numbers with direction	\mathbf{v}, \cdot, \lVert \rVert
Beginner	2. Matrices 2.1 A General m × n Matrix 2.2 Special Matrices 2.3 Addition and Scalar Multiplication 2.4 Transpose 2.5 Matrix Multiplication 2.6 Systems of Equations as a Matrix Equation 2.7 Outer Product and Element-wise Product 2.8 Frobenius Norm	grids of numbers and how to combine them	bmatrix, ^{\top}, \odot
Intermediate	3. Determinants and Inverses 3.1 Determinant of a 2 × 2 Matrix 3.2 Determinant of a 3 × 3 Matrix 3.3 Inverse of a 2 × 2 Matrix 3.4 Augmented Matrix 3.5 Row Reduction 3.6 Reduced Row Echelon Form 3.7 Rank and Trace	solving systems of equations	vmatrix, ^{-1}, \xrightarrow{}
Intermediate	4. Vector Spaces 4.1 Linear Combination and Span 4.2 Linear Independence 4.3 Basis and Dimension 4.4 Null Space, Column Space and the Rank-Nullity Theorem 4.5 Orthogonality and Projection 4.6 Linear Transformations	span, independence and dimension	\operatorname{span}, \dim, \operatorname{proj}
Advanced	5. Eigenvalues and Decompositions 5.1 Eigenvalues and Eigenvectors 5.2 Characteristic Equation 5.3 Diagonalisation 5.4 Orthogonal Matrices and the Spectral Theorem 5.5 Singular Value Decomposition 5.6 Positive Definite Matrices 5.7 Gram-Schmidt and QR Decomposition	breaking matrices into simpler parts	\lambda, \Lambda, \Sigma
Applied	6. Linear Algebra for Machine Learning 6.1 Dataset as a Matrix 6.2 Matrix Equation of a Neural Network Layer 6.3 Linear Regression and the Normal Equation 6.4 Covariance Matrix 6.5 Principal Component Analysis 6.6 Attention in Transformers	how models use matrices	\mathbf{x}^{(i)}, W\mathbf{x}, QK^{\top}

1. Vectors (Beginner)

1.1 Vector Notation

Column Vector: "\mathbf{x}" and "\mathbb{R}^{n}"

Example:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right]\in {\mathbb{R}}^n\overrightarrow{v}=(v_1,v_2)$$

\mathbf{} makes vectors bold, the most common style in Machine Learning. \vec{} draws an arrow, the common style in physics.

Where it's used: representing one data point as a list of $n$ features.

$$
\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix} \in \mathbb{R}^{n} \qquad \vec{v} = (v_1, v_2)
$$

1.2 Addition and Scalar Multiplication

Vector Addition and Scaling

Example:

$$\mathbf{u}+\mathbf{v}=\left[\begin{array}{c}{u}_1+v_1\\ u_2+v_2\end{array}\right]c\mathbf{v}=\left[\begin{array}{c}c{v}_1\\ c{v}_2\end{array}\right]$$

Where it's used: combining and rescaling vectors, such as updating weights in gradient descent.

$$
\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \end{bmatrix} \qquad c \, \mathbf{v} = \begin{bmatrix} c \, v_1 \\ c \, v_2 \end{bmatrix}
$$

1.3 Dot Product

Dot Product: "\cdot" and "^{\top}"

Example:

$$\mathbf{u}\cdot \mathbf{v}={\mathbf{u}}^{\mathrm{\top }}\mathbf{v}=\sum _{i=1}^n{u}_i{v}_i$$$$\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]\cdot \left[\begin{array}{c}4\\ 5\\ 6\end{array}\right]=1(4)+2(5)+3(6)=32$$

Where it's used: the weighted sum in linear regression and inside every neuron of a neural network.

$$
\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^{\top} \mathbf{v} = \sum_{i=1}^{n} u_i v_i
$$

$$
\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1(4) + 2(5) + 3(6) = 32
$$

1.4 Norms

Norm: "\lVert \rVert"

Example:

$$\Vert \mathbf{x}{\Vert }_2=\sqrt{\sum _{i=1}^n{x}_i^2}\Vert \mathbf{x}{\Vert }_1=\sum _{i=1}^n|x_i|$$$${\Vert \left[\begin{array}{c}3\\ 4\end{array}\right]\Vert }_2=\sqrt{3^2+4^2}=5$$

\lVert \rVert gives the double bars of a norm with correct spacing.

Where it's used: measuring the length of a vector or the distance between data points, and in L2 (Ridge) and L1 (Lasso) regularisation.

$$
\lVert \mathbf{x} \rVert_2 = \sqrt{\sum_{i=1}^{n} x_i^2} \qquad \lVert \mathbf{x} \rVert_1 = \sum_{i=1}^{n} \lvert x_i \rvert
$$

$$
\left\lVert \begin{bmatrix} 3 \\ 4 \end{bmatrix} \right\rVert_2 = \sqrt{3^2 + 4^2} = 5
$$

1.5 Unit Vectors

Unit Vector

Example:

$$\hat{\mathbf{u}}=\frac{\mathbf{u}}{\Vert \mathbf{u}\Vert }$$

A unit vector has length 1 and keeps the same direction.

Where it's used: normalising vectors before comparing them.

$$
\hat{\mathbf{u}} = \frac{\mathbf{u}}{\lVert \mathbf{u} \rVert}
$$

1.6 Angle and Cosine Similarity

Cosine Similarity

Example:

$$\cos \theta =\frac{\mathbf{u}\cdot \mathbf{v}}{\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert }$$

The result is 1 when the vectors point the same way, 0 when they are perpendicular, and -1 when they point in opposite directions.

Where it's used: comparing word embeddings and documents in NLP, and recommender systems.

$$
\cos \theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert \, \lVert \mathbf{v} \rVert}
$$

1.7 Cross Product

Cross Product in 3D

Example:

$$\mathbf{u}\times \mathbf{v}=\left|\begin{array}{ccc}\mathbf{i}& \mathbf{j}& \mathbf{k}\\ u_1& u_2& u_3\\ v_1& v_2& v_3\end{array}\right|$$

The result is a vector perpendicular to both $\mathbf{u}$ and $\mathbf{v}$.

Where it's used: physics, 3D graphics and computer vision.

$$
\mathbf{u} \times \mathbf{v} = \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ u_1 & u_2 & u_3 \\ v_1 & v_2 & v_3 \end{vmatrix}
$$

2. Matrices (Beginner)

2.1 A General m × n Matrix

General Matrix: "\cdots", "\vdots" and "\ddots" together

Example:

$$A=\left[\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1n}\\ a_{21}& a_{22}& \cdots & a_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mn}\end{array}\right]\in {\mathbb{R}}^{m\times n}$$

$a_{ij}$ is the entry in row $i$ and column $j$. \times writes the size $m\times n$.

Where it's used: any table of numbers, such as a dataset where each row is a sample and each column is a feature.

$$
A = \begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} \in \mathbb{R}^{m \times n}
$$

2.2 Special Matrices

Identity, Diagonal and Symmetric Matrices

Example:

$$I=\left[\begin{array}{cccc}1& 0& \cdots & 0\\ 0& 1& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & 1\end{array}\right]D=\left[\begin{array}{ccc}{d}_1& & \\ & \ddots & \\ & & d_n\end{array}\right]A=A^{\mathrm{\top }}$$

Leave a cell empty to hide the zeros, as in the diagonal matrix $D$. A matrix is symmetric when $A=A^{\mathrm{\top }}$.

Where it's used: the identity in inverses, diagonal matrices in eigen decompositions, and symmetric covariance matrices.

$$
I = \begin{bmatrix} 1 & 0 & \cdots & 0 \\ 0 & 1 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1 \end{bmatrix} \qquad D = \begin{bmatrix} d_1 & & \\ & \ddots & \\ & & d_n \end{bmatrix} \qquad A = A^{\top}
$$

2.3 Addition and Scalar Multiplication

Matrix Addition and Scaling

Example:

$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]+\left[\begin{array}{cc}5& 6\\ 7& 8\end{array}\right]=\left[\begin{array}{cc}6& 8\\ 10& 12\end{array}\right]2\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]=\left[\begin{array}{cc}2& 4\\ 6& 8\end{array}\right]$$

Add matching entries. Both matrices must be the same size.

Where it's used: updating weight matrices and combining datasets.

$$
\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix} \qquad 2 \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} = \begin{bmatrix} 2 & 4 \\ 6 & 8 \end{bmatrix}
$$

2.4 Transpose

Transpose: "^{\top}"

Example:

$$A=\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]\Rightarrow A^{\mathrm{\top }}=\left[\begin{array}{cc}1& 4\\ 2& 5\\ 3& 6\end{array}\right]$$

Rows become columns, so a $2\times 3$ matrix becomes $3\times 2$. ^{\top} is the standard symbol, and some books write ^{T}.

Where it's used: the normal equation, covariance matrices and backpropagation.

$$
A = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix} \quad \Rightarrow \quad A^{\top} = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}
$$

2.5 Matrix Multiplication

Matrix Multiplication

Example:

$$(AB{)}_{ij}=\sum _{k=1}^n{a}_{ik}{b}_{kj}$$$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\left[\begin{array}{cc}5& 6\\ 7& 8\end{array}\right]=\left[\begin{array}{cc}1(5)+2(7)& 1(6)+2(8)\\ 3(5)+4(7)& 3(6)+4(8)\end{array}\right]=\left[\begin{array}{cc}19& 22\\ 43& 50\end{array}\right]$$

An $m\times n$ matrix times an $n\times p$ matrix gives an $m\times p$ matrix. The order matters: $AB$ is usually not the same as $BA$.

Where it's used: every layer of a neural network, and transforming data.

$$
(AB)_{ij} = \sum_{k=1}^{n} a_{ik} \, b_{kj}
$$

$$
\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1(5) + 2(7) & 1(6) + 2(8) \\ 3(5) + 4(7) & 3(6) + 4(8) \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}
$$

2.6 Systems of Equations as a Matrix Equation

Matrix Form: "A\mathbf{x} = \mathbf{b}"

Example:

$$\begin{array}{rl}2x+y& =5\\ x-y& =1\end{array}⟺\left[\begin{array}{cc}2& 1\\ 1& -1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{c}5\\ 1\end{array}\right]$$

Where it's used: writing any system of linear equations compactly as $A\mathbf{x}=\mathbf{b}$.

$$
\begin{aligned} 2x + y &= 5 \\ x - y &= 1 \end{aligned} \quad \Longleftrightarrow \quad \begin{bmatrix} 2 & 1 \\ 1 & -1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 5 \\ 1 \end{bmatrix}
$$

2.7 Outer Product and Element-wise Product

Outer Product and Hadamard Product: "\odot"

Example:

$$\mathbf{u}{\mathbf{v}}^{\mathrm{\top }}=\left[\begin{array}{c}{u}_1\\ u_2\end{array}\right]\left[\begin{array}{cc}{v}_1& v_2\end{array}\right]=\left[\begin{array}{cc}{u}_1{v}_1& u_1{v}_2\\ u_2{v}_1& u_2{v}_2\end{array}\right]$$$$\left[\begin{array}{cc}1& 2\\ 3& 4\end{array}\right]\odot \left[\begin{array}{cc}5& 6\\ 7& 8\end{array}\right]=\left[\begin{array}{cc}5& 12\\ 21& 32\end{array}\right]$$

The outer product of two vectors gives a matrix, while the dot product gives a single number. \odot multiplies matching entries, which is called the Hadamard product.

Where it's used: computing the gradient of a weight matrix, applying masks, and the gates in LSTM and GRU networks.

$$
\mathbf{u} \mathbf{v}^{\top} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} \begin{bmatrix} v_1 & v_2 \end{bmatrix} = \begin{bmatrix} u_1 v_1 & u_1 v_2 \\ u_2 v_1 & u_2 v_2 \end{bmatrix}
$$

$$
\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \odot \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 5 & 12 \\ 21 & 32 \end{bmatrix}
$$

2.8 Frobenius Norm

Frobenius Norm

Example:

$$\Vert A{\Vert }_F=\sqrt{\sum _{i=1}^m\sum _{j=1}^n{a}_{ij}^2}{\Vert \left[\begin{array}{cc}1& 2\\ 2& 4\end{array}\right]\Vert }_F=\sqrt{1+4+4+16}=5$$

Square every entry, add them up, then take the square root, just like the L2 norm of a vector.

Where it's used: weight decay (L2 regularisation of weight matrices), and measuring how different two matrices are.

$$
\lVert A \rVert_F = \sqrt{\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}^2} \qquad \left\lVert \begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix} \right\rVert_F = \sqrt{1 + 4 + 4 + 16} = 5
$$

3. Determinants and Inverses (Intermediate)

3.1 Determinant of a 2 × 2 Matrix

Determinant: "\det" and "vmatrix"

Example:

$$det(A)=\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|=ad-bc\left|\begin{array}{cc}1& 2\\ 3& 4\end{array}\right|=1(4)-2(3)=-2$$

Where it's used: checking whether a matrix has an inverse. If $det(A)=0$, the matrix has no inverse.

$$
\det(A) = \begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc \qquad \begin{vmatrix} 1 & 2 \\ 3 & 4 \end{vmatrix} = 1(4) - 2(3) = -2
$$

3.2 Determinant of a 3 × 3 Matrix

Cofactor Expansion

Example:

$$\left|\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right|=a(ei-fh)-b(di-fg)+c(dh-eg)$$

Expand along the first row, alternating the signs $+$, $-$, $+$.

Where it's used: volumes, cross products, and checking invertibility of larger matrices.

$$
\begin{vmatrix} a & b & c \\ d & e & f \\ g & h & i \end{vmatrix} = a(ei - fh) - b(di - fg) + c(dh - eg)
$$

3.3 Inverse of a 2 × 2 Matrix

Inverse: "A^{-1}"

Example:

$$A^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}d& -b\\ -c& a\end{array}\right],A{A}^{-1}=A^{-1}A=I$$

Swap $a$ and $d$, change the signs of $b$ and $c$, then divide by the determinant.

Where it's used: solving $A\mathbf{x}=\mathbf{b}$ as $\mathbf{x}=A^{-1}\mathbf{b}$.

$$
A^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}, \qquad A A^{-1} = A^{-1} A = I
$$

3.4 Augmented Matrix

Augmented Matrix: "array" with a bar

Example:

$$\left[\begin{array}{ccc}2& 1& 5\\ 1& -1& 1\end{array}\right]$$

{cc|c} sets two centred columns, a vertical bar, then one more column. This is the system $2x+y=5$ and $x-y=1$.

Where it's used: setting up a system of equations for row reduction.

$$
\left[ \begin{array}{cc|c} 2 & 1 & 5 \\ 1 & -1 & 1 \end{array} \right]
$$

3.5 Row Reduction

Row Operations: "\xrightarrow{}"

Example:

$$\left[\begin{array}{ccc}2& 1& 5\\ 1& -1& 1\end{array}\right]\stackrel{R_2\to R_2-\frac{1}{2}{R}_1}{\to }\left[\begin{array}{ccc}2& 1& 5\\ 0& -\frac{3}{2}& -\frac{3}{2}\end{array}\right]$$

\xrightarrow{} writes the row operation above the arrow. From here, $y=1$ and then $x=2$.

Where it's used: Gaussian elimination, which is how computers solve systems of equations.

$$
\left[ \begin{array}{cc|c} 2 & 1 & 5 \\ 1 & -1 & 1 \end{array} \right] \xrightarrow{R_2 \to R_2 - \frac{1}{2} R_1} \left[ \begin{array}{cc|c} 2 & 1 & 5 \\ 0 & -\frac{3}{2} & -\frac{3}{2} \end{array} \right]
$$

3.6 Reduced Row Echelon Form

Reduced Row Echelon Form (RREF)

Example:

$$\left[\begin{array}{ccc}2& 1& 5\\ 1& -1& 1\end{array}\right]\stackrel{\text{row reduce}}{\to }\left[\begin{array}{ccc}1& 0& 2\\ 0& 1& 1\end{array}\right]$$

In RREF, every leading entry is 1 and it is the only non-zero number in its column. You can now read the solution directly: $x=2$ and $y=1$.

Where it's used: solving systems of equations, finding the rank, and finding inverses with Gauss-Jordan elimination.

$$
\left[ \begin{array}{cc|c} 2 & 1 & 5 \\ 1 & -1 & 1 \end{array} \right] \xrightarrow{\text{row reduce}} \left[ \begin{array}{cc|c} 1 & 0 & 2 \\ 0 & 1 & 1 \end{array} \right]
$$

3.7 Rank and Trace

Rank and Trace: "\operatorname{}"

Example:

$$\mathrm{rank}(A)\mathrm{tr}(A)=\sum _{i=1}^n{a}_{ii}$$

Rank is the number of linearly independent rows or columns. Trace is the sum of the diagonal entries.

Where it's used: checking whether a system has a unique solution, and low-rank methods such as LoRA for fine-tuning large language models.

$$
\operatorname{rank}(A) \qquad \operatorname{tr}(A) = \sum_{i=1}^{n} a_{ii}
$$

4. Vector Spaces (Intermediate)

4.1 Linear Combination and Span

Linear Combination and Span

Example:

$$\mathbf{v}=c_1{\mathbf{v}}_1+c_2{\mathbf{v}}_2+\cdots +c_k{\mathbf{v}}_k\mathrm{span}\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_k\}$$

The span is the set of all linear combinations of the vectors.

Where it's used: describing every vector you can build from a set of vectors.

$$
\mathbf{v} = c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_k \mathbf{v}_k \qquad \operatorname{span} \{ \mathbf{v}_1, \ldots, \mathbf{v}_k \}
$$

4.2 Linear Independence

Linear Independence

Example:

$$c_1{\mathbf{v}}_1+c_2{\mathbf{v}}_2+\cdots +c_k{\mathbf{v}}_k=\mathbf{0}\Rightarrow c_1=c_2=\cdots =c_k=0$$

The vectors are independent when the only way to get the zero vector is with all coefficients equal to 0.

Where it's used: spotting redundant features, such as two columns in a dataset that carry the same information.

$$
c_1 \mathbf{v}_1 + c_2 \mathbf{v}_2 + \cdots + c_k \mathbf{v}_k = \mathbf{0} \;\Rightarrow\; c_1 = c_2 = \cdots = c_k = 0
$$

4.3 Basis and Dimension

Basis and Dimension: "\dim"

Example:

$${\mathbf{e}}_1=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right],{\mathbf{e}}_2=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right],{\mathbf{e}}_3=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]\dim ({\mathbb{R}}^3)=3$$

These three vectors are the standard basis of ${\mathbb{R}}^3$.

Where it's used: describing how many independent directions a space has.

$$
\mathbf{e}_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}, \; \mathbf{e}_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}, \; \mathbf{e}_3 = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} \qquad \dim(\mathbb{R}^3) = 3
$$

4.4 Null Space, Column Space and the Rank-Nullity Theorem

Null Space and Column Space

Example:

$$\mathrm{Null}(A)=\{\mathbf{x}\mid A\mathbf{x}=\mathbf{0}\}\mathrm{rank}(A)+\dim \mathrm{Null}(A)=n$$

Here $n$ is the number of columns of $A$.

Where it's used: understanding when a system has no solution, one solution or infinitely many.

$$
\operatorname{Null}(A) = \{ \mathbf{x} \mid A\mathbf{x} = \mathbf{0} \} \qquad \operatorname{rank}(A) + \dim \operatorname{Null}(A) = n
$$

4.5 Orthogonality and Projection

Projection: "\operatorname{proj}"

Example:

$$\mathbf{u}\perp \mathbf{v}⟺\mathbf{u}\cdot \mathbf{v}=0{\mathrm{proj}}_{\mathbf{v}}\mathbf{u}=\frac{\mathbf{u}\cdot \mathbf{v}}{\mathbf{v}\cdot \mathbf{v}}\mathbf{v}$$

\perp is the symbol for perpendicular.

Where it's used: least squares regression, which projects the data onto the model, and the Gram-Schmidt process.

$$
\mathbf{u} \perp \mathbf{v} \iff \mathbf{u} \cdot \mathbf{v} = 0 \qquad \operatorname{proj}_{\mathbf{v}} \mathbf{u} = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{v} \cdot \mathbf{v}} \, \mathbf{v}
$$

4.6 Linear Transformations

Linear Transformation: "T: \mathbb{R}^{n} \to \mathbb{R}^{m}"

Example:

$$T:{\mathbb{R}}^n\to {\mathbb{R}}^m,T(\mathbf{x})=A\mathbf{x}$$$$T(c_1\mathbf{u}+c_2\mathbf{v})=c_{1}T(\mathbf{u})+c_{2}T(\mathbf{v})R_{\theta }=\left[\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right]$$

Every linear transformation from ${\mathbb{R}}^n$ to ${\mathbb{R}}^m$ is multiplication by an $m\times n$ matrix. $R_{\theta }$ rotates vectors by the angle $\theta$. The kernel and image of $T$ are the null space and column space of $A$.

Where it's used: rotating and scaling in computer graphics, augmenting image data, and each layer of a neural network before its activation function.

$$
T: \mathbb{R}^{n} \to \mathbb{R}^{m}, \qquad T(\mathbf{x}) = A\mathbf{x}
$$

$$
T(c_1 \mathbf{u} + c_2 \mathbf{v}) = c_1 T(\mathbf{u}) + c_2 T(\mathbf{v}) \qquad R_{\theta} = \begin{bmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{bmatrix}
$$

5. Eigenvalues and Decompositions (Advanced)

5.1 Eigenvalues and Eigenvectors

Eigen Equation: "A\mathbf{v} = \lambda \mathbf{v}"

Example:

$$A\mathbf{v}=\lambda \mathbf{v}$$

Multiplying by $A$ only stretches the eigenvector $\mathbf{v}$ by the eigenvalue $\lambda$.

Where it's used: Principal Component Analysis (PCA), which reduces the number of features while keeping the most information.

$$
A\mathbf{v} = \lambda \mathbf{v}
$$

5.2 Characteristic Equation

Finding Eigenvalues

Example:

$$det(A-\lambda I)=0$$$$A=\left[\begin{array}{cc}2& 1\\ 1& 2\end{array}\right]\Rightarrow \left|\begin{array}{cc}2-\lambda & 1\\ 1& 2-\lambda \end{array}\right|=(2-\lambda {)}^2-1=0\Rightarrow \lambda =1\text{ or }\lambda =3$$

Where it's used: calculating eigenvalues by hand, a common exam question.

$$
\det(A - \lambda I) = 0
$$

$$
A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \;\Rightarrow\; \begin{vmatrix} 2 - \lambda & 1 \\ 1 & 2 - \lambda \end{vmatrix} = (2 - \lambda)^2 - 1 = 0 \;\Rightarrow\; \lambda = 1 \text{ or } \lambda = 3
$$

5.3 Diagonalisation

Diagonalisation

Example:

$$A=PD{P}^{-1}$$

The columns of $P$ are the eigenvectors, and $D$ is a diagonal matrix of the eigenvalues.

Where it's used: computing powers like $A^k=P{D}^k{P}^{-1}$ quickly, such as in Markov chains.

$$
A = P D P^{-1}
$$

5.4 Orthogonal Matrices and the Spectral Theorem

Spectral Theorem

Example:

$$Q^{\mathrm{\top }}Q=IA=Q\mathrm{\Lambda }{Q}^{\mathrm{\top }}$$

Every symmetric matrix can be written this way, with orthogonal $Q$ and diagonal $\mathrm{\Lambda }$. \Lambda is a capital lambda.

Where it's used: decomposing covariance matrices in PCA.

$$
Q^{\top} Q = I \qquad A = Q \Lambda Q^{\top}
$$

5.5 Singular Value Decomposition

SVD

Example:

$$A=U\mathrm{\Sigma }{V}^{\mathrm{\top }}$$

$\mathrm{\Sigma }$ here is a diagonal matrix of singular values, not a sum sign.

Where it's used: dimensionality reduction, image compression, recommender systems and topic modelling.

$$
A = U \Sigma V^{\top}
$$

5.6 Positive Definite Matrices

Positive Definite

Example:

$${\mathbf{x}}^{\mathrm{\top }}A\mathbf{x}>0\text{for all }\mathbf{x}\ne \mathbf{0}$$

Where it's used: covariance matrices, and checking that a loss function is convex, so it has one minimum.

$$
\mathbf{x}^{\top} A \mathbf{x} > 0 \quad \text{for all } \mathbf{x} \neq \mathbf{0}
$$

5.7 Gram-Schmidt and QR Decomposition

Gram-Schmidt and QR

Example:

$${\mathbf{u}}_1={\mathbf{v}}_1,{\mathbf{u}}_2={\mathbf{v}}_2-{\mathrm{proj}}_{{\mathbf{u}}_1}{\mathbf{v}}_2,{\mathbf{e}}_i=\frac{{\mathbf{u}}_i}{\Vert {\mathbf{u}}_i\Vert }$$$$A=QR$$

Gram-Schmidt removes the projections onto earlier vectors, so the new vectors are orthogonal, then scales them to length 1. In $A=QR$, the columns of $Q$ are these orthonormal vectors and $R$ is upper triangular.

Where it's used: solving least squares problems accurately on a computer, and the QR algorithm for computing eigenvalues.

$$
\mathbf{u}_1 = \mathbf{v}_1, \qquad \mathbf{u}_2 = \mathbf{v}_2 - \operatorname{proj}_{\mathbf{u}_1} \mathbf{v}_2, \qquad \mathbf{e}_i = \frac{\mathbf{u}_i}{\lVert \mathbf{u}_i \rVert}
$$

$$
A = QR
$$

6. Linear Algebra for Machine Learning (Applied)

6.1 Dataset as a Matrix

Data Matrix

Example:

$$X=\left[\begin{array}{c}({\mathbf{x}}^{(1)}{)}^{\mathrm{\top }}\\ ({\mathbf{x}}^{(2)}{)}^{\mathrm{\top }}\\ ⋮\\ ({\mathbf{x}}^{(n)}{)}^{\mathrm{\top }}\end{array}\right]\in {\mathbb{R}}^{n\times d}$$

Each row is one training example ${\mathbf{x}}^{(i)}$, with $d$ features.

Where it's used: storing a dataset of $n$ samples, the input to almost every model.

$$
X = \begin{bmatrix} (\mathbf{x}^{(1)})^{\top} \\ (\mathbf{x}^{(2)})^{\top} \\ \vdots \\ (\mathbf{x}^{(n)})^{\top} \end{bmatrix} \in \mathbb{R}^{n \times d}
$$

6.2 Matrix Equation of a Neural Network Layer

Linear Layer: "W \mathbf{x} + \mathbf{b}"

Example:

$$\mathbf{y}=W\mathbf{x}+\mathbf{b}$$

If $\mathbf{x}\in {\mathbb{R}}^n$ and $W\in {\mathbb{R}}^{m\times n}$, then $\mathbf{y}\in {\mathbb{R}}^m$.

Where it's used: a fully connected layer of a neural network, where $W$ is the weight matrix and $\mathbf{b}$ is the bias vector.

$$
\mathbf{y} = W \mathbf{x} + \mathbf{b}
$$

6.3 Linear Regression and the Normal Equation

Normal Equation: "\hat{\boldsymbol{\beta}}"

Example:

$$\hat{\mathit{\beta }}=(X^{\mathrm{\top }}X{)}^{-1}{X}^{\mathrm{\top }}\mathbf{y}$$

Where it's used: the closed-form solution of linear regression, without gradient descent.

$$
\hat{\boldsymbol{\beta}} = (X^{\top} X)^{-1} X^{\top} \mathbf{y}
$$

6.4 Covariance Matrix

Covariance Matrix

Example:

$$\mathrm{\Sigma }=\frac{1}{n-1}{X}_c^{\mathrm{\top }}{X}_c$$

$X_c$ is the centred data matrix, where the mean of each column has been subtracted.

Where it's used: measuring how features vary together, the first step of PCA.

$$
\Sigma = \frac{1}{n - 1} X_c^{\top} X_c
$$

6.5 Principal Component Analysis

PCA Projection

Example:

$$Z=X_c{W}_k$$

The columns of $W_k$ are the top $k$ eigenvectors of the covariance matrix, so $Z$ keeps the $k$ most important directions.

Where it's used: reducing many features to a few, for visualisation and faster models.

$$
Z = X_c W_k
$$

6.6 Attention in Transformers

Scaled Dot-Product Attention

Example:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d_k}}\right)V$$

$Q$, $K$ and $V$ are the query, key and value matrices, and $d_k$ is the size of each key.

Where it's used: transformer models such as GPT and Claude.

$$
\operatorname{Attention}(Q, K, V) = \operatorname{softmax} \left( \frac{Q K^{\top}}{\sqrt{d_k}} \right) V
$$