Matrices & Multiplication

$A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}$

is a $2\times2$ matrix, and $A_{21} = 3$ (row 2, column 1).

But a matrix is more than storage. It is also a function: it takes a vector in and returns a new vector, stretching, rotating, and shearing space in a straight-line ("linear") way. Multiplying a vector by a matrix applies that transformation. Holding both pictures, the static grid and the active transformation, at once is what makes this concept hard, so let us look at it from three angles.

The inner-dimension rule

You can multiply $A$ by $B$ only when the number of columns of $A$ equals the number of rows of $B$. Line the shapes up side by side:

$(m \times n)\,(n \times p) \;\longrightarrow\; (m \times p)$

The two inner numbers (both $n$) must match; they cancel, and the two outer numbers ($m$ and $p$) become the shape of the result. So a $2\times2$ times a $2\times1$ gives a $2\times1$, and a $2\times2$ times a $2\times2$ gives a $2\times2$. If the inner numbers disagree, the product simply does not exist.

Worked example 1: matrix times vector

Take

$A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}, \qquad v = \begin{bmatrix} 5 \\ 6 \end{bmatrix}.$

Here $A$ is $2\times2$ and $v$ is $2\times1$; the inner dimensions (both $2$) match, so $Av$ is $2\times1$. Each output entry is one row of $A$ dotted with $v$:

$Av = \begin{bmatrix} 2\cdot 5 + 1\cdot 6 \\ 3\cdot 5 + 4\cdot 6 \end{bmatrix} = \begin{bmatrix} 16 \\ 39 \end{bmatrix}.$

That is exactly what a two-neuron linear layer with no bias computes on a single input.

Worked example 2: matrix times matrix

Now multiply two matrices:

$A = \begin{bmatrix} 2 & 1 \\ 3 & 4 \end{bmatrix}, \qquad B = \begin{bmatrix} 5 & 0 \\ 6 & 7 \end{bmatrix}.$

Both are $2\times2$, so $C = AB$ is $2\times2$. Compute each entry as (row of $A$) dotted with (column of $B$):

1. 1C11 = row 1 · col 12·5 + 1·6 = 10 + 6 = 16
2. 2C12 = row 1 · col 22·0 + 1·7 = 0 + 7 = 7
3. 3C21 = row 2 · col 13·5 + 4·6 = 15 + 24 = 39
4. 4C22 = row 2 · col 23·0 + 4·7 = 0 + 28 = 28

Collecting the four results:

$C = AB = \begin{bmatrix} 16 & 7 \\ 39 & 28 \end{bmatrix}.$

Further reading

• 3Blue1Brown, Essence of Linear Algebra, watch "Linear transformations" and "Matrix multiplication as composition" for the grid-as-transformation view.
• MIT 18.06 Linear Algebra (Gilbert Strang), the canonical university course; the opening lectures cover multiplication several different ways.
• NumPy matmul reference, how the operation and the @ operator behave in real code.