HEAD

PREVIOUS

Appendix A
Summary of Matrix Algebra

A.1  Vector and Matrix Multiplication

We consider a vector $\mathbf{v}$ of length $J$ (in an abstract vector space of dimensions $J$) to be an ordered sequence of $J$ numbers${}^{103}$. The vector can be displayed either as a column

${\displaystyle \mathbf{v}=\left(\begin{array}{c}\hfill v_1\hfill \\ \hfill v_2\hfill \\ \hfill :\hfill \\ \hfill v_J\hfill \end{array}\right)}$

${\displaystyle (A.1)}$

or as a row, which we regard as the transpose, denoted ${}^T$, of the column vector:

${\displaystyle {\mathbf{v}}^T=(v_1,v_2,...,v_J).}$

${\displaystyle (A.2)}$

Vectors of the same dimensions can be added together so that the $j^{\mathit{th}}$ entry of $\mathbf{u}+\mathbf{v}$ is $u_j+v_j$. The scalar product of two vectors $\mathbf{u}$, $\mathbf{v}$, in vector notation is indicated by a dot, but in matrix notation the dot is usually omitted. Instead we write it

${\displaystyle {\mathbf{u}}^T\mathbf{v}=\sum _{j=1}^J{u}_j{v}_j.}$

${\displaystyle (A.3)}$

If we have a set of $k$ column vectors ${\mathbf{v}}_k$, for $k=1,\dots ,K$, the $j^{\mathit{th}}$ element of the $k^{\mathit{th}}$ vector can be written, $V_{\mathit{jk}}$, and they can be arrayed compactly one after the other as

${\displaystyle \mathbf{V}=\left(\begin{array}{cccc}\hfill V_{11}\hfill & \hfill V_{12}\hfill & \hfill \dots \hfill & \hfill V_{1K}\hfill \\ \hfill V_{21}\hfill & \hfill V_{22}\hfill & \hfill \dots \hfill & \hfill V_{2K}\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill \ddots \hfill & \hfill :\hfill \\ \hfill V_{J1}\hfill & \hfill V_{J2}\hfill & \hfill \dots \hfill & \hfill V_{\mathit{JK}}\hfill \end{array}\right).}$

${\displaystyle (A.4)}$

This is a matrix. We can consider matrix multiplication to be a generalization of the scalar product. So premultiplying a $J\times K$ matrix $\mathbf{V}$, by a length $J$ row vector ${\mathbf{u}}^T$ gives a new row vector of length $K$

${\displaystyle {\mathbf{u}}^T\mathbf{V}=(\sum _{j=1}^J{u}_j{V}_{j1},\sum _{j=1}^J{u}_j{V}_{j2},\dots ,\sum _{j=1}^J{u}_j{V}_{\mathit{jK}}).}$

${\displaystyle (A.5)}$

If we further have a set of $M$ row vectors, we can display them as a matrix

${\displaystyle \mathbf{U}=\left(\begin{array}{cccc}\hfill U_{11}\hfill & \hfill U_{12}\hfill & \hfill \dots \hfill & \hfill U_{1J}\hfill \\ \hfill U_{21}\hfill & \hfill U_{22}\hfill & \hfill \dots \hfill & \hfill U_{2J}\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill \ddots \hfill & \hfill :\hfill \\ \hfill U_{M1}\hfill & \hfill U_{M2}\hfill & \hfill \dots \hfill & \hfill U_{\mathit{MJ}}\hfill \end{array}\right)}$

${\displaystyle (A.6)}$

(dispensing with the transpose notation for brevity and consistency). And multiplication of the matrices $\mathbf{U}$ ($M\times J$) and $\mathbf{V}$ ($J\times K$) can be considered to give an $M\times K$ matrix:

${\displaystyle \mathbf{U}\mathbf{V}=\left(\begin{array}{cccc}\hfill \sum _{j=1}^J{U}_{1j}{V}_{j1}\hfill & \hfill \sum _{j=1}^J{U}_{1j}{V}_{j2}\hfill & \hfill \dots \hfill & \hfill \sum _{j=1}^J{U}_{1j}{V}_{\mathit{jK}}\hfill \\ \hfill \sum _{j=1}^J{U}_{2j}{V}_{j1}\hfill & \hfill \sum _{j=1}^J{U}_{2j}{V}_{j2}\hfill & \hfill \dots \hfill & \hfill \sum _{j=1}^J{U}_{2j}{V}_{\mathit{jK}}\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill \ddots \hfill & \hfill :\hfill \\ \hfill \sum _{j=1}^J{U}_{\mathit{Mj}}{V}_{j1}\hfill & \hfill \sum _{j=1}^J{U}_{\mathit{Mj}}{V}_{j2}\hfill & \hfill \dots \hfill & \hfill \sum _{j=1}^J{U}_{\mathit{Mj}}{V}_{\mathit{jK}}\hfill \end{array}\right).}$

${\displaystyle (A.7)}$

This is the definition of matrix multiplication. A matrix (or vector) can also be multiplied by a single number: a scalar, $\mathit{\lambda }$ (say). The $(\mathit{jk})$th element of $\mathit{\lambda }\mathbf{V}$ is $\mathit{\lambda }{V}_{\mathit{jk}}$. The transpose of a matrix $\mathbf{A}=(A_{\mathit{ij}})$ is simply the matrix formed from reversing the order of suffixes${}^{104}$: ${\mathbf{A}}^T=(A_{\mathit{ij}}^T)=(A_{\mathit{ji}})$. The transpose of a product of two matrices is therefore the reverse of the product of the transposes:

${\displaystyle (\mathbf{A}\mathbf{B}{)}^T={\mathbf{B}}^T{\mathbf{A}}^T.}$

${\displaystyle (A.8)}$

A.2  Determinants

The determinant of a square matrix is a single scalar that is an important measure of its character. Determinants may be defined inductively. Suppose we know the definition of determinants of matrices of size $(M-1)\times (M-1)$. Define the determinant of an $M\times M$ matrix $\mathbf{A}$ whose ${\mathit{ij}}^{\mathit{th}}$ entry is $A_{\mathit{ij}}$, as the expression

${\displaystyle det(\mathbf{A})=|\mathbf{A}|=\sum _{j=1}^M{A}_{1j}{\mathrm{Co}}_{1j}(\mathbf{A})}$

${\displaystyle (A.9)}$

where ${\mathrm{Co}}_{\mathit{ij}}(\mathbf{A})$ is the ${\mathit{ij}}^{\mathit{th}}$ cofactor of the matrix $\mathbf{A}$. The ${\mathit{ij}}^{\mathit{th}}$ cofactor of an $M\times M$ matrix is $(-1{)}^{i+j}$ times the determinant of the $(M-1)\times (M-1)$ matrix obtained by removing the $i^{\mathit{th}}$ row and the $j^{\mathit{th}}$ column of the original matrix:

${\displaystyle {\mathrm{Co}}_{\mathit{ij}}(\mathbf{A})=(-1{)}^{i+j}\left|\left(\begin{array}{cccccc}\hfill A_{11}\hfill & \hfill \dots \hfill & \hfill A_{1j-1}\hfill & \hfill A_{1j+1}\hfill & \hfill \dots \hfill & \hfill A_{1M}\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill \\ \hfill A_{i-1,1}\hfill & \hfill \dots \hfill & \hfill A_{i-1,j-1}\hfill & \hfill A_{i-1,j+1}\hfill & \hfill \dots \hfill & \hfill A_{i-1,M}\hfill \\ \hfill A_{i+1,1}\hfill & \hfill \dots \hfill & \hfill A_{i+1,j-1}\hfill & \hfill A_{i+1,j+1}\hfill & \hfill \dots \hfill & \hfill A_{i+1,M}\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill & \hfill :\hfill \\ \hfill A_{M1}\hfill & \hfill \dots \hfill & \hfill A_{\mathit{Mj}-1}\hfill & \hfill A_{\mathit{Mj}+1}\hfill & \hfill \dots \hfill & \hfill A_{\mathit{MM}}\hfill \end{array}\right)\right|.}$

${\displaystyle (A.10)}$

The inductive definition is completed by defining the determinant of a $1\times 1$ matrix to be equal to its single element. The determinant of a $2\times 2$ matrix is then $A_{11}{A}_{22}-A_{12}{A}_{21}$, and of a $3\times 3$ matrix is $A_{11}(A_{22}{A}_{33}-A_{23}{A}_{32})+A_{12}(A_{23}{A}_{31}-A_{21}{A}_{13})+A_{13}(A_{21}{A}_{32}-A_{22}{A}_{31})$. The determinant of an $M\times M$ matrix may equivalently be defined as the sum over all the $M!$ possible permutations $P$ of the integers $1,...,M$, of the product of the entries $\underset{i}{\Pi }{A}_{i,P(i)}$ times the signum of $P$ (plus or minus 1 according to whether $P$ is even or odd):

${\displaystyle |\mathbf{A}|=\sum _P\mathrm{sgn}(P)A_{1,P(1)}{A}_{2,P(2)}...A_{M,P(M)}.}$

${\displaystyle (A.11)}$

This expression shows that there is nothing special about the first row in eq. (A.9). One could equally well have used any row, $i$, giving $|\mathbf{A}|=\sum _{j=1}^M{A}_{\mathit{ij}}{\mathrm{Co}}_{\mathit{ij}}(\mathbf{A})$; or one could have used any column, $j$, $|\mathbf{A}|=\sum _{i=1}^M{A}_{\mathit{ij}}{\mathrm{Co}}_{\mathit{ij}}(\mathbf{A})$. All the results are the same. The determinant of the transpose of a matrix $\mathbf{A}$ is equal to its determinant: $|{\mathbf{A}}^T|=|\mathbf{A}|$. The determinant of a product of two matrices is the product of the determinants: $|\mathbf{A}\mathbf{B}|=|\mathbf{A}||\mathbf{B}|$. A matrix is said to be singular if its determinant is zero, otherwise it is nonsingular. If a matrix has two identical (or proportional, i.e. dependent) rows or two identical columns, then its determinant is zero and it is singular${}^{105}$.

A.3  Inverses

The unit matrix is square,

${\displaystyle \mathbf{I}=({\mathit{\delta }}_{\mathit{ij}})=\left(\begin{array}{cccc}\hfill 1\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill \ddots \hfill & \hfill :\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 1\hfill \end{array}\right)}$

${\displaystyle (A.12)}$

with ones on the diagonal and zeroes elsewhere. It may be of any size, $N$, and if need be then denoted ${\mathbf{I}}_N$. For any $M\times N$ matrix $\mathbf{A}$,

${\displaystyle {\mathbf{I}}_M\mathbf{A}=\mathbf{A}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}}\text{and}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}}\mathbf{A}{\mathbf{I}}_N=\mathbf{A}.}$

${\displaystyle (A.13)}$

The inverse of a square matrix $\mathbf{A}$, if it exists, is another matrix written ${\mathbf{A}}^{-1}$ such that${}^{106}$

${\displaystyle {\mathbf{A}}^{-1}\mathbf{A}=\mathbf{A}{\mathbf{A}}^{-1}=\mathbf{I}.}$

${\displaystyle (A.14)}$

A nonsingular square matrix possesses an inverse. A singular matrix does not. The inverse of a matrix may be identified by considering the identity

${\displaystyle \sum _{j=1}^M{A}_{\mathit{ij}}{\mathrm{Co}}_{\mathit{kj}}(\mathbf{A})={\mathit{\delta }}_{\mathit{ik}}|\mathbf{A}|.}$

${\displaystyle (A.15)}$

For $i=k$, this equality arises as the expansion of the determinant by row $i$. For $i\ne k$, the sum represents the determinant, expanded by row $k$, of a matrix in which the row $k$ has been replaced by a copy of row $i$. The modified matrix has two rows identical, so its determinant is zero, as is ${\mathit{\delta }}_{\mathit{ij}},\hspace{0.5em}i\ne j$. Now if we regard $\mathrm{Co}(\mathbf{A})$ as a matrix, consisting of all the cofactors. Then we can consider $\sum _{j=1}^M{A}_{\mathit{ij}}{\mathrm{Co}}_{\mathit{kj}}(\mathbf{A})$ as being the matrix product of $\mathbf{A}$ by the transpose of the cofactor matrix, $\mathbf{A}\mathrm{Co}(\mathbf{A}{)}^T$. So if $|\mathbf{A}|$ is nonzero we may divide (A.15) through by it and find

${\displaystyle \mathbf{A}[\mathrm{Co}(\mathbf{A}{)}^T/|\mathbf{A}|]=\mathbf{I}.}$

${\displaystyle (A.16)}$

This equality shows that

${\displaystyle {\mathbf{A}}^{-1}=\mathrm{Co}(\mathbf{A}{)}^T/|\mathbf{A}|.}$

${\displaystyle (A.17)}$

Consequently the solution of the nonsingular matrix equation $\mathbf{A}\mathbf{x}=\mathbf{b}$ is

${\displaystyle \mathbf{x}=\frac{\mathrm{Co}(\mathbf{A}{)}^T\mathbf{b}}{|\mathbf{A}|},}$

${\displaystyle (A.18)}$

which for column vectors $\mathbf{x}$ and $\mathbf{b}$ is Cramer's rule. The inverse of the product of two nonsingular matrices is the reversed product of their inverses:

${\displaystyle (\mathbf{A}\mathbf{B}{)}^{-1}={\mathbf{B}}^{-1}{\mathbf{A}}^{-1}.}$

${\displaystyle (A.19)}$

A.4  Eigenanalysis

A square matrix $\mathbf{A}$ maps the linear space of column vectors onto itself via $\mathbf{A}\mathbf{x}=\mathbf{y}$, with $\mathbf{y}$ the vector onto which $\mathbf{x}$ is mapped. An eigenvector is a vector which is mapped onto a multiple of itself. That is

${\displaystyle \mathbf{A}\mathbf{x}=\mathit{\lambda }\mathbf{x},}$

${\displaystyle (A.20)}$

where $\mathit{\lambda }$ is a scalar called the eigenvalue. In general a square matrix of dimension $N$ has $N$ different eigenvectors. Obviously an eigenvector times any scalar is still an eigenvector, which is not considered to be different. Since eq. (A.20), which is $(\mathbf{A}-\mathit{\lambda }\mathbf{I})\mathbf{x}=0$, is a homogeneous equation for the elements of $\mathbf{x}$, in order for there to be a non-zero solution, $\mathbf{x}$, the determinant of the coefficients must be zero:

${\displaystyle |\mathbf{A}-\mathit{\lambda }\mathbf{I}|=0.}$

${\displaystyle (A.21)}$

For an $N\times N$ matrix, this determinant gives a polynomial of order $N$ for $\mathit{\lambda }$, whose $N$ roots are the $N$ eigenvalues. If $\mathbf{A}$ is symmetric, that is if ${\mathbf{A}}^T=\mathbf{A}$, then the eigenvectors corresponding to different eigenvalues are orthogonal, that is, their scalar product is zero. See this by considering two eigenvectors ${\mathbf{e}}_1$ and ${\mathbf{e}}_2$, corresponding to different eigenvalues ${\mathit{\lambda }}_1$, ${\mathit{\lambda }}_2$, and using the respective versions of eq. (A.20) and the properties of the transpose.

${\displaystyle {\mathbf{e}}_2^T\mathbf{A}{\mathbf{e}}_1={\mathbf{e}}_2^T{\mathit{\lambda }}_1{\mathbf{e}}_1,\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}{\mathbf{e}}_2^T{\mathbf{A}}^T{\mathbf{e}}_1=({\mathbf{e}}_1^T\mathbf{A}{\mathbf{e}}_2{)}^T=({\mathbf{e}}_1^T{\mathit{\lambda }}_2{\mathbf{e}}_2{)}^T={\mathbf{e}}_2^T{\mathit{\lambda }}_2{\mathbf{e}}_1.}$

${\displaystyle (A.22)}$

So by subtraction

${\displaystyle 0={\mathbf{e}}_2^T(\mathbf{A}-{\mathbf{A}}^T){\mathbf{e}}_1=({\mathit{\lambda }}_1-{\mathit{\lambda }}_2){\mathbf{e}}_2^T{\mathbf{e}}_1.}$

${\displaystyle (A.23)}$

If there are multiple independent eigenvectors with identical eigenvalues, they can be chosen to be orthogonal. In that standard case, the eigenvectors are all orthogonal: ${\mathbf{e}}_i^T{\mathbf{e}}_j=0$ for $i\ne j$. If we then take the eigenvectors also to be normalized such that ${\mathbf{e}}_j^T{\mathbf{e}}_j=1$, we can construct a square matrix $\mathbf{U}$ whose columns are equal to these eigenvectors (as in eq. (A.4)). The matrix $\mathbf{U}$ whose columns are orthonormal is said to be an orthonormal matrix (sometimes just called orthogonal). The inverse of $\mathbf{U}$ is its transpose: ${\mathbf{U}}^{-1}={\mathbf{U}}^T$. This $\mathbf{U}$ is a unitary basis transformation which diagonalizes $\mathbf{A}$. This fact follows from the observation that $\mathbf{A}\mathbf{U}=\mathbf{D}\mathbf{U}=\mathbf{U}\mathbf{D}$ where $\mathbf{D}$ is the diagonal matrix constructed from the eigenvalues:

${\displaystyle \mathbf{D}=\left(\begin{array}{cccc}\hfill {\mathit{\lambda }}_1\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\mathit{\lambda }}_2\hfill & \hfill \dots \hfill & \hfill 0\hfill \\ \hfill :\hfill & \hfill :\hfill & \hfill \ddots \hfill & \hfill :\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill {\mathit{\lambda }}_N\hfill \end{array}\right).}$

${\displaystyle (A.24)}$

Therefore

${\displaystyle {\mathbf{U}}^T\mathbf{A}\mathbf{U}={\mathbf{U}}^T\mathbf{U}\mathbf{D}=\mathbf{D}.}$

${\displaystyle (A.25)}$

---

HEAD

NEXT