## Linear Algebra and Its Applications, Exercise 3.3.13

Exercise 3.3.13. Using least squares, find the line that is the best fit to the following measurements: $b = 4$ at $t = -2$ $b = 3$ at $t = -1$ $b = 1$ at $t = 0$ $b = 0$ at $= 2$

Also, given the matrix $A = \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix}$

find the projection of $b = (4, 3, 1, 0)$ onto the column space $\mathcal{R}(A)$.

Answer: Assuming that the line in question has the form $C + Dt$ the problem can be expressed as that of finding a solution to the system $\begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 4 \\ 3 \\ 1 \\ 0 \end{bmatrix}$

or $Ax = b$ where $x = (C, D)$ is the exact solution.

In this case there is no exact solution, so we look for the least squares solution $\bar{x} = (\bar{C}, \bar{D})$ that minimizes the error vector $b-A\bar{x}$. The error vector is minimized when it is orthogonal to the column space of $A$ and is therefore in the left nullspace of $A$. We then have $A^T(b - A\bar{x}) = 0$ so that $\bar{x}$ is a solution to the system $A^TA\bar{x} = A^Tb$.

We have $A^TA = \begin{bmatrix} 1&1&1&1 \\ -2&-1&0&2 \end{bmatrix} \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix} = \begin{bmatrix} 4&-1 \\ -1&9 \end{bmatrix}$

and $A^Tb = \begin{bmatrix} 1&1&1&1 \\ -2&-1&0&2 \end{bmatrix} \begin{bmatrix} 4 \\ 3 \\ 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 8 \\ -11 \end{bmatrix}$

so that the system $A^TA\bar{x} = A^Tb$ reduces to $\begin{bmatrix} 4&-1 \\ -1&9 \end{bmatrix} \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 8 \\ -11 \end{bmatrix}$

or $\setlength\arraycolsep{0.2em}\begin{array}{rcrcl}4\bar{C}&-&\bar{D}&=&8 \\ -\bar{C}&+&9\bar{D}&=&-11 \end{array}$

expressed as a system of equations.

Multiplying the first equation by $\frac{1}{4}$ and adding it to the second equation produces the system $\setlength\arraycolsep{0.2em}\begin{array}{rcrcl}4\bar{C}&-&\bar{D}&=&8 \\ &&\frac{35}{4}\bar{D}&=&-9 \end{array}$

From the second equation we have $\bar{D} = -\frac{36}{35}$. Substituting that value into the first equation we have $4\bar{C} + \frac{36}{35} = 8$ or $\bar{C} = \frac{1}{4}(8 - \frac{36}{35}) = \frac{61}{35}$.

The line of best fit is therefore $\frac{61}{35} - \frac{36}{35}t$.

Given the matrix $A = \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix}$

the projection matrix $P$ onto the column space of $A$ can be computed as $P = A(A^TA)^{-1}A^T$

From above we have $A^TA = \begin{bmatrix} 4&-1 \\ -1&9 \end{bmatrix}$

so that its inverse is $(A^TA)^{-1} = \frac{1}{4 \cdot 9 - (-1)(-1)} \begin{bmatrix} 9&-(-1) \\ -(-1)&4 \end{bmatrix}$ $= \frac{1}{35} \begin{bmatrix} 9&1 \\ 1&4 \end{bmatrix}$

We then have $P = A(A^TA)^{-1}A^T$ $= \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix} \frac{1}{35} \begin{bmatrix} 9&1 \\ 1&4 \end{bmatrix} \begin{bmatrix} 1&1&1&1 \\ -2&-1&0&2 \end{bmatrix}$ $= \frac{1}{35} \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 7&8&9&11 \\ -7&-3&1&9 \end{bmatrix}$ $= \frac{1}{35} \begin{bmatrix} 21&14&7&-7 \\ 14&11&8&2 \\ 7&8&9&11 \\ -7&2&11&29 \end{bmatrix}$

The projection of the vector $b = (4, 3, 1, 0)$ onto the column space of $A$ is then $Pb = \frac{1}{35} \begin{bmatrix} 21&14&7&-7 \\ 14&11&8&2 \\ 7&8&9&11 \\ -7&2&11&29 \end{bmatrix} \begin{bmatrix} 4 \\ 3 \\ 1 \\ 0 \end{bmatrix}$ $= \frac{1}{35} \begin{bmatrix} 133 \\ 97 \\ 61 \\ -11 \end{bmatrix} = \begin{bmatrix} \frac{133}{35} \\ \frac{97}{35} \\ \frac{61}{35} \\ -\frac{11}{35} \end{bmatrix}$

The vector $Pb$ corresponds to the points on the least squares line of best fit $\bar{C} + \bar{D}t$ for the times $t = (-2, 1, 0, 2)$: $\frac{61}{35} - \frac{36}{35} (-2) = \frac{133}{35}$ $\frac{61}{35} - \frac{36}{35} (-1) = \frac{97}{35}$ $\frac{61}{35} - \frac{36}{35} (0) = \frac{61}{35}$ $\frac{61}{35} - \frac{36}{35} (2) = \frac{-11}{35}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition , Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books .

This entry was posted in linear algebra and tagged , . Bookmark the permalink.