## Linear Algebra and Its Applications, Exercise 3.3.1

Exercise 3.3.1. a) Given the system of equations consisting of $3x = 10$ and $4x = 5$ find the least squares solution $\bar{x}$ and describe the error $E^2$ being minimized by that solution. Confirm that the error vector $\left(10-3\bar{x}, 5-4\bar{x}\right)$ is orthogonal to the column $\left(3, 4\right)$.

Answer: This is a system $Ax = b$ where $A = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}$. We can then compute the least squares solution as $\bar{x} = \left(A^TA\right)^{-1}A^Tb$.

We have $A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25$

so that $\left(A^TA\right)^{-1} = \frac{1}{25}$. We then have $\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}$ $= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2$

So the least squares solution is $\bar{x} = 2$.

The error vector is then $b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}$

and the corresponding error $E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)$ $= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25$

The inner product of the error vector $b-A\bar{x} = \left(4, -3\right)$ and the column $\left(3, 4\right)$ of $A$ is $4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0$. Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition , Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books .

This entry was posted in linear algebra and tagged , . Bookmark the permalink.