Linear Algebra and Its Applications, Exercise 3.3.1

Exercise 3.3.1. a) Given the system of equations consisting of $3x = 10$ and $4x = 5$ find the least squares solution $\bar{x}$ and describe the error $E^2$ being minimized by that solution. Confirm that the error vector $\left(10-3\bar{x}, 5-4\bar{x}\right)$ is orthogonal to the column $\left(3, 4\right)$ .

Answer: This is a system $Ax = b$ where $A = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}$ . We can then compute the least squares solution as $\bar{x} = \left(A^TA\right)^{-1}A^Tb$ .

We have

$A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25$

so that $\left(A^TA\right)^{-1} = \frac{1}{25}$ . We then have

$\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}$

$= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2$

So the least squares solution is $\bar{x} = 2$ .

The error vector is then

$b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}$

and the corresponding error

$E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)$

$= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25$

The inner product of the error vector $b-A\bar{x} = \left(4, -3\right)$ and the column $\left(3, 4\right)$ of $A$ is $4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0$ . Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

Buy me a snack to sponsor more posts like this!

Linear Algebra and Its Applications, Exercise 3.3.1

Leave a Reply Cancel reply

Archives

Meta

Linear Algebra and Its Applications, Exercise 3.3.1

Share this:

Like this:

Related

Leave a Reply Cancel reply

Archives

Meta