Linear Algebra and Its Applications, Exercise 3.3.1

Exercise 3.3.1. a) Given the system of equations consisting of 3x = 10 and 4x = 5 find the least squares solution \bar{x} and describe the error E^2 being minimized by that solution. Confirm that the error vector \left(10-3\bar{x}, 5-4\bar{x}\right) is orthogonal to the column \left(3, 4\right).

Answer: This is a system Ax = b where A = \begin{bmatrix} 3 \\ 4 \end{bmatrix} and b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}. We can then compute the least squares solution as \bar{x} = \left(A^TA\right)^{-1}A^Tb.

We have

A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25

so that \left(A^TA\right)^{-1} = \frac{1}{25}. We then have

\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}

= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2

So the least squares solution is \bar{x} = 2.

The error vector is then

b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}

and the corresponding error

E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)

= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25

The inner product of the error vector b-A\bar{x} = \left(4, -3\right) and the column \left(3, 4\right) of A is 4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0. Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

This entry was posted in linear algebra and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s