Linear Algebra and Its Applications, Exercise 3.4.15

Exercise 3.4.15. Given the matrix

A = \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix}

find the orthonormal vectors q_1 and q_2 that span the column space of A. Next find the vector q_3 that completes the orthonormal set, and describe the subspace of A of which q_3 is an element. Finally, for b = (1, 2, 7) find the least squares solution \bar{x} to Ax = b.

Answer: With a and b as the columns of A, we first choose a' = a = (1, 2, -2). We then have

b' = b - \frac{(a')^Tb}{(a')^Ta'}a' = b - \frac{1 \cdot 1 + 2 \cdot (-1) + (-2) \cdot 4}{1^2 + 2^2  + (-2)^2}a' = b - \frac{-9}{9}a' = b + a'

= \begin{bmatrix} 1 \\ -1 \\ 4 \end{bmatrix} + \begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix} = \begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix}

Now that we have calculated the orthogonal vectors a' and b' we can normalize them to create the orthonormal vectors q_1 and q_2. We have

\|a'\| = \sqrt{1^2 + 2^2 + (-2)^2} = \sqrt{9} = 3

\|b'\| = \sqrt{2^2 + 1^2 + 2^2} = \sqrt{9} = 3

so that

q_1 = a' / \|a'\| = \frac{1}{3} \begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{2}{3} \\ -\frac{2}{3} \end{bmatrix}

q_2 = b' / \|b'\| = \frac{1}{3} \begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ \frac{1}{3} \\ \frac{2}{3} \end{bmatrix}

Since a and b span the column space of A, and the orthonormal vectors q_1 and q_2 are linear combinations of a and b, q_1 and q_2 also span the column space of A.

Next, we calculate q_3. We can do this by orthogonalizing any vector c that is linearly independent of a and b. For ease of calculation we start with c = (1, 0, 0). We then have

c' = c - \frac{(a')^Tc}{(a')^Ta'}a' - \frac{(b')^Tc}{(b')^Tb'}b' = c - \frac{1 \cdot 1 + 2 \cdot 0 + (-2) \cdot 0}{1^2 + 2^2  + (-2)^2}a' - \frac{2 \cdot 1 + 1 \cdot 0 + 2 \cdot 0}{1^2 + 2^2  + (-2)^2}b' = c - \frac{1}{9}a' - \frac{2}{9}b'

= \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} - \frac{1}{9} \begin{bmatrix} 1 \\ 2 \\ -2 \end{bmatrix} - \frac{2}{9} \begin{bmatrix} 2 \\ 1 \\ 2 \end{bmatrix} = \begin{bmatrix} \frac{4}{9} \\ -\frac{4}{9} \\ -\frac{2}{9} \end{bmatrix}

To normalize c' we divide by

\|c'\| = \sqrt{(-\frac{4}{9})^2 + (-\frac{4}{9})^2 + (-\frac{2}{9})^2} = \sqrt{\frac{36}{81}} = \frac{2}{3}

so that

q_3 = c' / \|c'\| = \frac{3}{2}  \begin{bmatrix} \frac{4}{9} \\ -\frac{4}{9} \\ -\frac{2}{9} \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ -\frac{2}{3} \\ -\frac{1}{3} \end{bmatrix}

Of the four fundamental subspaces of A, the left nullspace \mathcal{N}(A^T) is orthogonal to the column space \mathcal{R}(A). Since q_1 and q_2 span the column space \mathcal{R}(A) and q_3 is orthogonal to q_1 and q_2, q_3 must be an element of the left nullspace \mathcal{N}(A^T).

Finally, to find the least squares solution \bar{x} to Ax = b where b = (1, 2, 7), we factor A = QR and take advantage of the fact that R\bar{x} = Q^Tb.

The matrix Q is simply the 3 by 2 matrix with columns q_1 and q_2:

Q = \begin{bmatrix} \frac{1}{3}&\frac{2}{3} \\ \frac{2}{3}&\frac{1}{3} \\ -\frac{2}{3}&\frac{2}{3} \end{bmatrix}

The upper triangular matrix R is a 2 by 2 matrix with

R = \begin{bmatrix} q_1^Ta&q_1^Tb \\ 0&q_2^Tb \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \cdot 1 + \frac{2}{3} \cdot 2 + (-\frac{2}{3}) \cdot (-2)&\frac{1}{3} \cdot 1 + \frac{2}{3} \cdot (-1) + (-\frac{2}{3}) \cdot 4 \\ 0&\frac{2}{3} \cdot 1 + \frac{1}{3} \cdot (-1) + \frac{2}{3} \cdot 4 \end{bmatrix}

= \begin{bmatrix} 3&-3 \\ 0&3 \end{bmatrix}

On the right side of the equation R\bar{x} = Q^Tb we have

Q^Tb = \begin{bmatrix} \frac{1}{3}&\frac{2}{3}&-\frac{2}{3} \\ \frac{2}{3}&\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix} = \begin{bmatrix} -\frac{9}{3} \\ \frac{18}{3} \end{bmatrix} = \begin{bmatrix} -3 \\ 6 \end{bmatrix}

so that the entire system is then

\begin{bmatrix} 3&-3 \\ 0&3 \end{bmatrix} \begin{bmatrix} \bar{x_1} \\ \bar{x_2} \end{bmatrix} = \begin{bmatrix} -3 \\ 6 \end{bmatrix}

From the second equation we have 3\bar{x_2} = 6 or \bar{x_2} = 2. Substituting into the first equation we have 3\bar{x_1} - 3\bar{x_2} = 3\bar{x_1} - 3 \cdot 2 = 3 so that 3\bar{x_1} = 3 + 6 = 9 or \bar{x_1} = 3.

The least squares solution to Ax = b with b = (1, 2, 7) is therefore \bar{x} = (3, 2).

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

This entry was posted in linear algebra and tagged , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s