## Linear Algebra and Its Applications, Exercise 3.4.10

Exercise 3.4.10. Given the two orthonormal vectors $q_1$ and $q_2$ and an arbitrary vector $b$, what linear combination of $q_1$ and $q_2$ is the least distance from $b$? Show that the difference between $b$ and that combination (i.e., the error vector) is orthogonal to both $q_1$ and $q_2$.

Answer: This exercise is similar to the previous one. Any linear combination of $q_1$ and $q_2$ is in the plane formed by $q_1$ and $q_2$, and the combination closest to $b$ is the projection $p$ of $b$ onto that plane. Because $q_1$ and $q_2$ are orthonormal that projection is equal to the sum of the separate projections of $b$ onto $q_1$ and $q_2$ respectively:

$p = (q_1^Tb)q_1 + (q_2^Tb)q_2$

So $(q_1^Tb)q_1 + (q_2^Tb)q_2$ is the closest combination to $b$.

The error vector is then

$e = b - p = b - (q_1^Tbq_1 + q_2^Tbq_2)$

Taking the dot product of the error vector with $q_1$ we have

$q_1^Te = q_1^T [b - (q_1^Tbq_1 + q_2^Tbq_2)] = q_1^Tb - q_1^T(q_1^Tb)q_1 - q_1^T(q_2^Tb)q_2$

$= q_1^Tb - (q_1^Tb)q_1^Tq_1 - (q_2^Tb)q_1^Tq_2$

Since $q_1$ and $q_2$ are orthonormal we have $q_1^Tq_1 = 1$ and $q_1^Tq_2 = 0$. So we have

$q_1^Te = q_1^Tb - (q_1^Tb) \cdot 1 - (q_2^Tb) \cdot 0 = q_1^Tb - q_1^Tb = 0$

Similarly we have

$q_2^Te = q_2^T [b - (q_1^Tbq_1 + q_2^Tbq_2)] = q_2^Tb - q_2^T(q_1^Tb)q_1 - q_2^T(q_2^Tb)q_2$

$= q_2^Tb - (q_1^Tb)q_2^Tq_1 - (q_2^Tb)q_2^Tq_2 = q_2^Tb - q_2^Tb = 0$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.4.9

Exercise 3.4.9. Given the three orthonormal vectors $q_1$, $q_2$, and $q_3$, what linear combination of $q_1$ and $q_2$ is the least distance from $q_3$?

Answer: Any linear combination of $q_1$ and $q_2$ is in the plane formed by $q_1$ and $q_2$. The combination closest to $q_3$ is simply the projection of $q_3$ onto that plane. But because $q_3$ is orthogonal to both $q_1$ and $q_2$ it is orthogonal to that plane, and its projection onto the plane is the zero vector. So the linear combination of $q_1$ and $q_2$ closest to $q_3$ is $0 \cdot q_1 + 0 \cdot q_2$.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.4.8

Exercise 3.4.8. Project the vector $b = (1, 2)$ onto the two non-orthogonal vectors $a_1 = (1, 0)$ and $a_2 = (1, 1)$ and show that the sum of the two projections does not equal $b$ (as it would if $a_1$ and $a_2$ were orthogonal).

Answer: The projection of $b$ onto $a_1$ is $(a_1^Tb/a_1^Ta_1)a_1$. We have $a_1^Tb = (1 \cdot 1 + 0 \cdot 2) = 1$ and $a_1^Ta_1 = (1 \cdot 1 + 0 \cdot 0) = 1$. So the projection of $b$ onto $a_1$ is $\frac{1}{1} a_1 = (1, 0)$.

Similarly, the projection of $b$ onto $a_2$ is $(a_2^Tb/a_2^Ta_2)a_2$. We have $a_2^Tb = (1 \cdot 1 + 1 \cdot 2) = 3$ and $a_2^Ta_2 = (1 \cdot 1 + 1 \cdot 1) = 2$. So the projection of $b$ onto $a_2$ is $\frac{3}{2} a_2 = (\frac{3}{2}, \frac{3}{2})$.

The sum of the two projections is $(1, 0) + (\frac{3}{2}, \frac{3}{2}) = (\frac{5}{2}, \frac{3}{2})$, which is not equal to $b$.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged | Leave a comment

## Linear Algebra and Its Applications, Exercise 3.4.7

Exercise 3.4.7. Given $b = x_1q_1 + x_2q_2 + \cdots + x_nq_n$ where $q_1, q_2, \dots, q_n$ are orthonormal vectors, compute $b^Tb$ and show that

$\|b\|^2 = x_1^2 + x_2^2 + \cdots + x_n^2$

Answer:We have $b = \sum_i x_iq_i$ so that

$b^Tb = (\sum_i x_iq_i)^T(\sum_j x_jq_j) = (\sum_i x_iq_i^T)(\sum_j x_jq_j)$

since the transpose of a sum is equal to the sum of the transposes. The product of the sums can then be decomposed into two sums of products as follows:

$(\sum_i x_iq_i^T)(\sum_j x_jq_j) = \sum_{i=j} (x_iq_i^T)(x_jq_j) + \sum_{i \ne j} (x_iq_i^T)(x_jq_j)$

$= \sum_{i=j} x_ix_jq_i^Tq_j + \sum_{i \ne j} x_ix_jq_i^Tq_j$

But since $q_1, q_2, \dots, q_n$ are orthonormal vectors we have $q_i^Tq_j = 1$ when $i = j$ and $q_i^Tq_j = 0$ when $i \ne j$, so that

$\sum_{i=j} x_ix_jq_i^Tq_j + \sum_{i \ne j} x_ix_jq_i^Tq_j = \sum_{i = j} x_ix_j \cdot 1 + \sum_{i \ne j} x_ix_j \cdot 0 = \sum_i x_i^2$

We thus have

$\|b\|^2 = \sum_i x_i^2 = x_1^2 + x_2^2 + \cdots + x_n^2$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged | Leave a comment

## Linear Algebra and Its Applications, Exercise 3.4.6

Exercise 3.4.6. Given the matrix

$Q = \begin{bmatrix} \frac{1}{\sqrt{3}}&\frac{1}{\sqrt{14}}&\qquad \\ \frac{1}{\sqrt{3}}&\frac{2}{\sqrt{14}}&\qquad \\ \frac{1}{\sqrt{3}}&-\frac{3}{\sqrt{14}}&\qquad \end{bmatrix}$

find entries for the third column such that $Q$ is orthogonal. How much freedom do you have to choose the entries? Finally, verify that both the columns and rows are orthonormal.

Answer: In order for $Q$ to be orthogonal all three of its rows must be orthonormal, with length of 1. We start with the vector forming the first row of $Q$.  For its length to be 1 the square of the last entry of the first row must be equal to

$1 - (\frac{1}{\sqrt{3}})^2 - (\frac{1}{\sqrt{14}})^2 = 1 - \frac{1}{3} - \frac{1}{14} = 1 - \frac{14}{42} - \frac{3}{42} = \frac{25}{42}$

so that the entry itself would be $\pm \sqrt{\frac{25}{42}} = \pm \frac{5}{\sqrt{42}}$.

Similarly the square of the last entry of the second row must be

$1 - (\frac{1}{\sqrt{3}})^2 - (\frac{2}{\sqrt{14}})^2 = 1 - \frac{1}{3} - \frac{4}{14} = 1 - \frac{14}{42} - \frac{12}{42} = \frac{16}{42}$

so that the entry itself would be $\pm \sqrt{\frac{16}{42}} = \pm \frac{4}{\sqrt{42}}$.

Finally, the square of the last entry of the third row must be

$1 - (\frac{1}{\sqrt{3}})^2 - (\frac{3}{\sqrt{14}})^2 = 1 - \frac{1}{3} - \frac{9}{14} = 1 - \frac{14}{42} - \frac{27}{42} = \frac{1}{42}$

so that the entry itself would be $\pm \sqrt{\frac{1}{42}} = \pm \frac{1}{\sqrt{42}}$.

We have two possible choices for the last entry of the first row, $\frac{5}{\sqrt{42}}$ and $-\frac{5}{\sqrt{42}}$. Suppose that we choose $\frac{5}{\sqrt{42}}$. Since the first row and the second row are supposed to be orthogonal, we must choose $-\frac{4}{\sqrt{42}}$ for the last entry of the second row, so that the dot product of the two rows is zero:

$\frac{1}{\sqrt{3}} \cdot \frac{1}{\sqrt{3}} + \frac{1}{\sqrt{14}} \cdot \frac{2}{\sqrt{14}} + \frac{5}{\sqrt{42}} \cdot (-\frac{4}{\sqrt{42}}) = \frac{1}{3} + \frac{2}{14} - \frac{20}{42} = \frac{14}{42} + \frac{6}{42} - \frac{20}{42} = 0$

We must then choose $-\frac{1}{\sqrt{42}}$ for the last entry in the third row, so that the dot product of the first row and the third row is zero:

$\frac{1}{\sqrt{3}} \cdot \frac{1}{\sqrt{3}} + \frac{1}{\sqrt{14}} \cdot (-\frac{3}{\sqrt{14}}) + \frac{5}{\sqrt{42}} \cdot (-\frac{1}{\sqrt{42}}) = \frac{1}{3} - \frac{3}{14} - \frac{5}{42} = \frac{14}{42} - \frac{9}{42} - \frac{5}{42} = 0$

and the dot product of the second and third rows is zero:

$\frac{1}{\sqrt{3}} \cdot \frac{1}{\sqrt{3}} + \frac{2}{\sqrt{14}} \cdot (-\frac{3}{\sqrt{14}}) + (-\frac{4}{\sqrt{42}}) \cdot (-\frac{1}{\sqrt{42}}) = \frac{1}{3} - \frac{6}{14} + \frac{4}{42} = \frac{14}{42} - \frac{18}{42} + \frac{4}{42} = 0$

The resulting value for the matrix $Q$ is

$Q = \begin{bmatrix} \frac{1}{\sqrt{3}}&\frac{1}{\sqrt{14}}&\frac{5}{\sqrt{42}} \\ \frac{1}{\sqrt{3}}&\frac{2}{\sqrt{14}}&-\frac{4}{\sqrt{42}} \\ \frac{1}{\sqrt{3}}&-\frac{3}{\sqrt{14}}&-\frac{1}{\sqrt{42}} \end{bmatrix}$

The dot product of the first column with itself is

$(\frac{1}{\sqrt{3}})^2 + (\frac{1}{\sqrt{3}})^2 + (\frac{1}{\sqrt{3}})^2 = \frac{1}{3} + \frac{1}{3} + \frac{1}{3} = 1$

The dot product of the second column with itself is

$(\frac{1}{\sqrt{14}})^2 + (\frac{2}{\sqrt{14}})^2 + (-\frac{3}{\sqrt{14}})^2 = \frac{1}{14} + \frac{4}{14} + \frac{9}{14} = 1$

The dot product of the third column with itself is

$(\frac{5}{\sqrt{42}})^2 + (-\frac{4}{\sqrt{42}})^2 + (-\frac{1}{\sqrt{42}})^2 = \frac{25}{42} + \frac{16}{42} + \frac{1}{42} = 1$

Thus all three columns have length 1.

We also have the dot product of the first and second columns as zero:

$\frac{1}{\sqrt{3}} \cdot \frac{1}{\sqrt{14}} + \frac{1}{\sqrt{3}} \cdot \frac{2}{\sqrt{14}} + \frac{1}{\sqrt{3}} \cdot (-\frac{3}{\sqrt{14}}) = \frac{1}{\sqrt{42}} + \frac{2}{\sqrt{42}} - \frac{3}{\sqrt{42}} = 0$

the dot product of the first and third columns as zero:

$\frac{1}{\sqrt{3}} \cdot \frac{5}{\sqrt{42}} + \frac{1}{\sqrt{3}} \cdot (-\frac{4}{\sqrt{42}}) + \frac{1}{\sqrt{3}} \cdot (-\frac{1}{\sqrt{42}}) = \frac{5}{\sqrt{126}} - \frac{4}{\sqrt{126}} - \frac{1}{\sqrt{126}} = 0$

and the dot product of the second and third columns as zero:

$\frac{1}{\sqrt{14}} \cdot \frac{5}{\sqrt{42}} + \frac{2}{\sqrt{14}} \cdot (-\frac{4}{\sqrt{42}}) + (-\frac{3}{\sqrt{14}}) \cdot (-\frac{1}{\sqrt{42}}) = \frac{5}{\sqrt{588}} - \frac{8}{\sqrt{588}} + \frac{3}{\sqrt{588}} = 0$

Since all three rows are orthonormal (by construction) and all three columns are orthonormal, the matrix $Q$ is an orthogonal matrix.

Recall that we originally had two choices for the last entry of the first row. If we instead choose $-\frac{5}{\sqrt{42}}$ for the last entry in the first row, we must choose $\frac{4}{\sqrt{42}}$ for the last entry of the second row and $\frac{1}{\sqrt{42}}$ for the last entry in the third row, so that

$Q = \begin{bmatrix} \frac{1}{\sqrt{3}}&\frac{1}{\sqrt{14}}&-\frac{5}{\sqrt{42}} \\ \frac{1}{\sqrt{3}}&\frac{2}{\sqrt{14}}&\frac{4}{\sqrt{42}} \\ \frac{1}{\sqrt{3}}&-\frac{3}{\sqrt{14}}&\frac{1}{\sqrt{42}} \end{bmatrix}$

Verifying that this alternative value for $Q$ is an orthogonal matrix is left as an exercise for the reader.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged | Leave a comment

## Linear Algebra and Its Applications, Exercise 3.4.5

Exercise 3.4.5. Given a unit vector $u$ and $Q = I - 2uu^T$, prove that $Q$ is orthogonal. What is $Q$ when $u = (\frac{1}{2},\frac{1}{2}, -\frac{1}{2}, -\frac{1}{2})$?

$Q^TQ = (I - 2uu^T)^T(I - 2uu^T) = (I^T - 2(uu^T)^T)(I - 2uu^T)$

$= (I - 2(u^T)^Tu^T)(I - 2uu^T) = (I - 2uu^T)(I - 2uu^T)$

$= I \cdot I - 2Iuu^T - 2uu^TI + 4(uu^T)(uu^T) = I - 4uu^T +4u(u^Tu)u^T$

$= I - 4uu^T + 4uIu^T = I - 4uu^T + 4uu^T = I$

Since $Q^TQ = I$ the matrix $Q = I - 2uu^T$ is orthogonal.

If $u = (\frac{1}{2},\frac{1}{2}, -\frac{1}{2}, -\frac{1}{2})$ then

$uu^T = \begin{bmatrix} \frac{1}{2} \\ \frac{1}{2} \\ -\frac{1}{2} \\ -\frac{1}{2} \end{bmatrix} \begin{bmatrix} \frac{1}{2}&\frac{1}{2}&-\frac{1}{2}&-\frac{1}{2} \end{bmatrix} = \begin{bmatrix} \frac{1}{4}&\frac{1}{4}&-\frac{1}{4}&-\frac{1}{4} \\ \frac{1}{4}&\frac{1}{4}&-\frac{1}{4}&-\frac{1}{4} \\ -\frac{1}{4}&-\frac{1}{4}&\frac{1}{4}&\frac{1}{4} \\ -\frac{1}{4}&-\frac{1}{4}&\frac{1}{4}&\frac{1}{4} \end{bmatrix}$

so that

$Q = I - 2uu^T = \begin{bmatrix} 1&0&0&0 \\ 0&1&0&0 \\ 0&0&1&0 \\ 0&0&0&1 \end{bmatrix} - 2 \cdot \begin{bmatrix} \frac{1}{4}&\frac{1}{4}&-\frac{1}{4}&-\frac{1}{4} \\ \frac{1}{4}&\frac{1}{4}&-\frac{1}{4}&-\frac{1}{4} \\ -\frac{1}{4}&-\frac{1}{4}&\frac{1}{4}&\frac{1}{4} \\ -\frac{1}{4}&-\frac{1}{4}&\frac{1}{4}&\frac{1}{4} \end{bmatrix}$

$= \begin{bmatrix} 1&0&0&0 \\ 0&1&0&0 \\ 0&0&1&0 \\ 0&0&0&1 \end{bmatrix} - \begin{bmatrix} \frac{1}{2}&\frac{1}{2}&-\frac{1}{2}&-\frac{1}{2} \\ \frac{1}{2}&\frac{1}{2}&-\frac{1}{2}&-\frac{1}{2} \\ -\frac{1}{2}&-\frac{1}{2}&\frac{1}{2}&\frac{1}{2} \\ -\frac{1}{2}&-\frac{1}{2}&\frac{1}{2}&\frac{1}{2} \end{bmatrix} = \begin{bmatrix} \frac{1}{2}&-\frac{1}{2}&\frac{1}{2}&\frac{1}{2} \\ -\frac{1}{2}&\frac{1}{2}&\frac{1}{2}&\frac{1}{2} \\ \frac{1}{2}&\frac{1}{2}&\frac{1}{2}&-\frac{1}{2} \\ \frac{1}{2}&\frac{1}{2}&-\frac{1}{2}&\frac{1}{2} \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged | Leave a comment

## Linear Algebra and Its Applications, Exercise 3.4.4

Exercise 3.4.4. Given two orthogonal matrices $Q_1$ and $Q_2$, show that their product $Q_1Q_2$ is also orthogonal. If $Q_1$ represents rotation through the angle $\theta$ and $Q_1$ represents rotation through the angle $\phi$, what does $Q_1Q_2$ represent? What trigonometric identities for $\sin (\theta + \phi)$ and $\cos (\theta + \phi)$ can be found in multiplying $Q_1$ and $Q_2$?

Answer: For any orthogonal matrix $Q$ we have $Q^TQ = I$. We then have

$(Q_1Q_2)^T(Q_1Q2) = (Q_2^TQ_1^T)(Q_1Q2) = Q_2^T(Q_1^TQ_1)Q_2 = Q_2^TQ_2 = I$

Since $(Q_1Q_2)^T(Q_1Q2) = I$ the matrix $Q_1Q_2$ is orthogonal.

If the matrices $Q_1$ and $Q_2$ represent rotations through $\theta$ and $\phi$ respectively, then $Q_1Q_2$ represents rotation through $\theta + \phi$, and will contain elements including $\sin (\theta + \phi)$ and $\cos (\theta + \phi)$. These will be produced in the matrix multiplication through the following trigonometric identities:

$\sin (\theta + \phi) = \sin \theta \cos \phi + \cos \theta \sin \phi$

$\cos (\theta + \phi) = \cos \theta \cos \phi - \sin \theta \sin \phi$

where $\sin \theta$ and $\cos \theta$ come from $Q_1$  and $\sin \phi$ and $\cos \phi$ come from $Q_2$.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

## Linear Algebra and Its Applications, Exercise 3.4.3

Exercise 3.4.3. Given the orthonormal vectors $a_1 = (\frac{2}{3}, \frac{2}{3}, -\frac{1}{3})$ and $a_2 = (-\frac{1}{3}, \frac{2}{3}, \frac{2}{3})$ and the vector $b = (0, 3, 0)$ from the previous exercise, project $b$ onto a third orthonormal vector $a_3 = (\frac{2}{3}, -\frac{1}{3}, \frac{2}{3})$. What is the sum of the three projections? Why? Why is the matrix $P = a_1a_1^T + a_2a_2^T + a_3a_3^T$ equal to the identity matrix $I$?

Answer: Since $a_3$ is orthonormal, the projection of $b$ onto $a_3$ is given by

$a_3^Tba_3 = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 0 \\ 3 \\ 0 \end{bmatrix} \begin{bmatrix} \frac{2}{3} \\ -\frac{1}{3} \\ \frac{2}{3} \end{bmatrix} = -1 \cdot \begin{bmatrix} \frac{2}{3} \\ -\frac{1}{3} \\ \frac{2}{3} \end{bmatrix} = \begin{bmatrix} -\frac{2}{3} \\ \frac{1}{3} \\ -\frac{2}{3} \end{bmatrix}$

The sum of all three projections is then:

$a_1^Tba_1 + a_2^Tba_2 + a_3^Tba_3 = \begin{bmatrix} \frac{4}{3} \\ \frac{4}{3} \\ -\frac{2}{3} \end{bmatrix} + \begin{bmatrix} -\frac{2}{3} \\ \frac{4}{3} \\ \frac{4}{3} \end{bmatrix} + \begin{bmatrix} -\frac{2}{3} \\ \frac{1}{3} \\ -\frac{2}{3} \end{bmatrix} = \begin{bmatrix} 0 \\ 3 \\ 0 \end{bmatrix} = b$

This is because the vectors $a_1$, $a_2$, and $a_3$ together form an orthonormal basis for $\mathbb{R}^3$, and any vector $v$ in $\mathbb{R}^3$ can be expressed as a linear combination of those basis vectors with coefficients given by the projection of $v$ onto each basis vector.

We then have

$b = a_1^Tba_1 + a_2^Tba_2 + a_3^Tba_3 = a_1a_1^Tb + a_2a_2^Tb + a_3a_3^Tb = (a_1a_1^T + a_2a_2^T + a_3a_3^T)b$

(taking advantage of the fact that $a_1^Tb$, etc., are scalars) so that the 3 by 3 matrix

$P = a_1a_1^T + a_2a_2^T + a_3a_3^T = I$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.4.2

Exercise 3.4.2. Given two orthonormal vectors $a_1 = (\frac{2}{3}, \frac{2}{3}, -\frac{1}{3})$ and $a_2 = (-\frac{1}{3}, \frac{2}{3}, \frac{2}{3})$ and the vector $b = (0, 3, 0)$, project $b$ onto $a_1$ and $a_2$. Also find the projection $p$ of $b$ onto the plane formed by $a_1$ and $a_2$.

Answer: Since $a_1$ is orthonormal, the projection of $b$ onto $a_1$ is given by

$a_1^Tba_1 = \begin{bmatrix} \frac{2}{3}&\frac{2}{3}&-\frac{1}{3} \end{bmatrix} \begin{bmatrix} 0 \\ 3 \\ 0 \end{bmatrix} \begin{bmatrix} \frac{2}{3} \\ \frac{2}{3} \\ -\frac{1}{3} \end{bmatrix} = 2 \cdot \begin{bmatrix} \frac{2}{3} \\ \frac{2}{3} \\ -\frac{1}{3} \end{bmatrix} = \begin{bmatrix} \frac{4}{3} \\ \frac{4}{3} \\ -\frac{2}{3} \end{bmatrix}$

Similarly the projection of $b$ onto $a_2$ is given by

$a_2^Tba_2 = \begin{bmatrix} -\frac{1}{3}&\frac{2}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 0 \\ 3 \\ 0 \end{bmatrix} \begin{bmatrix} -\frac{1}{3} \\ \frac{2}{3} \\ \frac{2}{3} \end{bmatrix} = 2 \cdot \begin{bmatrix} -\frac{1}{3} \\ \frac{2}{3} \\ \frac{2}{3} \end{bmatrix} = \begin{bmatrix} -\frac{2}{3} \\ \frac{4}{3} \\ \frac{4}{3} \end{bmatrix}$

The projection $p$ of $b$ onto the plane formed by $a_1$ and $a_2$ is simply the sum of the projections above:

$p = a_1^Tba_1 + a_2^Tba_2 = \begin{bmatrix} \frac{4}{3} \\ \frac{4}{3} \\ -\frac{2}{3} \end{bmatrix} + \begin{bmatrix} -\frac{2}{3} \\ \frac{4}{3} \\ \frac{4}{3} \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ \frac{8}{3} \\ \frac{2}{3} \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.4.1

Exercise 3.4.1. a) Given the following four data points:

$y = -4 \quad\textrm{at}\quad t = -2 \quad\textrm{and}\quad y = -3 \quad\textrm{at}\quad t = -1$

$y = -1 \quad\textrm{at}\quad t = 1 \quad\textrm{and}\quad y = 0 \quad\textrm{at}\quad t = 2$

write down the four equations for fitting $C + Dt$ to the data.

b) Find the line fit by least squares and calculate the error $E^2$.

c) Given the value of $E^2$ what is $b$ in relation to the column space? What is the projection $p$ of $b$ on the column space?

Answer: a) This corresponds to a system of the form $Ax = b$ as follows:

$\begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&1 \\ 1&2 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} -4 \\ -3 \\ -1 \\ 0 \end{bmatrix}$

The equivalent system of equations is:

$C - 2D = -4$

$C - D = -3$

$C + D = -1$

$C + 2D = 0$

b) To find the least squares solution we multiply both sides by $A^T$ to create a system of the form $A^TA\bar{x} = A^Tb$. We have

$A^TA = \begin{bmatrix} 1&1&1&1 \\ -2&-1&1&2 \end{bmatrix} \begin{bmatrix} 1&-2 \\ 1&-1 \\ 1&1 \\ 1&2 \end{bmatrix} = \begin{bmatrix} 4&0 \\ 0&10 \end{bmatrix}$

and

$A^Tb = \begin{bmatrix} 1&1&1&1 \\ -2&-1&1&2 \end{bmatrix} \begin{bmatrix} -4 \\ -3 \\ -1 \\ 0 \end{bmatrix} = \begin{bmatrix} -8 \\ 10 \end{bmatrix}$

so that the new system is

$\begin{bmatrix} 4&0 \\ 0&10 \end{bmatrix} \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = \begin{bmatrix} -8 \\ 10 \end{bmatrix}$

From the second equation we have $\bar{D} = \frac{10}{10} = 1$ and from the first equation we have $\bar{C} = \frac{-8}{4} = -2$.

The resulting graph is a line $-2 + t$ with slope of 1 and $y$-intercept of -2. For the values of $t$ of -2, -1, 1, and 2 the values of $-2 + t$ are -4, -3, -1, and 0 respectively. But these are exactly the same as the values of $y$ in the given data points. Therefore we have $E^2 = 0$.

c) Since $E^2 = 0$ the vector $b$ must be in the column space of $A$, and the projection $p$ of $b$ onto the column space is simply $b$ itself.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.