## Linear Algebra and Its Applications, Exercise 3.3.6

Exercise 3.3.6. Given the matrix $A$ and vector $b$ defined as follows

$A = \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix}$

find the projection of $b$ onto the column space of $A$.

Decompose the vector $b$ into the sum $p + q$ of two orthogonal vectors $p$ and $q$ where $p$ is in the column space. Which subspace is $q$ in?

Answer: We have $p = A(A^TA)^{-1}A^Tb$ per equation (3) of 3L on page 156. We first compute

$A^TA = \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix}$

$= \begin{bmatrix} 9&-9 \\ -9&18 \end{bmatrix}$

and then compute

$(A^TA)^{-1} = \frac{1}{9 \cdot 18 - (-9)(-9)} \begin{bmatrix} 18&-(-9) \\ -(-9)&9 \end{bmatrix}$

$= \frac{1}{81} \begin{bmatrix} 18&9 \\ 9&9 \end{bmatrix} = \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix}$

Finally we compute

$p = A(A^TA)^{-1}A^Tb$

$= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix} \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix}$

$= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix} \begin{bmatrix} -9 \\ 27 \end{bmatrix}$

$= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \\ 6 \end{bmatrix}$

Since $b = p + q$ we have

$q = p - b = \begin{bmatrix} 3 \\ 0 \\ 6 \end{bmatrix} - \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix} = \begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix}$

We have

$p \cdot q = 3 \cdot (-2) + 0 \cdot 2 + 6 \cdot 1 = -6 + 0 + 6 = 0$

so that $p$ and $q$ are orthogonal.

The vector $p$ is in $\cal{R}(A)$, the column space of A, and the orthogonal subspace of $\cal{R}(A)$ is $\cal{N}(A^T)$, the left nullspace of $A$. Since $p$ is in $\cal{R}(A)$ and $q$ is orthogonal to $p$, $q$ must be in $\cal{N}(A^T)$, so that $A^Tq = 0$. We confirm this:

$A^Tq = \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix}$

$= \begin{bmatrix} -2 + 4 -2 \\ -2 - 2 + 4 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} = 0$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | | 2 Comments

## Linear Algebra and Its Applications, Exercise 3.3.5

Exercise 3.3.5. Given the system

$Ax = \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 4 \\ 5 \\ 9 \end{bmatrix} = b$

with no solution, provide a graph of a straight line that minimizes

$(C - D -4)^2 + (C - 5)^2 + (C + D - 9)^2$

and solve for the equation of the line. What is the result of projecting the vector $b$ onto the column space of $A$?

$(C - D -4)^2 + (C - 5)^2 + (C + D - 9)^2 = \| Ax - b \|^2$

In other words, this problem is essentially that of finding $\bar{x} = \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix}$, the least squares solution to $Ax = b$ that minimizes the error vector $E^2 = \| Ax - b \|^2$.

From 3L on page 156 we have $\bar{x} = (A^TA)^{-1}A^Tb$. In this case we have

$A^TA = \begin{bmatrix} 1&1&1 \\ -1&0&1 \end{bmatrix} \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 3&0 \\ 0&2 \end{bmatrix}$

and thus

$(A^TA)^{-1} = \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix}$

so that

$\bar{x} = (A^TA)^{-1}A^Tb = \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix} \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} 4 \\ 5 \\ 9 \end{bmatrix}$

$= \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix} \begin{bmatrix} 18 \\ 5 \end{bmatrix} = \frac{1}{6} \begin{bmatrix} 36 \\ 15 \end{bmatrix} = \begin{bmatrix} 6 \\ \frac{5}{2} \end{bmatrix}$

Thus the line that minimizes

$E^2 = \|Ax -b\|^2$

$= (C - D - 4)^2 + (C - 5)^2 + (C + D - 9)^2$

is $\bar{C} + \bar{D}t$ with  slope $\bar{D} = \frac{5}{2}$ and intercept $\bar{C} = 6$.

Also from 3L on page 156 the projection of $b$ onto the column space of $A$ is

$p = A\bar{x} = \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} 6 \\ \frac{5}{2} \end{bmatrix} = \begin{bmatrix} \frac{7}{2} \\ 6 \\ \frac{17}{2} \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.3.4

Exercise 3.3.4. Given

$A = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \qquad x = \begin{bmatrix} u \\ v \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix}$

expand the expression $E^2 = \| Ax - b \|^2$, compute its partial derivatives with respect to $u$ and $v$, and set them to zero. Compare the resulting equations to $A^TA\bar{x} = A^Tb$ to confirm that you obtain the same normal equations in both cases (i.e., using calculus vs. geometry). Then find the projection $p = A\bar{x}$ and explain why $p = b$.

$Ax-b = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} - \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix}$

$= \begin{bmatrix} u \\ v \\ u+v \end{bmatrix} - \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} u-1 \\ v-3 \\ (u+v)-4 \end{bmatrix}$

so that

$E^2 = \| Ax-b \| = \left(Ax-b\right)^T\left(Ax-b\right)$

$= (u-1)^2 + (v-3)^2 + [(u+v)-4]^2$

$= (u^2-2u+1) + (v^2-6v+9) +[ (u+v)^2 - 8(u+v) + 16]$

$= u^2-2u+1+v^2-6v+9+u^2+2uv+v^2-8u-8v+16$

$= 2u^2+2v^2+2uv-10u-14v+26$

Taking the partial derivative of this expression with respect to $u$ we have

$\partial E^2/{\partial u} = \partial/{\partial u}(2u^2+2v^2+2uv-10u-14v+26)$

$= 4u+2v-10$

and setting it to zero we have

$4\bar{u}+2\bar{v}-10 = 0$

where $\bar{u}$ and $\bar{v}$ represent the least squares solution.

Taking the partial derivative of this expression with respect to $v$ we have

$\partial E^2/{\partial v} = \partial/{\partial v}(2u^2+2v^2+2uv-10u-14v+26)$

$= 4v+2u-14$

and setting it to zero we have

$4\bar{v}+2\bar{u}-14 = 0$

where again $\bar{u}$ and $\bar{v}$ represent the least squares solution.

We then have a system of two equations

$\begin{array}{rcrcr} 4\bar{u}&+&2\bar{v}&=&10 \\ 2\bar{u}&+&4\bar{v}&=&14 \end{array}$

that is equivalent to the matrix equation

$\begin{bmatrix} 4&2 \\ 2&4 \end{bmatrix} \begin{bmatrix} \bar{u} \\ \bar{v}\end{bmatrix} = \begin{bmatrix} 10 \\ 14 \end{bmatrix}$

We now compare this to the normal equations $A^TA\bar{x} = A^Tb$. We have

$A^TA = \begin{bmatrix} 1&0&0 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$

and

$A^Tb = \begin{bmatrix} 1&0&0 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}$

so that the normal equation in matrix form is

$\begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix} \begin{bmatrix} \bar{u} \\ \bar{v} \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}$

Note that this equation is the same as the previous equation calculated from taking the derivatives, except with both sides divided by 2.

We can now solve for $\bar{x} = \left(A^TA\right)^{-1}A^Tb$. We have

$\left(A^TA\right) = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}^{-1} = \frac{1}{2 \cdot 2 - 1 \cdot 1} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix}$

$= \frac{1}{3} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix}$

so that

$\bar{x} = \left(A^TA\right)^{-1}A^Tb = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 5 \\ 7 \end{bmatrix}$

$= \begin{bmatrix} \frac{10}{3}-\frac{7}{3} \\ -\frac{5}{3}+\frac{14}{3} \end{bmatrix} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}$

We can multiply $\bar{x}$ by $A$ to obtain the projection $p$ of $b$ onto the column space of $A$:

$p = A\bar{x} = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 3 \end{bmatrix} = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = b$

We have $p = b$ because $b$ is already in the column space of $A$; in particular, $b$ is equal to the first column of $A$ plus 3 times the second column of $A$:

$b = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} + 3 \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.3.3

Exercise 3.3.3. Given

$A = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}$

solve $Ax = b$ to find $p = A\bar{x}$. Show that $b-p$ is orthogonal to every column in $A$.

Answer: We have $A^TA\bar{x} = A^Tb$ or $\bar{x} = \left(A^TA\right)^{-1}A^Tb$ if $A^TA$ is invertible. In this case we have

$A^TA = \begin{bmatrix} 1&0&1 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}$

so that

$\left(A^TA\right)^{-1} = \frac{1}{2 \cdot 2 - 1 \cdot 1} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix}$

$= \frac{1}{3} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix}$

We then have

$\bar{x} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1&0&1 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}$

$= \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \end{bmatrix}$

and

$p = A\bar{x} = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \frac{2}{3} \end{bmatrix}$

The error vector is then

$b - p = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} - \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \frac{2}{3} \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ \frac{2}{3} \\ -\frac{2}{3} \end{bmatrix}$

The inner product of the error vector $b-p$ with column 1 of $A$ is

$\begin{bmatrix} \frac{2}{3}&\frac{2}{3}&-\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} = \frac{2}{3} + 0 - \frac{2}{3} = 0$

The inner product of $b-p$ with column 2 of $A$ is

$\begin{bmatrix} \frac{2}{3}&\frac{2}{3}&-\frac{2}{3} \end{bmatrix} \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} = 0 + \frac{2}{3} - \frac{2}{3} = 0$

Thus $b-p$ is othogonal to all columns of $A$ (and thus to the column space of $A$ as well).

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.3.2

Exercise 3.3.2. We have the value $b_1 = 1$ at time $t_1 = 1$ and the value $b_2 = 7$ at time $t_2 = 2$. We wish to fit these values using a line constrained to go through the origin, i.e., with an equation of the form $b = Dt$. Solve the system using least squares and describe the best fit line.

Answer: In general we would fit a line of the form $C + Dt = b$; however since we want the fitted line to go through the origin we have $C = 0$ and are fitting a line of the form $0 + Dt = b$. Given the values of $b_1$ and $b_2$ at $t_1 = 1$ and $t_2 = 2$ respectively we have the following system of two equations:

$\begin{array}{rcrcr} 0&+&D&=&1 \\ 0&+&2D&=&7 \end{array}$

This system corresponds to the general matrix equation

$A \begin{bmatrix} C \\ D \end{bmatrix} = b$

which in this case becomes

$\begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ D \end{bmatrix} = \begin{bmatrix} 1 \\ 7 \end{bmatrix}$

This system has no solution. However we can find a least squares solution in general using the matrix equation

$A^TA \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = A^Tb$

In this case this corresponds to the equation

$\begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 1 \\ 7 \end{bmatrix}$

or

$\begin{bmatrix} 0&0 \\ 0&5 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0 \\ 15 \end{bmatrix}$

We then have $5 \bar{D} = 15$ or $\bar{D} = 3$.

So the least squares solution is the line $b = 3t$. This line passes through the origin (i.e., at $t = 0$ it has a value of zero). At time $t_1 = 1$ it has the value 3, which is greater than the value $b_1 = 1$. At time $t_2 = 2$ it has the value 6, which is less than the value $b_2 = 7$.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.3.1

Exercise 3.3.1. a) Given the system of equations consisting of $3x = 10$ and $4x = 5$ find the least squares solution $\bar{x}$ and describe the error $E^2$ being minimized by that solution. Confirm that the error vector $\left(10-3\bar{x}, 5-4\bar{x}\right)$ is orthogonal to the column $\left(3, 4\right)$.

Answer: This is a system $Ax = b$ where $A = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}$. We can then compute the least squares solution as $\bar{x} = \left(A^TA\right)^{-1}A^Tb$.

We have

$A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25$

so that $\left(A^TA\right)^{-1} = \frac{1}{25}$. We then have

$\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}$

$= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2$

So the least squares solution is $\bar{x} = 2$.

The error vector is then

$b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}$

and the corresponding error

$E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)$

$= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25$

The inner product of the error vector $b-A\bar{x} = \left(4, -3\right)$ and the column $\left(3, 4\right)$ of $A$ is $4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0$. Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

## Linear Algebra and Its Applications, Exercise 3.2.16

Exercise 3.2.16. a) Given the projection matrix $P$ projecting vectors onto the line through $a$ and two vectors $x$ and $y$, show that the inner products of $x$ with $Py$ and $y$ with $Px$ are equal.

b) In general would the angles between $x$ and $a$ and $y$ and $a$ be equal to each other? If $a = \left(1, 1, -1\right)$, $x = \left(2, 0, 1\right)$, and $y = \left(2, 1, 2\right)$ what are the cosines of the two angles?

c) Show that the inner product of $Px$ and $Py$ is the same as the inner products of $x$ with $Py$ and $y$ with $Px$ and explain why this is. What is the angle between the vectors $Px$ and $Py$?

Answer: a) We have $P = aa^T/a^Ta$ so that

$Px = \left(aa^T/a^Ta\right)x = \frac{1}{a^Ta}\left(aa^Tx\right) = \frac{a^Tx}{a^Ta}a$

and

$Py = \left(aa^T/a^Ta\right)y = \frac{1}{a^Ta}\left(aa^Ty\right) = \frac{a^Ty}{a^Ta}a$

The inner product of $x$ with $Py$ is then

$x^T\left(Py\right) = x^T\left(a^Ty/a^Ta\right)a$

$= \left(a^Ty/a^Ta\right)x^Ta = \left(a^Tyx^Ta\right)/a^Ta$

$= \left(a^Tya^Tx\right)/a^Ta = \left(a^Txa^Ty\right)/a^Ta$

and the inner product of $y$ with $Px$ is then

$y^T\left(Px\right) = y^T\left(a^Tx/a^Ta\right)a$

$= \left(a^Tx/a^Ta\right)y^Ta = \left(a^Txy^Ta\right)/a^Ta$

$= \left(a^Txa^Ty\right)/a^Ta$

So the inner products of $x$ with $Py$ and $y$ with $Px$ are equal.

b) Since $x$ and $y$ are arbitrary vectors, in general they would not make the same angle with respect to $a$. For example, consider the case when $a = \left(1, 1, -1\right)$, $x = \left(2, 0, 1\right)$, and $y = \left(2, 1, 2\right)$.

If $\theta_1$ is the angle between $x$ and $a$ then $\cos \theta_1 = a^Tx/\left(\|a\|\|x\|\right)$. Similarly, if $\theta_2$ is the angle between $y$ and $a$ then $\cos \theta_2 = a^Ty/\left(\|a\|\|y\|\right)$

In this case we have

$a^Tx = 1 \cdot 2 + 1 \cdot 1 - 1 \cdot 2 = 2 + 1 - 2 = 1$

$a^Ty = 1 \cdot 2 + 1 \cdot 0 - 1 \cdot 1 = 2 + 0 - 1 = 1$

$\|a\| = \sqrt{1^2 + 1^2 + \left(-1\right)^2} = \sqrt{1 + 1 + 1} = \sqrt{3}$

$\|x\| = \sqrt{2^2 + 0^2 + 1^2} = \sqrt{4 + 0 + 1} = \sqrt{5}$

$\|y\| = \sqrt{2^2 + 1^2 + 2^2} = \sqrt{4 + 1 + 4} = \sqrt{9} = 3$

So $\cos \theta_1 = \frac{1}{\sqrt{3}\sqrt{5}} = \frac{1}{\sqrt{15}}$ and $\cos \theta_2 = \frac{1}{\sqrt{3} \cdot 3} = \frac{1}{3\sqrt{3}}$. The two cosines are different and thus $\theta_1$ and $\theta_2$ are not equal to each other.

c) As noted above we have $Px = \frac{a^Tx}{a^Ta}a$ and $Py = \frac{a^Ty}{a^Ta}a$. Their inner product is then

$\left(Px\right)^T\left(Py\right) = \left(\frac{a^Tx}{a^Ta}a\right)^T \left(\frac{a^Ty}{a^Ta}a\right)$

$\frac{a^Tx}{a^Ta} \frac{a^Ty}{a^Ta} \left(a^Ta\right) = \left(a^Txa^Ty\right)/a^Ta$

But this is the same as the inner products $x^TPy$ and $y^TPx$ of $x$ with $Py$ and $y$ with $Px$ respectively.

This can be understood geometrically as follows: The vector $x$ can be thought of as consisting of two components $p_x$ and $q_x$ where $p_x$ is the projection of $x$ onto $a$ and $q_x$ is the projection of $x$ onto the vector $b$ that is orthogonal to $a$. The inner product between $x$ and $Py$ is then

$\left(Py\right)^Tx = \left(Py\right)^T\left(p_x+q_x\right)$

$= \left(Py\right)^Tp_x + \left(Py\right)^Tq_x$

But $p_x$ is simply $Px$. Also, since $Px$ is a projection on $a$ and $q_x$ is a projection onto $b$ their inner product is zero since $a$ is orthogonal to $b$. We therefore have $\left(Py\right)^Tx = \left(Py\right)^T\left(Px\right)$.

Similarly the vector $y$ can be thought of as consisting of two components $p_y = Py$ parallel to $a$ and $q_y$ orthogonal to $a$, with the inner product between $y$ and $Px$ then being

$\left(Px\right)^Ty = \left(Px\right)^T\left(p_y+q_y\right)$

$= \left(Px\right)^Tp_y + \left(Px\right)^Tq_y$

$= \left(Px\right)^TPy = \left(Py\right)^TPx$

We thus have $\left(Py\right)^Tx = \left(Py\right)^TPx = \left(Px\right)^Ty$.

Since $Px$ and $Py$ are both projections onto $a$ the angle between them is zero.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.