Linear Algebra and Its Applications, Exercise 3.3.6

Exercise 3.3.6. Given the matrix A and vector b defined as follows

A = \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix}

find the projection of b onto the column space of A.

Decompose the vector b into the sum p + q of two orthogonal vectors p and q where p is in the column space. Which subspace is q in?

Answer: We have p = A(A^TA)^{-1}A^Tb per equation (3) of 3L on page 156. We first compute

A^TA = \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix}

= \begin{bmatrix} 9&-9 \\ -9&18 \end{bmatrix}

and then compute

(A^TA)^{-1} = \frac{1}{9 \cdot 18 - (-9)(-9)} \begin{bmatrix} 18&-(-9) \\ -(-9)&9 \end{bmatrix}

= \frac{1}{81} \begin{bmatrix} 18&9 \\ 9&9 \end{bmatrix} = \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix}

Finally we compute

p = A(A^TA)^{-1}A^Tb

= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix} \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix}

= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} \frac{2}{9}&\frac{1}{9} \\ \frac{1}{9}&\frac{1}{9} \end{bmatrix} \begin{bmatrix} -9 \\ 27 \end{bmatrix}

= \begin{bmatrix} 1&1 \\ 2&-1 \\ -2&4 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \\ 6 \end{bmatrix}

Since b = p + q we have

q = p - b = \begin{bmatrix} 3 \\ 0 \\ 6 \end{bmatrix} - \begin{bmatrix} 1 \\ 2 \\ 7 \end{bmatrix} = \begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix}

We have

p \cdot q = 3 \cdot (-2) + 0 \cdot 2 + 6 \cdot 1 = -6 + 0 + 6 = 0

so that p and q are orthogonal.

The vector p is in \cal{R}(A), the column space of A, and the orthogonal subspace of \cal{R}(A) is \cal{N}(A^T), the left nullspace of A. Since p is in \cal{R}(A) and q is orthogonal to p, q must be in \cal{N}(A^T), so that A^Tq = 0. We confirm this:

A^Tq = \begin{bmatrix} 1&2&-2 \\ 1&-1&4 \end{bmatrix} \begin{bmatrix} -2 \\ 2 \\ 1 \end{bmatrix}

= \begin{bmatrix} -2 + 4 -2 \\ -2 - 2 + 4 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix} = 0

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , , , , | 2 Comments

Linear Algebra and Its Applications, Exercise 3.3.5

Exercise 3.3.5. Given the system

Ax = \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = \begin{bmatrix} 4 \\ 5 \\ 9 \end{bmatrix} = b

with no solution, provide a graph of a straight line that minimizes

(C - D -4)^2 + (C - 5)^2 + (C + D - 9)^2

and solve for the equation of the line. What is the result of projecting the vector b onto the column space of A?

Answer: We have

(C - D -4)^2 + (C - 5)^2 + (C + D - 9)^2 = \| Ax - b \|^2

In other words, this problem is essentially that of finding \bar{x} = \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix}, the least squares solution to Ax = b that minimizes the error vector E^2 = \| Ax - b \|^2 .

From 3L on page 156 we have \bar{x} = (A^TA)^{-1}A^Tb. In this case we have

A^TA = \begin{bmatrix} 1&1&1 \\ -1&0&1 \end{bmatrix} \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 3&0 \\ 0&2 \end{bmatrix}

and thus

(A^TA)^{-1} = \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix}

so that

\bar{x} = (A^TA)^{-1}A^Tb = \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix} \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} 4 \\ 5 \\ 9 \end{bmatrix}

= \frac{1}{6} \begin{bmatrix} 2&0 \\ 0&3 \end{bmatrix} \begin{bmatrix} 18 \\ 5 \end{bmatrix} = \frac{1}{6} \begin{bmatrix} 36 \\ 15 \end{bmatrix} = \begin{bmatrix} 6 \\ \frac{5}{2} \end{bmatrix}

Thus the line that minimizes

E^2 = \|Ax -b\|^2

= (C - D - 4)^2 + (C - 5)^2 + (C + D - 9)^2

is \bar{C} + \bar{D}t with  slope \bar{D} = \frac{5}{2} and intercept \bar{C} = 6.

Also from 3L on page 156 the projection of b onto the column space of A is

p = A\bar{x} = \begin{bmatrix} 1&-1 \\ 1&0 \\ 1&1 \end{bmatrix} \begin{bmatrix} 6 \\ \frac{5}{2} \end{bmatrix} = \begin{bmatrix} \frac{7}{2} \\ 6 \\ \frac{17}{2} \end{bmatrix}

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , , , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.3.4

Exercise 3.3.4. Given

A = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \qquad x = \begin{bmatrix} u \\ v \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix}

expand the expression E^2 = \| Ax - b \|^2, compute its partial derivatives with respect to u and v, and set them to zero. Compare the resulting equations to A^TA\bar{x} = A^Tb to confirm that you obtain the same normal equations in both cases (i.e., using calculus vs. geometry). Then find the projection p = A\bar{x} and explain why p = b.

Answer: We have

Ax-b = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} - \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix}

= \begin{bmatrix} u \\ v \\ u+v \end{bmatrix} - \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} u-1 \\ v-3 \\ (u+v)-4 \end{bmatrix}

so that

E^2 = \| Ax-b \| = \left(Ax-b\right)^T\left(Ax-b\right)

= (u-1)^2 + (v-3)^2 + [(u+v)-4]^2

= (u^2-2u+1) + (v^2-6v+9) +[ (u+v)^2 - 8(u+v) + 16]

= u^2-2u+1+v^2-6v+9+u^2+2uv+v^2-8u-8v+16

= 2u^2+2v^2+2uv-10u-14v+26

Taking the partial derivative of this expression with respect to u we have

\partial E^2/{\partial u} = \partial/{\partial u}(2u^2+2v^2+2uv-10u-14v+26)

= 4u+2v-10

and setting it to zero we have

4\bar{u}+2\bar{v}-10 = 0

where \bar{u} and \bar{v} represent the least squares solution.

Taking the partial derivative of this expression with respect to v we have

\partial E^2/{\partial v} = \partial/{\partial v}(2u^2+2v^2+2uv-10u-14v+26)

= 4v+2u-14

and setting it to zero we have

4\bar{v}+2\bar{u}-14 = 0

where again \bar{u} and \bar{v} represent the least squares solution.

We then have a system of two equations

\begin{array}{rcrcr} 4\bar{u}&+&2\bar{v}&=&10 \\ 2\bar{u}&+&4\bar{v}&=&14 \end{array}

that is equivalent to the matrix equation

\begin{bmatrix} 4&2 \\ 2&4 \end{bmatrix} \begin{bmatrix} \bar{u} \\ \bar{v}\end{bmatrix} = \begin{bmatrix} 10 \\ 14 \end{bmatrix}

We now compare this to the normal equations A^TA\bar{x} = A^Tb. We have

A^TA = \begin{bmatrix} 1&0&0 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}

and

A^Tb = \begin{bmatrix} 1&0&0 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}

so that the normal equation in matrix form is

\begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix} \begin{bmatrix} \bar{u} \\ \bar{v} \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}

Note that this equation is the same as the previous equation calculated from taking the derivatives, except with both sides divided by 2.

We can now solve for \bar{x} = \left(A^TA\right)^{-1}A^Tb. We have

\left(A^TA\right) = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}^{-1} = \frac{1}{2 \cdot 2 - 1 \cdot 1} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix}

= \frac{1}{3} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix}

so that

\bar{x} = \left(A^TA\right)^{-1}A^Tb = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 5 \\ 7 \end{bmatrix}

= \begin{bmatrix} \frac{10}{3}-\frac{7}{3} \\ -\frac{5}{3}+\frac{14}{3} \end{bmatrix} = \begin{bmatrix} 1 \\ 3 \end{bmatrix}

We can multiply \bar{x} by A to obtain the projection p of b onto the column space of A:

p = A\bar{x} = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 3 \end{bmatrix} = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = b

We have p = b because b is already in the column space of A; in particular, b is equal to the first column of A plus 3 times the second column of A:

b = \begin{bmatrix} 1 \\ 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} + 3 \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged , , , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.3.3

Exercise 3.3.3. Given

A = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \qquad b = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}

solve Ax = b to find p = A\bar{x}. Show that b-p is orthogonal to every column in A.

Answer: We have A^TA\bar{x} = A^Tb or \bar{x} = \left(A^TA\right)^{-1}A^Tb if A^TA is invertible. In this case we have

A^TA = \begin{bmatrix} 1&0&1 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} = \begin{bmatrix} 2&1 \\ 1&2 \end{bmatrix}

so that

\left(A^TA\right)^{-1} = \frac{1}{2 \cdot 2 - 1 \cdot 1} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix}

= \frac{1}{3} \begin{bmatrix} 2&-1 \\ -1&2 \end{bmatrix} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix}

We then have

\bar{x} = \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1&0&1 \\ 0&1&1 \end{bmatrix} \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}

= \begin{bmatrix} \frac{2}{3}&-\frac{1}{3} \\ -\frac{1}{3}&\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \end{bmatrix}

and

p = A\bar{x} = \begin{bmatrix} 1&0 \\ 0&1 \\ 1&1 \end{bmatrix} \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \frac{2}{3} \end{bmatrix}

The error vector is then

b - p = \begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix} - \begin{bmatrix} \frac{1}{3} \\ \frac{1}{3} \\ \frac{2}{3} \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ \frac{2}{3} \\ -\frac{2}{3} \end{bmatrix}

The inner product of the error vector b-p with column 1 of A is

\begin{bmatrix} \frac{2}{3}&\frac{2}{3}&-\frac{2}{3} \end{bmatrix} \begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix} = \frac{2}{3} + 0 - \frac{2}{3} = 0

The inner product of b-p with column 2 of A is

\begin{bmatrix} \frac{2}{3}&\frac{2}{3}&-\frac{2}{3} \end{bmatrix} \begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix} = 0 + \frac{2}{3} - \frac{2}{3} = 0

Thus b-p is othogonal to all columns of A (and thus to the column space of A as well).

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged , , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.3.2

Exercise 3.3.2. We have the value b_1 = 1 at time t_1 = 1 and the value b_2 = 7 at time t_2 = 2. We wish to fit these values using a line constrained to go through the origin, i.e., with an equation of the form b = Dt. Solve the system using least squares and describe the best fit line.

Answer: In general we would fit a line of the form C + Dt = b; however since we want the fitted line to go through the origin we have C = 0 and are fitting a line of the form 0 + Dt = b. Given the values of b_1 and b_2 at t_1 = 1 and t_2 = 2 respectively we have the following system of two equations:

\begin{array}{rcrcr} 0&+&D&=&1 \\ 0&+&2D&=&7 \end{array}

This system corresponds to the general matrix equation

A \begin{bmatrix} C \\ D \end{bmatrix} = b

which in this case becomes

\begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ D \end{bmatrix} = \begin{bmatrix} 1 \\ 7 \end{bmatrix}

This system has no solution. However we can find a least squares solution in general using the matrix equation

A^TA \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = A^Tb

In this case this corresponds to the equation

\begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 1 \\ 7 \end{bmatrix}

or

\begin{bmatrix} 0&0 \\ 0&5 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0 \\ 15 \end{bmatrix}

We then have 5 \bar{D} = 15 or \bar{D} = 3.

So the least squares solution is the line b = 3t. This line passes through the origin (i.e., at t = 0 it has a value of zero). At time t_1 = 1 it has the value 3, which is greater than the value b_1 = 1. At time t_2 = 2 it has the value 6, which is less than the value b_2 = 7.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.3.1

Exercise 3.3.1. a) Given the system of equations consisting of 3x = 10 and 4x = 5 find the least squares solution \bar{x} and describe the error E^2 being minimized by that solution. Confirm that the error vector \left(10-3\bar{x}, 5-4\bar{x}\right) is orthogonal to the column \left(3, 4\right).

Answer: This is a system Ax = b where A = \begin{bmatrix} 3 \\ 4 \end{bmatrix} and b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}. We can then compute the least squares solution as \bar{x} = \left(A^TA\right)^{-1}A^Tb.

We have

A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25

so that \left(A^TA\right)^{-1} = \frac{1}{25}. We then have

\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}

= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2

So the least squares solution is \bar{x} = 2.

The error vector is then

b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}

and the corresponding error

E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)

= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25

The inner product of the error vector b-A\bar{x} = \left(4, -3\right) and the column \left(3, 4\right) of A is 4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0. Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.16

Exercise 3.2.16. a) Given the projection matrix P projecting vectors onto the line through a and two vectors x and y, show that the inner products of x with Py and y with Px are equal.

b) In general would the angles between x and a and y and a be equal to each other? If a = \left(1, 1, -1\right), x = \left(2, 0, 1\right), and y = \left(2, 1, 2\right) what are the cosines of the two angles?

c) Show that the inner product of Px and Py is the same as the inner products of x with Py and y with Px and explain why this is. What is the angle between the vectors Px and Py?

Answer: a) We have P = aa^T/a^Ta so that

Px = \left(aa^T/a^Ta\right)x = \frac{1}{a^Ta}\left(aa^Tx\right) = \frac{a^Tx}{a^Ta}a

and

Py = \left(aa^T/a^Ta\right)y = \frac{1}{a^Ta}\left(aa^Ty\right) = \frac{a^Ty}{a^Ta}a

The inner product of x with Py is then

x^T\left(Py\right) = x^T\left(a^Ty/a^Ta\right)a

= \left(a^Ty/a^Ta\right)x^Ta = \left(a^Tyx^Ta\right)/a^Ta

= \left(a^Tya^Tx\right)/a^Ta = \left(a^Txa^Ty\right)/a^Ta

and the inner product of y with Px is then

y^T\left(Px\right) = y^T\left(a^Tx/a^Ta\right)a

= \left(a^Tx/a^Ta\right)y^Ta = \left(a^Txy^Ta\right)/a^Ta

= \left(a^Txa^Ty\right)/a^Ta

So the inner products of x with Py and y with Px are equal.

b) Since x and y are arbitrary vectors, in general they would not make the same angle with respect to a. For example, consider the case when a = \left(1, 1, -1\right), x = \left(2, 0, 1\right), and y = \left(2, 1, 2\right).

If \theta_1 is the angle between x and a then \cos \theta_1 = a^Tx/\left(\|a\|\|x\|\right). Similarly, if \theta_2 is the angle between y and a then \cos \theta_2 = a^Ty/\left(\|a\|\|y\|\right)

In this case we have

a^Tx = 1 \cdot 2 + 1 \cdot 1 - 1 \cdot 2 = 2 + 1 - 2 = 1

a^Ty = 1 \cdot 2 + 1 \cdot 0 - 1 \cdot 1 = 2 + 0 - 1 = 1

\|a\| = \sqrt{1^2 + 1^2 + \left(-1\right)^2} = \sqrt{1 + 1 + 1} = \sqrt{3}

\|x\| = \sqrt{2^2 + 0^2 + 1^2} = \sqrt{4 + 0 + 1} = \sqrt{5}

\|y\| = \sqrt{2^2 + 1^2 + 2^2} = \sqrt{4 + 1 + 4} = \sqrt{9} = 3

So \cos \theta_1 = \frac{1}{\sqrt{3}\sqrt{5}} = \frac{1}{\sqrt{15}} and \cos \theta_2 = \frac{1}{\sqrt{3} \cdot 3} = \frac{1}{3\sqrt{3}}. The two cosines are different and thus \theta_1 and \theta_2 are not equal to each other.

c) As noted above we have Px = \frac{a^Tx}{a^Ta}a and Py = \frac{a^Ty}{a^Ta}a. Their inner product is then

\left(Px\right)^T\left(Py\right) = \left(\frac{a^Tx}{a^Ta}a\right)^T \left(\frac{a^Ty}{a^Ta}a\right)

\frac{a^Tx}{a^Ta} \frac{a^Ty}{a^Ta} \left(a^Ta\right) = \left(a^Txa^Ty\right)/a^Ta

But this is the same as the inner products x^TPy and y^TPx of x with Py and y with Px respectively.

This can be understood geometrically as follows: The vector x can be thought of as consisting of two components p_x and q_x where p_x is the projection of x onto a and q_x is the projection of x onto the vector b that is orthogonal to a. The inner product between x and Py is then

\left(Py\right)^Tx = \left(Py\right)^T\left(p_x+q_x\right)

= \left(Py\right)^Tp_x + \left(Py\right)^Tq_x

But p_x is simply Px. Also, since Px is a projection on a and q_x is a projection onto b their inner product is zero since a is orthogonal to b. We therefore have \left(Py\right)^Tx = \left(Py\right)^T\left(Px\right).

Similarly the vector y can be thought of as consisting of two components p_y = Py parallel to a and q_y orthogonal to a, with the inner product between y and Px then being

\left(Px\right)^Ty = \left(Px\right)^T\left(p_y+q_y\right)

= \left(Px\right)^Tp_y + \left(Px\right)^Tq_y

= \left(Px\right)^TPy = \left(Py\right)^TPx

We thus have \left(Py\right)^Tx = \left(Py\right)^TPx = \left(Px\right)^Ty.

Since Px and Py are both projections onto a the angle between them is zero.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

 Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged , | Leave a comment