My Math Homework

Linear Algebra and Its Applications, Exercise 3.3.2

Posted on June 22, 2014 by hecker

Exercise 3.3.2. We have the value $b_1 = 1$ at time $t_1 = 1$ and the value $b_2 = 7$ at time $t_2 = 2$ . We wish to fit these values using a line constrained to go through the origin, i.e., with an equation of the form $b = Dt$ . Solve the system using least squares and describe the best fit line.

Answer: In general we would fit a line of the form $C + Dt = b$ ; however since we want the fitted line to go through the origin we have $C = 0$ and are fitting a line of the form $0 + Dt = b$ . Given the values of $b_1$ and $b_2$ at $t_1 = 1$ and $t_2 = 2$ respectively we have the following system of two equations:

$\begin{array}{rcrcr} 0&+&D&=&1 \\ 0&+&2D&=&7 \end{array}$

This system corresponds to the general matrix equation

$A \begin{bmatrix} C \\ D \end{bmatrix} = b$

which in this case becomes

$\begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ D \end{bmatrix} = \begin{bmatrix} 1 \\ 7 \end{bmatrix}$

This system has no solution. However we can find a least squares solution in general using the matrix equation

$A^TA \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = A^Tb$

In this case this corresponds to the equation

$\begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 0&1 \\ 0&2 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0&0 \\ 1&2 \end{bmatrix} \begin{bmatrix} 1 \\ 7 \end{bmatrix}$

$\begin{bmatrix} 0&0 \\ 0&5 \end{bmatrix} \begin{bmatrix} 0 \\ \bar{D} \end{bmatrix} = \begin{bmatrix} 0 \\ 15 \end{bmatrix}$

We then have $5 \bar{D} = 15$ or $\bar{D} = 3$ .

So the least squares solution is the line $b = 3t$ . This line passes through the origin (i.e., at $t = 0$ it has a value of zero). At time $t_1 = 1$ it has the value 3, which is greater than the value $b_1 = 1$ . At time $t_2 = 2$ it has the value 6, which is less than the value $b_2 = 7$ .

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged least squares, line through origin | Leave a comment

Linear Algebra and Its Applications, Exercise 3.3.1

Posted on June 16, 2014 by hecker

Exercise 3.3.1. a) Given the system of equations consisting of $3x = 10$ and $4x = 5$ find the least squares solution $\bar{x}$ and describe the error $E^2$ being minimized by that solution. Confirm that the error vector $\left(10-3\bar{x}, 5-4\bar{x}\right)$ is orthogonal to the column $\left(3, 4\right)$ .

Answer: This is a system $Ax = b$ where $A = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $b = \begin{bmatrix} 10 \\ 5 \end{bmatrix}$ . We can then compute the least squares solution as $\bar{x} = \left(A^TA\right)^{-1}A^Tb$ .

We have

$A^TA = \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 3 \\ 4 \end{bmatrix} = 3 \cdot 3 + 4 \cdot 4 = 25$

so that $\left(A^TA\right)^{-1} = \frac{1}{25}$ . We then have

$\bar{x} = \left(A^TA\right)^{-1}A^Tb = \frac{1}{25} \begin{bmatrix} 3&4 \end{bmatrix} \begin{bmatrix} 10 \\ 5 \end{bmatrix}$

$= \frac{1}{25} \left(3 \cdot 10 + 4 \cdot 5\right) = \frac{50}{25} = 2$

So the least squares solution is $\bar{x} = 2$ .

The error vector is then

$b - A\bar{x} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} - 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 4 \\ -3 \end{bmatrix}$

and the corresponding error

$E^2 = \|b-A\bar{x}\|^2 = \left(b-A\bar{x}\right)^T\left(b-A\bar{x}\right)$

$= \begin{bmatrix} 4&-3 \end{bmatrix} \begin{bmatrix} 4 \\ -3 \end{bmatrix} = 16 + 9 = 25$

The inner product of the error vector $b-A\bar{x} = \left(4, -3\right)$ and the column $\left(3, 4\right)$ of $A$ is $4 \cdot 3 - 3 \cdot 4 = 12 - 12 = 0$ . Since the inner product is zero the two vectors are orthogonal.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged least squares, orthogonal vectors | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.16

Posted on June 15, 2014 by hecker

Exercise 3.2.16. a) Given the projection matrix $P$ projecting vectors onto the line through $a$ and two vectors $x$ and $y$ , show that the inner products of $x$ with $Py$ and $y$ with $Px$ are equal.

b) In general would the angles between $x$ and $a$ and $y$ and $a$ be equal to each other? If $a = \left(1, 1, -1\right)$ , $x = \left(2, 0, 1\right)$ , and $y = \left(2, 1, 2\right)$ what are the cosines of the two angles?

c) Show that the inner product of $Px$ and $Py$ is the same as the inner products of $x$ with $Py$ and $y$ with $Px$ and explain why this is. What is the angle between the vectors $Px$ and $Py$ ?

Answer: a) We have $P = aa^T/a^Ta$ so that

$Px = \left(aa^T/a^Ta\right)x = \frac{1}{a^Ta}\left(aa^Tx\right) = \frac{a^Tx}{a^Ta}a$

and

$Py = \left(aa^T/a^Ta\right)y = \frac{1}{a^Ta}\left(aa^Ty\right) = \frac{a^Ty}{a^Ta}a$

The inner product of $x$ with $Py$ is then

$x^T\left(Py\right) = x^T\left(a^Ty/a^Ta\right)a$

$= \left(a^Ty/a^Ta\right)x^Ta = \left(a^Tyx^Ta\right)/a^Ta$

$= \left(a^Tya^Tx\right)/a^Ta = \left(a^Txa^Ty\right)/a^Ta$

and the inner product of $y$ with $Px$ is then

$y^T\left(Px\right) = y^T\left(a^Tx/a^Ta\right)a$

$= \left(a^Tx/a^Ta\right)y^Ta = \left(a^Txy^Ta\right)/a^Ta$

$= \left(a^Txa^Ty\right)/a^Ta$

So the inner products of $x$ with $Py$ and $y$ with $Px$ are equal.

b) Since $x$ and $y$ are arbitrary vectors, in general they would not make the same angle with respect to $a$ . For example, consider the case when $a = \left(1, 1, -1\right)$ , $x = \left(2, 0, 1\right)$ , and $y = \left(2, 1, 2\right)$ .

If $\theta_1$ is the angle between $x$ and $a$ then $\cos \theta_1 = a^Tx/\left(\|a\|\|x\|\right)$ . Similarly, if $\theta_2$ is the angle between $y$ and $a$ then $\cos \theta_2 = a^Ty/\left(\|a\|\|y\|\right)$

In this case we have

$a^Tx = 1 \cdot 2 + 1 \cdot 1 - 1 \cdot 2 = 2 + 1 - 2 = 1$

$a^Ty = 1 \cdot 2 + 1 \cdot 0 - 1 \cdot 1 = 2 + 0 - 1 = 1$

$\|a\| = \sqrt{1^2 + 1^2 + \left(-1\right)^2} = \sqrt{1 + 1 + 1} = \sqrt{3}$

$\|x\| = \sqrt{2^2 + 0^2 + 1^2} = \sqrt{4 + 0 + 1} = \sqrt{5}$

$\|y\| = \sqrt{2^2 + 1^2 + 2^2} = \sqrt{4 + 1 + 4} = \sqrt{9} = 3$

So $\cos \theta_1 = \frac{1}{\sqrt{3}\sqrt{5}} = \frac{1}{\sqrt{15}}$ and $\cos \theta_2 = \frac{1}{\sqrt{3} \cdot 3} = \frac{1}{3\sqrt{3}}$ . The two cosines are different and thus $\theta_1$ and $\theta_2$ are not equal to each other.

c) As noted above we have $Px = \frac{a^Tx}{a^Ta}a$ and $Py = \frac{a^Ty}{a^Ta}a$ . Their inner product is then

$\left(Px\right)^T\left(Py\right) = \left(\frac{a^Tx}{a^Ta}a\right)^T \left(\frac{a^Ty}{a^Ta}a\right)$

$\frac{a^Tx}{a^Ta} \frac{a^Ty}{a^Ta} \left(a^Ta\right) = \left(a^Txa^Ty\right)/a^Ta$

But this is the same as the inner products $x^TPy$ and $y^TPx$ of $x$ with $Py$ and $y$ with $Px$ respectively.

This can be understood geometrically as follows: The vector $x$ can be thought of as consisting of two components $p_x$ and $q_x$ where $p_x$ is the projection of $x$ onto $a$ and $q_x$ is the projection of $x$ onto the vector $b$ that is orthogonal to $a$ . The inner product between $x$ and $Py$ is then

$\left(Py\right)^Tx = \left(Py\right)^T\left(p_x+q_x\right)$

$= \left(Py\right)^Tp_x + \left(Py\right)^Tq_x$

But $p_x$ is simply $Px$ . Also, since $Px$ is a projection on $a$ and $q_x$ is a projection onto $b$ their inner product is zero since $a$ is orthogonal to $b$ . We therefore have $\left(Py\right)^Tx = \left(Py\right)^T\left(Px\right)$ .

Similarly the vector $y$ can be thought of as consisting of two components $p_y = Py$ parallel to $a$ and $q_y$ orthogonal to $a$ , with the inner product between $y$ and $Px$ then being

$\left(Px\right)^Ty = \left(Px\right)^T\left(p_y+q_y\right)$

$= \left(Px\right)^Tp_y + \left(Px\right)^Tq_y$

$= \left(Px\right)^TPy = \left(Py\right)^TPx$

We thus have $\left(Py\right)^Tx = \left(Py\right)^TPx = \left(Px\right)^Ty$ .

Since $Px$ and $Py$ are both projections onto $a$ the angle between them is zero.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged inner product, projection matrix | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.15

Posted on June 9, 2014 by hecker

Exercise 3.2.15. For a matrix $A$ show that if $AA^T = A^TA$ then the length of $Ax$ is equal to the length of $A^Tx$ for all $x$ .

Answer: We have $\|A^Tx\|^2 = \left(A^Tx\right)^T\left(A^Tx\right)$ . By the rule for transposes of products we have $\left( A^Tx \right)^T = x^T \left(A^T \right)^T =x^TA$ so that $\|A^Tx\|^2 = x^TAA^Tx$ . But since we assumed that $AA^T = A^TA$ we then have

$\|A^Tx\|^2 = x^TAA^Tx = x^TA^TAx$

Applying the rule for transposes of products once more we have $x^TA^T = \left(Ax\right)^T$ so that

$\|A^Tx\|^2 = \left(Ax\right)^T\left(Ax\right) = \|Ax\|^2$

So the length of $Ax$ is equal to the length of $A^Tx$ .

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged transpose, vector length | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.14

Posted on June 8, 2014 by hecker

Exercise 3.2.14. In $\mathbb{R}^3$ the planes corresponding to the equations $x+y+t =0$ and $x-t=0$ intersect in a line. What is the projection matrix $P$ that projects points in $\mathbb{R}^3$ onto that line?

Answer: To find the line of intersection we solve the system of equations $Av = 0$ where

$A = \begin{bmatrix} 1&1&1 \\ 1&0&-1 \end{bmatrix}$

We use Gaussian elimination, subtracting 1 times row 1 from row 2 to produce the echelon matrix

$U = \begin{bmatrix} 1&1&1 \\ 0&-1&-2 \end{bmatrix}$

Since $U$ has pivots in columns 1 and 2 we have $x$ and $y$ as basic variables and $t$ as a free variable. Setting $t=1$ from the second row of $U$ we have $-y-2t = -y-2 =0$ or $y = -2$ . From the first row of $U$ we then have $x+y+t = x -2 +1 = 0$ or $x = 1$ . So $\left(1, -2, 1 \right)$ is a solution to the system, as is any other vector on the line through the origin and $\left(1, -2, 1 \right)$ .

The projection matrix projecting points in $\mathbb{R}^3$ onto the line through $v = \left(1, -2, 1 \right)$ is then

$P = vv^T/v^Tv = \frac{1}{1 + 4 +1} \begin{bmatrix} 1 \\ -2 \\ 1 \end{bmatrix} \begin{bmatrix} 1&-2&-1 \end{bmatrix}$

$= \frac{1}{6} \begin{bmatrix} 1&-2&1 \\ -2&4&-2 \\ 1&-2&-1 \end{bmatrix} = \begin{bmatrix} \frac{1}{6}&-\frac{1}{3}&\frac{1}{6} \\ -\frac{1}{3}&\frac{2}{3}&-\frac{1}{3} \\ \frac{1}{6}&-\frac{1}{3}&\frac{1}{6} \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged projection matrix | 1 Comment

Linear Algebra and Its Applications, Exercise 3.2.13

Posted on June 1, 2014 by hecker

Exercise 3.2.13. For a projection matrix $P = aa^T/a^Ta$ show that the sum of the diagonal entries of $P$ (the “trace” of $P$ ) always equals one.

Answer: For $a = \left( a_1, a_2, \dots, a_n \right)$ the diagonal entries of $P = aa^T/a^Ta$ are $a_1^2 / (a^Ta)$ , $a_2^2 / (a^Ta)$ , through $a_n^2 / (a^Ta)$ so that the trace of $P$ is

$\left( a_1^2 + a_2^2 + \cdots + a_n^2 \right) / (a^Ta) = a^Ta / (a^Ta) = 1$

UPDATE: Corrected the formulas for the diagonal entries of $P$ .

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged projection matrix, trace | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.12

Posted on May 31, 2014 by hecker

Exercise 3.2.12. Find a projection matrix $P$ that projects every vector in $\mathbb{R}^2$ onto the line described by the equation $x+2y=0$ .

Answer: One solution to the equation $x+2y=0$ is $a = \left( -2, 1 \right)$ . The projection matrix that projects vectors onto the line through $a$ is

$P = aa^T/a^Ta = \frac{1}{-2 \cdot -2 + 1 \cdot 1} \begin{bmatrix} -2 \\ 1 \end{bmatrix}\begin{bmatrix} -2&1 \end{bmatrix}$

$= \frac{1}{5} \begin{bmatrix} 4&-2 \\ -2&1 \end{bmatrix} = \begin{bmatrix} \frac{4}{5}&-\frac{2}{5} \\ -\frac{2}{5}&\frac{1}{5} \end{bmatrix}$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged projection matrix | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.11

Posted on May 25, 2014 by hecker

Exercise 3.2.11. a) Given the line through the origin and $a = \left( 1, 3 \right)$ find the matrix $P_1$ that projects onto this line, as well as the matrix $P_2$ that projects onto the line perpendicular to the original line.

b) What is $P_1 + P_2$ ? What is $P_1P_2$ ? Explain your answers.

Answer: a) We have

$P_1 = aa^T/a^Ta = \frac{1}{1^2 + 3^2} \begin{bmatrix} 1 \\ 3 \end{bmatrix}\begin{bmatrix} 1&3 \end{bmatrix} = \frac{1}{10} \begin{bmatrix} 1&3 \\ 3&9 \end{bmatrix}$

We next find the perpendicular line. If this line passes through the origin and $b = \left(b_1, b_2 \right)$ then we will have $b$ orthogonal to $a$ so that $a^Tb = a_1b_1 + a_2b_2 = b_1 + 3b_2 = 0$ . One vector satisfying this condition is $b = \left( -3, 1 \right)$ . We then have

$P_2 = bb^T/b^Tb = \frac{1}{(-3)^2 + 1^2} \begin{bmatrix} -3 \\ 1 \end{bmatrix}\begin{bmatrix} -3&1 \end{bmatrix} = \frac{1}{10} \begin{bmatrix} 9&-3 \\ -3&1 \end{bmatrix}$

b) Taking the product of the two projection matrices first, we have

$P_1P_2 = \frac{1}{10} \begin{bmatrix} 1&3 \\ 3&9 \end{bmatrix} \frac{1}{10} \begin{bmatrix} 9&-3 \\ -3&1 \end{bmatrix}$

$= \frac{1}{100} \begin{bmatrix} 9-9&-3+3 \\ 27-27&-9+9 \end{bmatrix} = \frac{1}{100} \begin{bmatrix} 0&0 \\ 0&0 \end{bmatrix} = 0$

This true because multiplying a vector by $P_2$ projects it onto the line through $b$ , and multiplying by $P_1$ projects the resulting vector onto the line through $a$ . But because $b$ is perpendicular to $a$ in multiplying by $P_1$ all points on the line through $b$ get projected onto the origin. The effect of multiplying by the product matrix $P_1P_2$ is thus the same as multiplying by zero. (Note also that $P_2P_1 = 0$ for the same reason.)

Taking the sum of the two projection matrices, we have

$P_1 + P_2 = \frac{1}{10} \begin{bmatrix} 1&3 \\ 3&9 \end{bmatrix} + \frac{1}{10} \begin{bmatrix} 9&-3 \\ -3&1 \end{bmatrix}$

$= \frac{1}{10} \begin{bmatrix} 1+9&3-3 \\ 3-3&9+1 \end{bmatrix} = \frac{1}{10} \begin{bmatrix} 10&0 \\ 0&10 \end{bmatrix}$

$= \begin{bmatrix} 1&0 \\ 0&1 \end{bmatrix} = I$

In other words, multiplying a vector by the sum of the two projection matrices results in the same vector:

$(P_1+P_2)v = P_1v + P_2v = v$

In effect we have transformed the vector $v$ into two perpendicular component vectors: one component $P_1v$ on the line through $a$ and a second component $P_2v$ on the perpendicular line through $b$ . Adding the components back together reconstitutes the original vector.

UPDATE: Corrected the calculations above to include the factors $1/a^Ta$ and $1/b^Tb$ when calculating the projection matrices. Thanks go to Trystyn for catching my original error.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged orthogonal vectors, projection matrix | 2 Comments

Linear Algebra and Its Applications, Exercise 3.2.10

Posted on May 24, 2014 by hecker

Exercise 3.2.10. Discuss whether the projection matrix $P$ from exercise 3.2.9 is invertible or not. Explain your answer.

Answer:The projection matrix

$P = aa^T/a^Ta = \frac{1}{a_1a_1 + a_2a_2 + \dots + a_na_n} \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}\begin{bmatrix} a_1&a_2&\dots&a_n \end{bmatrix}$

$= \frac{1}{a_1a_1 + a_2a_2 + \dots + a_na_n} \begin{bmatrix} a_1 \cdot a_1&a_1 \cdot a_2&\cdots&a_1 \cdot a_n \\ a_2 \cdot a_1&a_2 \cdot a_2&\cdots&a_2 \cdot a_n \\ \vdots&\vdots&\ddots&\vdots \\ a_n \cdot a_1&a_n \cdot a_2&\cdots&a_n \cdot a_n \end{bmatrix}$

Note that each column $i$ of the resulting matrix is equal to the original vector $a = \left(a_1, a_2, \dots, a_n \right)$ multiplied by a scalar factor equal to $a_i/a^Ta$ . The columns are therefore linearly dependent with each of them expressible as the first column times the factor $a_i/a_1$ , and thus the rank of the projection matrix $P$ is $r = 1$ . Since the rank $r < n$ the matrix $P$ is singular and not invertible.

Also note that this is consistent with the geometry of the situation: Since $P$ projects vectors from a higher-dimensional vector space onto a line, many points in the space will end up being mapped into a single point on a line. There is thus no way to reverse the mapping and map a point on the line onto a single unique vector in the higher-dimensional space.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged invertible matrices, projection matrix | Leave a comment

Linear Algebra and Its Applications, Exercise 3.2.9

Posted on May 18, 2014 by hecker

Exercise 3.2.9. Consider the projection matrix $P = aa^T/a^Ta$ that projects onto a line. Show that $P^2 = P$ .

Answer: Note that $a^Ta$ is a scalar (the inner product of $a$ with itself) and $aa^T$ is a matrix. We have

$P^2 = \left(aa^T/a^Ta\right)^2 = \left(1/a^Ta\right)^2 \left(aa^T\right)^2$

$= \left(1/a^Ta\right)^2 \left(aa^Taa^T\right) = \left(1/a^Ta\right)^2 a\left(a^Ta\right)a^T$

$= \left(1/a^Ta\right)^2 \left(a^Ta\right)aa^T = \left(1/a^Ta\right) aa^T$

$= aa^T/a^Ta = P$

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

Buy me a snack to sponsor more posts like this!

Posted in linear algebra | Tagged angle, cosine, methane, tetrahedon | Leave a comment

My Math Homework

Linear Algebra and Its Applications, Exercise 3.3.2

Linear Algebra and Its Applications, Exercise 3.3.1

Linear Algebra and Its Applications, Exercise 3.2.16

Linear Algebra and Its Applications, Exercise 3.2.15

Linear Algebra and Its Applications, Exercise 3.2.14

Linear Algebra and Its Applications, Exercise 3.2.13

Linear Algebra and Its Applications, Exercise 3.2.12

Linear Algebra and Its Applications, Exercise 3.2.11

Linear Algebra and Its Applications, Exercise 3.2.10

Linear Algebra and Its Applications, Exercise 3.2.9

Archives

Meta