Exercise 3.3.4. Given

expand the expression , compute its partial derivatives with respect to and , and set them to zero. Compare the resulting equations to to confirm that you obtain the same normal equations in both cases (i.e., using calculus vs. geometry). Then find the projection and explain why .

Answer: We have

so that

Taking the partial derivative of this expression with respect to we have

and setting it to zero we have

where and represent the least squares solution.

Taking the partial derivative of this expression with respect to we have

and setting it to zero we have

where again and represent the least squares solution.

We then have a system of two equations

that is equivalent to the matrix equation

We now compare this to the normal equations . We have

and

so that the normal equation in matrix form is

Note that this equation is the same as the previous equation calculated from taking the derivatives, except with both sides divided by 2.

We can now solve for . We have

so that

We can multiply by to obtain the projection of onto the column space of :

We have because is already in the column space of ; in particular, is equal to the first column of plus 3 times the second column of :

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fourth Edition and the accompanying free online course, and Dr Strang’s other books.