Linear Algebra and Its Applications, Exercise 3.4.26

Exercise 3.4.26. In the Gram-Schmidt orthogonalization process the third component c' is computed as c' = c - (q_1^Tc)q_1 - (q_2^Tc)q_2. Verify that c' is orthogonal to both q_1 and q_2.

Answer: Taking the dot product of q_1 and c' we have

q_1^Tc' = q_1^T \left[ c - (q_1^Tc)q_1 - (q_2^Tc)q_2 \right] = q_1^Tc - q_1^T(q_1^Tc)q_1 - q_1^T(q_2^Tc)q_2

Since q_1^Tc and q_2^Tc are scalars and q_1 and q_2 are orthonormal we then have

q_1^Tc' = q_1^Tc - q_1^T(q_1^Tc)q_1 - q_1^T(q_2^Tc)q_2 = q_1^Tc - (q_1^Tc)q_1^Tq_1 - (q_2^Tc)q_1^Tq_2

= q_1^Tc - (q_1^Tc) \cdot 1 - (q_2^Tc) \cdot 0 = q_1^Tc - q_1^Tc = 0

So c' is orthogonal to q_1.

Taking the dot product of q_2 and c' we have

q_2^Tc' = q_2^T \left[ c - (q_1^Tc)q_1 - (q_2^Tc)q_2 \right] = q_2^Tc - q_2^T(q_1^Tc)q_1 - q_2^T(q_2^Tc)q_2

= q_1^Tc - (q_1^Tc)q_1^Tq_1 - (q_2^Tc)q_1^Tq_2 = q_2^Tc - q_2^Tc = 0

So c' is also orthogonal to q_2.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.25

Exercise 3.4.25. Given y = x^2 over the interval -1 \le x \le 1 what is the closest line C + Dx to the parabola formed by y?

Answer: This amounts to finding a least-squares solution to the equation \begin{bmatrix} 1&x \end{bmatrix} \begin{bmatrix} C \\ D \end{bmatrix} = y, where the entries 1, x, and y = x^2 are understood as functions of x over the interval -1 to 1 (as opposed to being scalar values).

Interpreting the traditional least squares equation A^TAx = A^Tb in this context, here the matrix A = \begin{bmatrix} 1&x \end{bmatrix} and we have

A^TA = \begin{bmatrix} 1 \\ x \end{bmatrix} \begin{bmatrix} 1&x \end{bmatrix} = \begin{bmatrix} (1, 1)&(1, x) \\ (x, 1)&(x, x) \end{bmatrix}

where the entries of A^TA are the dot products of the functions, i.e., the integrals of their products over the interval -1 to 1.

We then have

(1, 1) = \int_{-1}^1 1 \cdot 1 \;\mathrm{d}x = 2

(1, x) = (x, 1) = \int_{-1}^1 1 \cdot x \;\mathrm{d}x = \left( \frac{1}{2}x^2 \right) \;\big|_{-1}^1 = \frac{1}{2} \cdot 1^2 - \frac{1}{2} \cdot (-1)^2 = \frac{1}{2} - \frac{1}{2} = 0

(x, x) = \int_{-1}^1 x^2 \;\mathrm{d}x = \left( \frac{1}{3}x^3 \right) \;\big|_{-1}^1 = \frac{1}{3} \cdot 1^3 - \frac{1}{3} \cdot (-1)^3 = \frac{1}{3} + \frac{1}{3} = \frac{2}{3}

so that

A^TA = \begin{bmatrix} (1, 1)&(1, x) \\ (x, 1)&(x, x) \end{bmatrix} = \begin{bmatrix} 2&0 \\ 0&\frac{2}{3} \end{bmatrix}

Continuing the interpretation of the least squares equation A^TAx = A^Tb in this context, the role of b is played by the function y = x^2, and we have

A^Ty = \begin{bmatrix} 1 \\ x \end{bmatrix} x^2 = \begin{bmatrix} (1,x^2) \\ (x, x^2) \end{bmatrix}

where again the entries are dot products of the functions. From above we have

(1, x^2) = \int_{-1}^1 1 \cdot x^2 \;\mathrm{d}x = \frac{2}{3}

and from previous exercises we have

(x, x^2) = \int_{-1}^1 x \cdot x^2 \;\mathrm{d}x = \int_{-1}^1 x^3 \;\mathrm{d}x = 0

so that

A^Ty = \begin{bmatrix} \frac{2}{3} \\ 0 \end{bmatrix}

To get the least squares solution \bar{C} + \bar{D}x we then have

\begin{bmatrix} 2&0 \\ 0&\frac{2}{3} \end{bmatrix} \begin{bmatrix} \bar{C} \\ \bar{D} \end{bmatrix} = \begin{bmatrix} \frac{2}{3} \\ 0 \end{bmatrix}

From the second equation we have \bar{D} = 0. From the first equation we have 2\bar{C} = \frac{2}{3} or C = \frac{1}{3}.

The line of best fit to the parabola y = x^2 over the interval -1 \le x \le 1 is therefore the horizontal line with y-intercept of \frac{1}{3}.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.24

Exercise 3.4.24. As discussed on page 178, the first three Legendre polynomials are 1, x, and x^2 - \frac{1}{3}. Find the next Legendre polynomial; it will be a cubic polynomial defined for -1 \le x \le 1 and will be orthogonal to the first three Legendre polynomials.

Answer: The process of finding the fourth Legendre poloynomial is essentially an application of Gram-Schmidt orthogonalization. The first three polynomials are

v_1 = 1 \qquad v_2 = x \qquad v_3 = x^2 - \frac{1}{3}

We can find the fourth Legendre polynomial by starting with x^3 and subtracting off the projections of x_3 on the first three polynomials:

v_4 = x^3 - \frac{(v_1, x^3)}{(v_1, v_1)}v_1 - \frac{(v_2, x^3)}{(v_2, v_2)}v_2 - \frac{(v_3, x^3)}{(v_3, v_3)}v_3

= \frac{(1, x^3)}{(1, 1)}\cdot 1 - \frac{(x, x^3)}{(x, x)}x - \frac{(x^2-\frac{1}{3}, x^3)}{(x^2-\frac{1}{3}, x^2-\frac{1}{3})}(x^2-\frac{1}{3})

For the first term we have

(1, x^3) = \int_{-1}^1 1 \cdot x^3 \;\mathrm{d}x = \int_{-1}^1 x^3 \;\mathrm{d}x = 0

so that the first term \frac{(v_1, x^3)}{(v_1, v_1)}v_1 does not appear in the expression for v_4.

The third term \frac{(v_3, x^3)}{(v_3, v_3)}v_3 drops out for the same reason: its numerator is

(x^2-\frac{1}{3}, x^3) = \int_{-1}^1 (x^2 - \frac{1}{3}) x^3 \;\mathrm{d}x

= \int_{-1}^1 x^5 \;\mathrm{d}x - \frac{1}{3} \int_{-1}^1 x^3 \;\mathrm{d}x = 0 - \frac{1}{3} \cdot 0 = 0

That leaves the second term \frac{(v_2, x^3)}{(v_2, v_2)}v_2 with numerator of

(x, x^3) = \int_{-1}^1 x \cdot x^3 \;\mathrm{d}x = \int_{-1}^1 x^4 \;\mathrm{d}x

= \left( \frac{1}{5} x^5 \right) \;\big|_{-1}^1 = \frac{1}{5} \cdot 1^5 - \frac{1}{5} \cdot (-1)^5 = \frac{1}{5} - (-\frac{1}{5}) = \frac{2}{5}

and denominator

(x, x) = \int_{-1}^1 x^2 \;\mathrm{d}x = \left( \frac{1}{3}x^3 \right) \;\big|_{-1}^1 = \frac{1}{3} \cdot 1^3 - \frac{1}{3} \cdot (-1)^3 = \frac{1}{3} + \frac{1}{3} = \frac{2}{3}

We then have

v_4 = x^3 - \left[ \frac{2}{5}/\frac{2}{3} \right] x = x^3 - \frac{3}{5}x

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.23

Exercise 3.4.23. Given the step function y with y(x) = 1 for 0 \le x \le \pi and y(x) = 0 for \pi < x < 2\pi, find the following Fourier coefficients:

a_0 = \frac{(y, 1)}{(1, 1)} \qquad a_1 = \frac{(y, \cos x)}{(\cos x, \cos x)} \qquad b_1 = \frac{(y, \sin x)}{(\sin x, \sin x)}

Answer: For a_0 the numerator is

(y, 1) = \int_0^{2\pi} y(x) \cdot 1 \;\mathrm{d}x = \int_0^{\pi} 1 \;\mathrm{d}x + \int_{\pi}^{2\pi} 0 \;\mathrm{d}x = \pi

and the denominator is

(1, 1) = \int_0^{2\pi} 1^2 \;\mathrm{d}x = 2\pi

so that a_0 = \frac{\pi}{2\pi} = \frac{1}{2}.

For a_1 the numerator is

(y, \cos x) = \int_0^{2\pi} y(x) \cos x \;\mathrm{d}x = \int_0^{\pi} 1 \cdot \cos x \;\mathrm{d}x + \int_{\pi}^{2\pi} 0 \cdot \cos x \;\mathrm{d}x

= \int_0^{\pi} \cos x = \sin x \;\big|_0^{\pi} = 0 - 0 = 0

so that a_1 = 0.

For b_1 the numerator is

(y, \sin x) = \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x = \int_0^{\pi} 1 \cdot \sin x \;\mathrm{d}x + \int_{\pi}^{2\pi} 0 \cdot \sin x \;\mathrm{d}x

= \int_0^{\pi} \sin x = (-\cos x) \;\big|_0^{\pi} = -(-1) - (-1) = 1 + 1 = 2

and the denominator is

(\sin x, \sin x) = \int_0^{2\pi} \sin^2 x \;\mathrm{d}x = \left[ \frac{1}{2}x - \frac{1}{4} \sin 2x \right] \;\big|_0^{2\pi}

= \left[ \frac{1}{2}\cdot(2\pi) - \frac{1}{4} \sin 2\pi \right] - \left[ \frac{1}{2} \cdot 0 - \frac{1}{4} \sin 2 \cdot 0 \right] = \pi - \frac{1}{4} \cdot 0 - 0 + \frac{1}{4} \cdot 0 = \pi

so that b_1 = \frac{2}{\pi}.

So we have a_0 = \frac{1}{2}, a_1 = 0, and b_1 = \frac{2}{\pi}.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.22

Exercise 3.4.22. Given an arbitrary function y find the coefficient b_1 that minimizes the quantity

\|b_1\sin x - y\|^2 = \int_0^{2\pi} (b_1\sin x - y(x))^2 \;\mathrm{d}x

(Use the method of setting the derivative to zero.) How does this value of b_1 compare with the Fourier coefficient b_1? What is b_1 if y(x) = \cos x?

Answer: We are looking for a value of b_1 that minimizes the expression on the right, so we need to differentiate with respect to b_1. Expanding the right-hand side of the equation above, we have

\int_0^{2\pi} (b_1\sin x - y(x))^2 \;\mathrm{d}x = \int_0^{2\pi} [b_1^2\sin^2 x - 2b_1y(x)\sin x +  y(x)^2] \;\mathrm{d}x

= \int_0^{2\pi} b_1^2\sin^2 x \;\mathrm{d}x - 2 \int_0^{2\pi} b_1y(x)\sin x \;\mathrm{d}x + \int_0^{2\pi} y(x)^2 \;\mathrm{d}x

Since b_1 is not dependent on x we can pull it out of the integral, so that

\int_0^{2\pi} (b_1\sin x - y(x))^2 \;\mathrm{d}x = b_1^2 \int_0^{2\pi} \sin^2 x \;\mathrm{d}x - 2b_1 \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x + \int_0^{2\pi} y(x)^2 \;\mathrm{d}x

Differentiating with respect to b_1 we have

\frac{\mathrm{d}}{\mathrm{d}b_1} \int_0^{2\pi} (b_1\sin x - y(x))^2 \;\mathrm{d}x

\frac{\mathrm{d}}{\mathrm{d}b_1} \left[ b_1^2 \int_0^{2\pi} \sin^2 x \;\mathrm{d}x - 2b_1 \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x + \int_0^{2\pi} y(x)^2 \;\mathrm{d}x \right]

= 2b_1 \int_0^{2\pi} \sin^2 x \;\mathrm{d}x - 2 \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x

Equating the derivative to zero gives us

2b_1 \int_0^{2\pi} \sin^2 x \;\mathrm{d}x = 2 \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x

or

b_1 = \left( \int_0^{2\pi} y(x) \sin x \;\mathrm{d}x \right) / \left( \int_0^{2\pi} \sin^2 x \;\mathrm{d}x \right)

Note that this is identical to the expression for the Fourier coefficient b_1 on page 178; the numerator is the dot product of y(x) with \sin x and the denominator is the dot product of \sin x with itself.

If y(x) = \cos x then the numerator of b_1 becomes

\int_0^{2\pi} \cos x \sin x \;\mathrm{d}x = 0

since \cos x and \sin x are orthogonal, and we therefore have b_1 = 0.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in Uncategorized | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.21

Exercise 3.4.21. Given the function f(x) = \sin 2x on the interval -\pi \le x \le \pi, what is the closest function a \cos x + b \sin x to f? What is the closest line c + dx to f?

Answer: To find the closest function a \cos x + b \sin x to the function f(x) = \sin 2x we first project f onto the function \cos x on the given interval to obtain a, and then project f onto \sin x to obtain b.

We project f onto \cos x by taking the dot product of f with \cos x and then normalizing by dividing by the dot product of \cos x with itself:

a = (f, \cos x)/(\cos x, \cos x)

The numerator is

(f, \cos x) = \int_{-\pi}^{\pi} f(x) \cos x \;\mathrm{d}x = \int_{-\pi}^{\pi} \sin 2x \cos x \;\mathrm{d}x

= 2 \int_{-\pi}^{\pi} \sin x \cos^2 x \;\mathrm{d}x

where we used the trigonometric identity \sin 2\theta = \sin \theta \cos \theta.

To integrate we substitute the variable u = \cos x so that \mathrm{d}u = -\sin x \;\mathrm{d}x. We then have

\int \sin x \cos^2 x \;\mathrm{d}x = -\int \cos^2 x (-\sin x) \;\mathrm{d}x

-\int u^2 \;\mathrm{d}u = -\frac{1}{3}u^3 = -\frac{1}{3} \cos^3 x

We then have

(f, \cos x) = 2 \int_{-\pi}^{\pi} \sin x \cos^2 x \;\mathrm{d}x = 2 (-\frac{1}{3} \cos^3 x) \;\big|_{-\pi}^{\pi}

= -\frac{2}{3} \cos^3 \pi - [-\frac{2}{3} \cos^3 (-\pi)] = -\frac{2}{3} (-1)^3 - [-\frac{2}{3} (-1)^3]

= -\frac{2}{3} \cdot (-1) - [-\frac{2}{3} \cdot (-1)] = \frac{2}{3} - \frac{2}{3} = 0

Since the numerator in the expression for a is zero, we have a = 0.

(Note that we do not need to calculate the denominator in the expression for a. We know it must be positive, and thus the quotient is defined. See below for a sketch of a proof of this.)

We next project f onto \sin x by taking the dot product of f with \sin x and then normalizing by dividing by the dot product of \sin x with itself:

a = (f, \sin x)/(\sin x, \sin x)

The numerator is

(f, \sin x) = \int_{-\pi}^{\pi} f(x) \sin x \;\mathrm{d}x = \int_{-\pi}^{\pi} \sin 2x \sin x \;\mathrm{d}x

= 2 \int_{-\pi}^{\pi} \sin^2 x \cos x \;\mathrm{d}x

where we used the trigonometric identity \sin 2\theta = \sin \theta \cos \theta.

To integrate we substitute the variable u = \sin x so that \mathrm{d}u = \cos x \;\mathrm{d}x. We then have

\int \sin^2 x \cos x \;\mathrm{d}x = \int u^2 \;\mathrm{d}u = \frac{1}{3}u^3 = \frac{1}{3} \sin^3 x

We then have

(f, \sin x) = 2 \int_{-\pi}^{\pi} \sin^2 x \cos x \;\mathrm{d}x = 2 (\frac{1}{3} \sin^3 x) \;\big|_{-\pi}^{\pi}

= \frac{2}{3} \sin^3 \pi - \frac{2}{3} \sin^3 (-\pi) = \frac{2}{3} (0)^3 - \frac{2}{3} (0)^3

= 0 - 0 = 0

Since the numerator in the expression for b is zero, we have b = 0. (Again, we are guaranteed that the denominator is positive and the quotient defined.)

So the closest function a \cos x + b \sin x to f(x) = \sin 2x is 0 \cdot \cos x + 0 \cdot \sin x = 0.

To find the closest function c + dx to the function f(x) = \sin 2x we first project f onto the constant function with the value 1 on the given interval to obtain c, and then project f onto the function x to obtain d.

We project f onto the constant function with value 1 by taking the dot product of f with 1 and then normalizing by dividing by the dot product of 1 with itself:

c = (f, 1)/(1, 1)

The numerator is

(f, 1) = \int_{-\pi}^{\pi} f(x) \cdot 1 \;\mathrm{d}x = \int_{-\pi}^{\pi} \sin 2x \;\mathrm{d}x

To integrate we substitute the variable u = 2x so that \mathrm{d}u = 2 \;\mathrm{d}x. We then have

\int \sin 2x \;\mathrm{d}x = \int \frac{1}{2} \sin 2x \cdot 2 \;\mathrm{d}x

= \frac{1}{2} \int \sin u \;\mathrm{d}u = \frac{1}{2}(-\cos u) = -\frac{1}{2} \cos 2x

We then have

(f, 1) = \int_{-\pi}^{\pi} \sin 2x \;\mathrm{d}x = -\frac{1}{2} \cos 2x \;\big|_{-\pi}^{\pi}

= -\frac{1}{2} \cos 2\pi - (-\frac{1}{2} \cos (-2\pi) = -\frac{1}{2} (1)^3 - (-\frac{1}{2} (1)^3

= -\frac{1}{2} + \frac{1}{2}= 0

Since the numerator in the expression for c = (f, 1)/(1, 1) is zero we have c = 0. (Recall that the denominator is guaranteed to be positive.)

We project f onto the function x by taking the dot product of f with x and then normalizing by dividing by the dot product of x with itself:

d = (f, x)/(x, x)

The numerator is

(f, x) = \int_{-\pi}^{\pi} f(x) \cdot x \;\mathrm{d}x = \int_{-\pi}^{\pi} x \sin 2x \;\mathrm{d}x

To integrate this we use integration by parts, taking advantage of the formula \int u \;\mathrm{d}v = uv - \int v \;\mathrm{d}u. (The following is adapted from a post on socratic.org.) We let \mathrm{d}v = \sin 2x \;\mathrm{d}x and u = x. Then \mathrm{d}u is simply \mathrm{d}x, and v = -\frac{1}{2} \cos 2x (the integrand of \sin 2x, as discussed above).

We then have

\int x \sin x \;\mathrm{d}x = \int u \;\mathrm{d}v = uv - \int v \;\mathrm{d}u

= x (-\frac{1}{2} \cos 2x) - \int (-\frac{1}{2} \cos 2x) \;\mathrm{d}x

= -\frac{1}{2} x \cos 2x + \frac{1}{2} \int \cos 2x \;\mathrm{d}x

The second integral we can evaluate by substituting w = 2x and \mathrm{d}w = 2 \;\mathrm{d}x so that

\int \cos 2x \;\mathrm{d}x = \frac{1}{2} \int \cos w \;\mathrm{d}w = \frac{1}{2} \sin w = \frac{1}{2} \sin 2x

Substituting for the second integral above we then have

\int x \sin x \;\mathrm{d}x = -\frac{1}{2} x \cos 2x + \frac{1}{2} \int \cos 2x \;\mathrm{d}x = -\frac{1}{2} x \cos 2x + \frac{1}{2} (\frac{1}{2} \sin 2x)

= -\frac{1}{2} x \cos 2x + \frac{1}{4} \sin 2x

We then have

(f, x) = \int_{-\pi}^{\pi} x \sin 2x \;\mathrm{d}x = -\frac{1}{2} x \cos 2x \;\big|_{-\pi}^{\pi} + \frac{1}{4} \sin 2x \;\big|_{-\pi}^{\pi}

= -\frac{1}{2} \pi \cos 2\pi - (-\frac{1}{2} (-\pi) \cos (-2\pi) + \frac{1}{4} \sin 2\pi - \frac{1}{4} \sin 2(-\pi)

= -\frac{1}{2} \pi \cdot 1 + \frac{1}{2} (-\pi) \cdot 1 + \frac{1}{4} \cdot 0 - \frac{1}{4} \cdot 0 = -\frac{\pi}{2} - \frac{\pi}{2}= -\pi

The denominator in the expression for d is

(x, x) = \int_{-\pi}^{\pi} x^2 \;\mathrm{d}x = \frac{1}{3} x^3 \;\big|_{-\pi}^{\pi}

= \frac{1}{3} \pi^3 - \frac{1}{3} (-\pi)^3 = \frac{2}{3} \pi^3

We then have

d = (f, x)/(x, x) = -\pi / (\frac{2}{3} \pi^3) = -\frac{3}{2\pi^2}

The straight line c + dx closest to the function \sin 2x is thus the line -\frac{3}{2\pi^2} x.

ADDENDUM: Suppose that g is a continuous function defined on the interval [a, b] and g(t) \ne 0 for some a \le t \le b. Then we want to show that the inner product (g, g) > 0.

The basic idea of the proof is as follows: The function g^2 is always nonnegative, and thus its integral over the interval [a, b] is nonnegative as well. If g(t) is nonzero for some a \le t \le b then since g is continuous g will also be nonzero for some interval [c, d] that includes t, with a \le c < d \le b. This implies that the integral of g^2 over that subinterval [c, d] will be positive.

But we also have \int_a^b g(x)^2 \;\mathrm{d}x \ge \int_c^d g(x)^2 \;\mathrm{d}x since g(x)^2 \ge 0 and [c, d] is contained within [a, b]. So if \int_c^d g(x)^2 \;\mathrm{d}x > 0 then we also have \int_a^b g(x)^2 \;\mathrm{d}x > 0 and the inner product (g, g) is positive.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.20

Exercise 3.4.20. Given the vector v = (\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{4}}, \frac{1}{\sqrt{8}}, \ldots) what is the length \|v\|? Given the function f(x) = e^x for 0 \le x \le 1 what is the length of the function over the interval? Given the function g(x) = e^{-x} for 0 \le x \le 1 what is the inner product of f(x) and g(x)?

Answer: We have

\|v\|^2 = v^Tv = (\frac{1}{\sqrt{2}})^2 + (\frac{1}{\sqrt{4}})^2 + (\frac{1}{\sqrt{8}})^2 + \ldots

= \frac{1}{2} + \frac{1}{4} + \frac{1}{8} + \ldots = 1

so that \|v\| = \sqrt{1} = 1.

We have

\|f\|^2 = \int_0^1 f(x)^2 \mathrm{d}x = \int_0^1 (e^x)^2 \mathrm{d}x = \int_0^1 e^{2x} \mathrm{d}x

= \frac{e^{2x}}{2} \big|_0^1 = \frac{e^2}{2} - \frac{e^0}{2} = \frac{e^2 - 1}{2}

so that \|f\| = \sqrt{\frac{e^2 - 1}{2}}.

Finally, the inner product of f(x) and g(x) is

\int_0^1 f(x)g(x) \mathrm{d}x = \int_0^1 e^x e^{-x} \mathrm{d}x = \int_0^1 e^0 \mathrm{d}x = \int_0^1 1 \mathrm{d}x = 1

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.19

Exercise 3.4.19. When doing Gram-Schmidt orthogonalization, an alternative approach to computing c' = c - (q_1^Tc)q_1 - (q_2^Tc)q_2 (equation 7 on page 173) is to instead compute c' in two separate steps:

c'' = c - (q_1^Tc)q_1 \qquad c' = c'' - (q_2^Tc'')q_2

Show that the second method is equivalent to the first.

Answer: We substitute the expression for c'' from the first equation into the second equation:

c' = c'' - (q_2^Tc'')q_2 = [c - (q_1^Tc)q_1] - \left[q_2^T\left[c - (q_1^Tc)q_1\right]\right]q_2

= c - (q_1^Tc)q_1 - \left[q_2^Tc - q_2^T(q_1^Tc)q_1\right]q_2

= c - (q_1^Tc)q_1 - \left[q_2^Tc - (q_1^Tc)q_2^Tq_1\right]q_2

= c - (q_1^Tc)q_1 - \left[q_2^Tc - (q_1^Tc) \cdot 0\right]q_2

= c - (q_1^Tc)q_1 - (q_2^Tc)q_2

The resulting equation c' = c - (q_1^Tc)q_1 - (q_2^Tc)q_2 is the original calculation of c' from equation 7 on page 173. So the second method using c'' is equivalent to the first.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.18

Exercise 3.4.18. If P is the projection matrix onto the column space of the matrix A and A = QR, what is a simple formula for P?

Answer: The projection matrix P onto the column space of A can be calculated as P = A(A^TA)^{-1}A^T.

Since the columns of Q are linear combinations of the columns of A the column space of Q is the same as the column space of A. Thus P is also the projection matrix onto the column space of Q, and we can alternatively calculate P as P = Q(Q^TQ)^{-1}Q^T.

But since Q has orthonormal columns we have Q^TQ = I so that P = QI^{-1}Q^T = QQ^T. Thus P = QQ^T is a simplified formula for calculating the projection matrix onto the column space of A = QR.

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , , | Leave a comment

Linear Algebra and Its Applications, Exercise 3.4.17

Exercise 3.4.17. Given the matrix

A = \begin{bmatrix} 1&1 \\ 2&3 \\ 2&1 \end{bmatrix}

from the previous exercise and the vector b = (1, 1, 1), solve Ax = b by least squares using the factorization A = QR.

Answer: From the previous exercise we have

Q = \begin{bmatrix} \frac{1}{3}&0 \\ \frac{2}{3}&\frac{1}{\sqrt{2}} \\ \frac{2}{3}&-\frac{1}{\sqrt{2}} \end{bmatrix}

R = \begin{bmatrix} 3&3 \\ 0&\sqrt{2} \end{bmatrix}

To find the least squares solution \bar{x} to Ax = b where b = (1, 1, 1), we take advantage of the fact that R\bar{x} = Q^Tb.

On the right side of the equation we have

Q^Tb = \begin{bmatrix} \frac{1}{3}&\frac{2}{3}&\frac{2}{3} \\ 0&\frac{1}{\sqrt{2}}&-\frac{1}{\sqrt{2}} \end{bmatrix} \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} = \begin{bmatrix} \frac{5}{3} \\ 0 \end{bmatrix}

so that the system R\bar{x} = Q^Tb is then

\begin{bmatrix} 3&3 \\ 0&\sqrt{2} \end{bmatrix} \begin{bmatrix} \bar{x_1} \\ \bar{x_2} \end{bmatrix} = \begin{bmatrix} \frac{5}{3} \\ 0 \end{bmatrix}

From the second equation we have \bar{x_2} = 0. Substituting into the first equation we have 3\bar{x_1} + 3\bar{x_2} = 3\bar{x_1} + 3 \cdot 0 = \frac{5}{3} so that 3\bar{x_1} = \frac{5}{3} or \bar{x_1} = \frac{5}{9}.

The least squares solution to Ax = b with b = (1, 1, 1) is therefore \bar{x} = (\frac{5}{9}, 0).

NOTE: This continues a series of posts containing worked out exercises from the (out of print) book Linear Algebra and Its Applications, Third Edition by Gilbert Strang.

If you find these posts useful I encourage you to also check out the more current Linear Algebra and Its Applications, Fourth Edition, Dr Strang’s introductory textbook Introduction to Linear Algebra, Fifth Edition and the accompanying free online course, and Dr Strang’s other books.

Posted in linear algebra | Tagged , | Leave a comment