Numerical Analysis/Differentiation/Examples

When deriving a finite difference approximation of the $j$th derivative of a function $f:ℝ\to ℝ$, we wish to find $a_1,a_2,...,a_n\in ℝ$ and $b_1,b_2,...,b_n\in ℝ$ such that

$f^{(j)}(x_0)=h^{-j}\sum _{i=1}^n{a}_{i}f(x_0+b_{i}h)+O(h^k)\text{ as }h\to 0$

or, equivalently,

$h^{-j}\sum _{i=1}^n{a}_{i}f(x_0+b_{i}h)=f^{(j)}(x_0)+O(h^k)\text{ as }h\to 0$

where $O(h^k)$ is the error, the difference between the correct answer and the approximation, expressed using Big-O notation. Because $h$ may be presumed to be small, a larger value for $k$ is better than a smaller value.

A general method for finding the coefficients is to generate the Taylor expansion of $h^{-j}\sum _{i=1}^n{a}_{i}f(x_0+b_{i}h)$ and choose $a_1,a_2,...,a_n$ and $b_1,b_2,...,b_n$ such that $f^{(j)}(x_0)$ and the remainder term are the only non-zero terms. If there are no such coefficients, a smaller value for $k$ must be chosen.

For a function of $m$ variables $g:{ℝ}^m\to ℝ$, the procedure is similar, except $x_0,b_1,b_2,...,b_n$ are replaced by points in ${ℝ}^m$ and the multivariate extension of Taylor's theorem is used.

Single-Variable

In all single-variable examples, $x_0\in ℝ$ and $f:ℝ\to ℝ$ are unknown, and $h\in ℝ$ is small. Additionally, let $f$ be 5 times continuously differentiable on $ℝ$.

First Derivative

Find $a,b,c\in ℝ$ such that $\frac{af(x_0+h)+bf(x_0+ch)}{h}$ best approximates $f^{\prime }(x_0)$.

Solution:

First, we find the Taylor series expansion of $\frac{af(x_0+h)+bf(x_0+ch)}{h}$ with remainder term to be

$\frac{af(x_0+h)+bf(x_0+ch)}{h}=\frac{a+b}{h}f(x_0)+(a+bc)f^{\prime }(x_0)+\frac{h}{2}(a+b{c}^2)f^{″}(x_0)+O(h^2)\text{ as }h\to 0.$

If we can find a solution to the system

$\begin{array}{ccccc}0& =& a+& b& \\ 1& =& a+& bc& \\ 0& =& a+& b{c}^2& \end{array}$

then we can substitute that solution into the Taylor expansion to obtain

$\frac{af(x_0+h)+bf(x_0+ch)}{h}=f^{\prime }(x_0)+O(h^2)\text{ as }h\to 0.$

The system of equations has exactly one solution: $a=\frac{1}{2}$, $b=\frac{1}{-2}$, $c=-1$, so

$\frac{f(x_0+h)-f(x_0-h)}{2h}=f^{\prime }(x_0)+O(h^2)\text{ as }h\to 0.$

Let $f:ℝ\to ℝ$ be 42 times continuously differentiable on $ℝ$. Find the largest $n\in \mathbb{N}$ such that

$\frac{df}{dx}(x_0)=\frac{-f(x_0+2h)+8f(x_0+h,y_0)-8f(x_0-h)+f(x_0-2h)}{12h}+O(h^n)\text{ as }h\to 0$

In other words, find the order of the error of the method.

Solution:

The Taylor expansion of the method is

$\begin{array}{cc}\frac{-f(x_0+2h)+8f(x_0+h,y_0)-8f(x_0-h)+f(x_0-2h)}{12h}& =\frac{-1+8-8+1}{12h}f(x_0)+\frac{-2+8+8-2}{12}{f}^{\prime }(x_0)\\ & +h\frac{(-4+8-8+4)}{24}{f}^{″}(x_0)+h^2\frac{(-8+8+8-8}{72}{f}^{‴}(x_0)\\ & +h^3\frac{(-16+8-8+16}{288}{f}^{(4)}(x_0)+h^4\frac{(-32+8+8-32)}{1440}{f}^{(4)}(x_0)+O(h^5)\text{ as }h\to 0.\end{array}$

Simplifying this algebraically gives

$\frac{-f(x_0+2h)+8f(x_0+h,y_0)-8f(x_0-h)+f(x_0-2h)}{12h}=f^{\prime }(x_0)+\frac{h^4}{-30}{f}^{(5)}(x_0)+O(h^5)\text{ as }h\to 0.$

The multiple of $h^4$ cannot be removed, so $n=4$ and by properties of Big-O notation,

$\frac{-f(x_0+2h)+8f(x_0+h,y_0)-8f(x_0-h)+f(x_0-2h)}{12h}=f^{\prime }(x_0)+O(h^4)\text{ as }h\to 0.$

Second Derivative

Find $a,b,c\in ℝ$ such that $\frac{af(x_0-h)+bf(x_0)+cf(x_0+h)}{h^2}$ best approximates $f^{″}(x_0)$.

Solution:

First, we find the Taylor series expansion of $\frac{af(x_0-h)+bf(x_0)+cf(x_0+h)}{h^2}$, with remainder term to be

$\frac{af(x_0-h)+bf(x_0)+cf(x_0+h)}{h^2}=\frac{a+b+c}{h^2}f(x_0)+\frac{c-a}{h}{f}^{\prime }(x_0)+\frac{a+c}{2}{f}^{″}(x_0)+\frac{h}{6}(c-a)f^{‴}(x_0)+O(h^2)\text{ as }h\to 0.$

If we can find a solution to the system

$\begin{array}{ccccccccc}0& =& & a& +& b& +& c& \\ 0& =& & -a& +& 0& +& c& \\ 2& =& & a& +& 0& +& c& \\ 0& =& & -a& +& 0& +& c& \end{array}$

then we can substitute that solution into the Taylor expansion and obtain

$\frac{af(x_0-h)+bf(x_0)+cf(x_0+h)}{h^2}=f^{″}(x_0)+O(h^2)\text{ as }h\to 0.$

The system of equations has exactly one solution: $a=1$, $b=-2$, $c=1$ so

$\frac{f(x_0-h)-2f(x_0)+f(x_0+h)}{h^2}=f^{″}(x_0)+O(h^2)\text{ as }h\to 0.$

Multivariate

In all two-variable examples, $x_0,y_0\in ℝ$ and $f:{ℝ}^2\to ℝ$ are unknown, and $h\in ℝ$ is small.

Non-Mixed Derivatives

Because of the nature of partial derivatives, some of them may be calculated using single-variable methods. This is done by holding constant all but one variable to form a new function of one variable. For example if $g_y(y)=f(x_0,y)$, then $\frac{df}{dy}(x_0,y_0)=\frac{dg}{dy}(y_0)$.

Find an approximation of $\frac{d}{dy}f(x_0,y_0)$

Solution:

Because we are differentiating with respect to only one variable, we can hold x constant and use the result of one of the single-variable examples:

$\frac{f(x_0,y_0+h)-f(x_0,y_0-h)}{2h}=\frac{df}{dy}(x_0,y_0)+O(h^2)\text{ as }h\to 0$

Mixed Derivatives

Mixed derivatives may require the multivariate extension of Taylor's theorem.

Let $f:{ℝ}^2\to ℝ$ be 42 times continuously differentiable on ${ℝ}^2$ and let $g:{ℝ}^3\to ℝ$ be defined as

$g(x_0,y_0,h)=\frac{f(x_0+h,y_0+h)+f(x_0-h,y_0-h)-f(x_0-h,y_0+h)-f(x_0+h,y_0-h)}{4h^2}.$

Find the largest $n\in \mathbb{N}$ such that

$\frac{d^{2}f}{dxdy}(x_0,y_0)=g(x_0,y_0,h)+O(h^n)\text{ as }h\to 0.$

In other words, find the order of the error of the approximation.

Solution:

The first few terms of the multivariate Taylor expansion of $f$ around $(x_0,y_0)$ are

$\begin{array}{cc}f(x_0+x,y_0+y)& =f(x_0,y_0)\\ & +x\frac{df}{dx}(x_0,y_0)+y\frac{df}{dy}(x_0,y_0)\\ & +\frac{x^2}{2}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+xy\frac{d}{dy}\frac{df}{dx}(x_0,y_0)+\frac{y^2}{2}\frac{d^{2}f}{d{y}^2}(x_0,y_0)\\ & +\frac{x^3}{6}\frac{d^{3}f}{d{x}^3}(x_0,y_0)+\frac{x^{2}y}{2}\frac{d}{dy}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+\frac{x{y}^2}{2}\frac{d^2}{d{y}^2}\frac{df}{dx}(x_0,y_0)+\frac{y^3}{6}\frac{d^{3}f}{d{y}^3}(x_0,y_0)\\ & +\frac{x^4}{24}\frac{d^{4}f}{d{x}^4}(x_0,y_0)+\frac{x^{3}y}{6}\frac{d}{dy}\frac{d^{3}f}{d{x}^3}(x_0,y_0)+\frac{x^2{y}^2}{4}\frac{d^2}{d{y}^2}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+\frac{x{y}^3}{6}\frac{d^3}{d{y}^3}\frac{df}{dx}(x_0,y_0)+\frac{y^4}{24}\frac{d^{4}f}{d{y}^4}(x_0,y_0)\\ & +O(x^5)+O(y^5)\text{ as }x,y\to 0.\end{array}$

We substitute the expansion for $f$ into the approximation $g$ to obtain

$\begin{array}{cc}g(x_0,y_0,h)& =\frac{1+1-1-1}{4h^2}f(x_0,y_0)\\ & +\frac{1-1+1-1}{4h}\frac{df}{dx}(x_0,y_0)+\frac{1-1-1+1}{4h}\frac{df}{dy}(x_0,y_0)\\ & +\frac{1+1-1-1}{8}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+\frac{1+1+1+1}{4}\frac{d}{dy}\frac{df}{dx}(x_0,y_0)+\frac{1+1-1-1}{8}\frac{d^{2}f}{d{y}^2}(x_0,y_0)\\ & +h(\frac{1-1+1-1}{24}\frac{d^{3}f}{d{x}^3}(x_0,y_0)+\frac{1-1-1+1}{8}\frac{d}{dy}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+\frac{1-1+1-1}{8}\frac{d^2}{d{y}^2}\frac{df}{dx}(x_0,y_0)+\frac{1-1-1+1}{24}\frac{d^{3}f}{d{y}^3}(x_0,y_0))\\ & +h^2(\frac{1+1-1-1}{96}\frac{d^{4}f}{d{x}^4}(x_0,y_0)+\frac{1+1+1+1}{24}\frac{d}{dy}\frac{d^{3}f}{d{x}^3}(x_0,y_0)+\frac{1+1-1-1}{16}\frac{d^2}{d{y}^2}\frac{d^{2}f}{d{x}^2}(x_0,y_0)+\frac{1+1+1+1}{24}\frac{d^3}{d{y}^3}\frac{df}{dx}(x_0,y_0)+\frac{1+1-1-1}{96}\frac{d^{4}f}{d{y}^4}(x_0,y_0))\\ & +O(h^3)\text{ as }h\to 0.\end{array}$

Because of the careful choices of coefficients, we can simplify this to

$g(x_0,y_0,h)=\frac{d}{dy}\frac{df}{dx}(x_0,y_0)+\frac{h^2}{6}(\frac{d}{dy}\frac{d^{3}f}{d{x}^3}(x_0,y_0)+\frac{d^3}{d{y}^3}\frac{df}{dx}(x_0,y_0))+O(h^3)\text{ as }h\to 0.$

We note that Big-O notation permits us to write the last 3 terms as $O(h^2)\text{ as }h\to 0$. Thus,

$g(x_0,y_0,h)=\frac{d}{dy}\frac{df}{dx}(x_0,y_0)+O(h^2)\text{ as }h\to 0.$

Because the multiples of $h^2$ are unaffected by adding more terms to the Taylor expansion, $n=2$ is the greatest natural number satisfying the conditions given in the problem.

Example Code

Implementing these methods is reasonably simple in programming languages that support higher-order functions. For example, the method from the first example may be implemented in C++ using function pointers, as follows:

//  Returns an approximation of the derivative of f at x.
double derivative (double (*f)(double), double x, double h =0.01) {
  return (f(x + h) - f(x - h)) / (2 * h);
}